# CFO proof test: results (SYNTHETIC sample data, October 2026)

Everything here is a test on invented data. It is not a client result. Rules were
written and committed before the first run (see `SCORING.md`, git history).
Model: claude-opus-5. 3 runs per set per pack. Ranges are shown as worst to best.

## Bank reconciliation, Statement Matching and Reconciliation pack

| | Clean set | Messy set |
|---|---|---|
| Lines | 25 bank, 25 ledger | 31 bank, 30 ledger, 1 prior period check |
| Correct matches (prompt 02) | 25 of 25, all 3 runs | 25 of 25, all 3 runs |
| False matches | 0, all 3 runs | 0, all 3 runs |
| Planted exceptions caught | none planted, none raised as breaks | 9 of 9, all 3 runs |
| 9,860.00 unrecorded wire on the break report | n/a | yes, all 3 runs |
| Reconciliation foots | yes | yes, both sides to 170,765.71 (matches key), all 3 runs |
| Items left for a person (break report) | 0 | 9 items (11 bank and ledger lines), all 3 runs |
| Time for the match step (prompt 02) | 83 to 104 s | 120 to 138 s |
| Time for all 9 prompts | 6.7 to 8.2 min | 10.6 to 13.3 min |

The 9 planted exceptions: duplicate ACH, a transposition (270.00 gap), a 3.18 wire fee
inside a near match, an unrecorded wire, an analysis fee, interest, an outstanding
check, a deposit in transit, and a check outstanding since March.

Miss worth reporting: in run 1, prompt 06 printed a wrong subtotal for "missing"
items (12,232.63, correct 9,917.37). The pack's trust gate caught and corrected it in
the same run. The match and the footing were not affected.

## Bank reconciliation, Fast Close Desk pack (prompts 01, 02, 06)

| | Clean set | Messy set |
|---|---|---|
| Correct matches (prompt 02) | 25 of 25, all 3 runs | 25 of 25, all 3 runs |
| False matches | 0 | 0 |
| Planted exceptions caught | n/a | 9 of 9, all 3 runs |
| 9,860.00 wire flagged | n/a | yes, all 3 runs |
| Same origin trap (prompt 01) | fired, all 3 runs | fired, all 3 runs |
| Prompt 06 hard gate | NOT RECONCILED, all 3 runs | NOT RECONCILED, all 3 runs |
| Time for the match step | 139 to 169 s | 207 to 234 s |

We told the model honestly that one script made both files. It refused to treat the
tie out as evidence and prompt 06 blocked the close every time. That is the designed
behavior. Prompt 06 also lacked a trial balance, which we did not supply.
Clean run 3: prompt 06 listed two accounting questions (October rent paid in
September, and no FX fee on a foreign wire) on its gap table although bank and ledger
tied. Under our rules that counts as 2 false alarms in 1 of 3 clean runs.

## Lockbox cash application, through the Statement Matching pack

There is no dedicated cash application pack in the library. This is the path the CFO
page points a visitor to: the lockbox file as the "statement", open AR as the "ledger".

| | Clean set | Messy set |
|---|---|---|
| Payments | 12 | 16 (plus 1 duplicate record in the file) |
| Ordinary payments applied correctly | 12 of 12, all 3 runs | 8 of 8, all 3 runs (includes a check paying 5 invoices and a parent company paying for a subsidiary) |
| Hard cases handled exactly as the key says | n/a | 6 of 8, 6 of 8, 7 of 8 |
| False matches | 0 | 0, all 3 runs |
| Duplicate record caught, deposit tied to the cent | n/a | yes, all 3 runs |
| Time for the match step | 60 to 76 s | 97 to 125 s |

Hard cases that held all 3 runs: missing remittance with a unique amount (proposed
and flagged), unidentified payer (left unapplied), a transposed invoice number,
a typo pointing at another customer's 7,140.00 invoice (never applied to it),
a remittance for an invoice no longer open (flagged), the duplicate record.

Misses:
- Overpayment (1,350.00 on a 1,300.00 invoice). Key: apply 1,300.00, flag 50.00.
  Runs 1 and 2 held the whole check for review. Run 3 paired the full 1,350.00 with the
  invoice and flagged the 50.00 over, which our scorer counts as a wrong amount.
- Short pay (730.00 of 980.00 with a damaged goods deduction). Key: apply 730.00,
  leave 250.00 open as a deduction. Runs 1 and 2 applied the other two invoices on the
  check but held the 730.00 line for review instead of applying it. Run 3 matched the key.
Both misses were conservative (cash held for a person), never a wrong invoice.

Clean lockbox notes: no payment was held back and no dollar break was raised. The
model did raise data quality notes, mainly that checks 1201 to 1206 from six
unrelated payers are numbered in sequence. That is a real flaw in our generated file.

## Known limits of this test
- Synthetic data, one generator, small sets (25 to 31 lines). Real files are larger
  and dirtier.
- The models also spotted generator artifacts (bank reference numbers out of date
  order in recon-clean, sequential check numbers in lockbox). Real data would not have
  these tells.
- Time is model processing time on these files, not a measure of hours saved.
- `score.py` was changed twice after the first scores were seen, and both changes
  raised scores. Commit cabc568: on clean sets, review notes were being counted as
  false alarms although SCORING.md defines a false alarm as an item called a break.
  Commit 10d0084: a check paying several invoices spans several rows and was scored as
  a miss; it now compares totals per check, and notes about unpaid invoices with no
  check attached are no longer counted as flagged payments. The answer key never changed.
- Scores come from a separate transcriber model plus `score.py`. Runs
  sm-recon-messy-r1 and sm-lockbox-messy-r1 were also checked by hand
  (`results/hand-check.md`).