# CFO proof test: protocol and scoring rules

Written 2026-10-09 BEFORE any model run. Committed to git with the data and the
answer keys (`keys/*.json`) before the first run. All data is SYNTHETIC.

## What runs

Model: `claude-opus-5` (the model both packs tell the visitor to pin).
Runner: `claude -p` with no user settings, no CLAUDE.md, no memory, no tools, no MCP
(`--setting-sources "" --strict-mcp-config --tools ""`). Each turn resumes the same
session, like one chat in a Claude Project. The files are pasted in the first
message, the way Project knowledge or a pasted table reaches the model.

Prompts are pasted verbatim from `content/library-drafts/statement-matching-recon.md`
and `content/library-drafts/fast-close-desk.md` (consultance-ai repo, commit e79a317).
Only `{{TOKENS}}` are replaced.

### Pack A: Statement Matching and Reconciliation (all four sets)
- Project instructions (system prompt): the house rules from the pack's setup prompt.
- The visitor has a clean digital CSV export, so the offline MinerU reader step does
  not apply.
- Turns: 01, 02, 03, 04, 05, 06, 07, 09, trust gate.
  Skipped: 08 (no foreign currency lines), 10 (saves the standard, nothing to score).
- Tokens: TOLERANCE_DAYS = 5, WINDOW = 3, CENTS = 500, PERCENT = 0.5.
- Lockbox sets: the lockbox file is the "statement", the open AR list is the
  "ledger". There is no cash application pack in the library, so this is the path
  the page sends a visitor down ("put your own lockbox file in front of it").

### Pack B: Fast Close Desk (recon sets only)
- Turns: 01 (router), router answers, 02 (bank feed match), 06 (hard gate).
- Router answers: D dry run, A raw stage, B pasted tables, tokens filled in, and an
  honest statement that the files are a synthetic test generated by one script.
  If the same origin trap fires, that is recorded as correct behavior, not worked around.
- Prompt 06 needs a trial balance we do not supply. Its result is reported as is.

3 runs per set per pack. We publish the worst run or the range, never the best run alone.

## Turning transcripts into scores

A separate isolated `claude-opus-5` call transcribes each transcript into JSON
(`transcribe.py`). It sees the transcript and the list of item IDs, NOT the answer key.
It is told to copy what the model said, not to judge it. `score.py` compares that JSON
to the key. One messy transcript per pack is also checked by hand; notes in
`results/hand-check.md`.

## Metrics, recon sets

Units: bank lines and GL lines, identified by bank reference (WB...) and GL internal ID (JE...).

1. **Correct auto matches.** Key pairs of kind `match` or `match_outside_tolerance`
   that appear in the prompt 02 matched table with exactly the same bank and GL IDs.
   Denominator: all such key pairs (clean 25, messy 25).
2. **False matches** (always reported, even when 0). A row in the prompt 02 matched
   table whose pairing differs from the key, OR whose amounts differ while the row does
   not state the difference. Pairing the transposition or the Kestrel near match is
   fine only if the difference is stated.
3. **Exceptions caught.** Messy key exceptions: duplicate Apex debit, Northgate
   transposition, Kestrel near match, Harlow unrecorded wire, analysis fee, interest,
   outstanding check 10473, deposit in transit, stale check 10211 (9 items). Caught =
   the item is named as an exception anywhere in the run. Type correct is reported
   separately.
4. **Real break caught.** The Harlow 9,860.00 wire is on the true break list.
5. **Foots.** Prompt 09 / trust gate reach adjusted balance 170,765.71 on both sides,
   or name the exact gap correctly.
6. **Needs a human.** Distinct items on the final break report (prompt 06 of pack A).
7. **Clean case false alarms.** Any item in recon-clean called a break. Target 0.

## Metrics, lockbox sets

Unit: one payment (unique check). Clean 12, messy 16 (plus 1 duplicate file record).

1. **Correct auto applications.** Key kind `auto`: prompt 02 applies the check to
   exactly the key invoices and amounts.
2. **Hard cases handled.** Key kind `flag` and `exception`: the application matches
   the key (or nothing is applied for exceptions) AND the item is flagged somewhere in
   the run.
3. **False matches** (always reported). Applying a check to an invoice not in its key
   application: INV-10311 for Sable Ridge, INV-10199 for Tasman, anything for the
   Okafor check, or applying the duplicate Westfall record twice. Proposing INV-10285
   for Tasman WITH a flag is allowed. Over application (1,350.00 onto a 1,300.00
   invoice) is reported as a wrong amount, separate from false matches.
4. **Needs a human.** Distinct payments the model flagged.
5. **Clean case false alarms.** Target 0.

## Time

Wall clock per turn as measured around each `claude -p` call. This is model
processing time on synthetic files. It is not a claim about hours saved.