Route low-confidence fields to review

Every extracted field carries its own confidence. The point of that number is to make review selective: instead of a person checking every document or none, a person checks the fields that are actually uncertain.

Confidence is per field

A document-level score hides the thing you need. A statement can be parsed almost perfectly while one smudged transaction descriptor is a guess, and averaging that into a single number loses which one it was.

{
  "endingBalanceMinor": { "value": 4523167, "confidence": 1.0 },
  "accountHolder":      { "value": "R. Kumar", "confidence": 0.99 },
  "accountNumber":      { "value": "****4471", "confidence": 0.72 }
}

Set thresholds by consequence

One global threshold is the wrong shape, because fields do not cost the same when they are wrong. A misread account number sends money to the wrong place; a misread merchant name makes a category slightly off.

const THRESHOLDS: Record<string, number> = {
  accountNumber: 0.95, // Wrong value sends money somewhere else.
  routingNumber: 0.98,
  endingBalanceMinor: 0.9,
  accountHolder: 0.85,
  descriptor: 0.6, // Only affects categorization.
};

const DEFAULT_THRESHOLD = 0.8;

export function fieldsNeedingReview(doc: ParsedDocument): string[] {
  return Object.entries(doc.fields)
    .filter(([name, f]) => f.confidence < (THRESHOLDS[name] ?? DEFAULT_THRESHOLD))
    .map(([name]) => name);
}

Branch inside a workflow

Rather than writing the routing yourself, express it as a workflow so the review step, its assignment, and its history are recorded with the run.

{
  "steps": [
    { "id": "parse", "type": "parse" },
    {
      "id": "gate",
      "type": "branch",
      "when": [
        {
          "if": "parse.reconciliation.ok == false",
          "then": "review"
        },
        {
          "if": "any(parse.fields, .confidence < 0.85)",
          "then": "review"
        }
      ],
      "else": "deliver"
    },
    { "id": "review", "type": "human", "queue": "ops", "next": "deliver" },
    { "id": "deliver", "type": "relay", "target": "webhook" }
  ]
}

Tune with the outcomes

Reviewers either correct a field or confirm it. Both are labels. Track the rate at which flagged fields turn out to be correct: if reviewers confirm 98% of what a threshold catches, that threshold is too high and is spending human attention for nothing.

The goal is not fewer review queues. It is a review queue whose contents are usually wrong.