Inspiration

Status pages, QBRs and incident reviews in Confluence are full of numbers that came from somewhere else: a Jira ticket, a finance page, a dashboard snapshot. Someone copies the number once, the source moves on, and nobody re-checks. Leaders then make decisions on numbers that were never true or stopped being true.

What it does

Countercheck reads a Confluence page, finds every numeric claim (counts, percentages, durations, money like $1.2M), and checks each one against the Jira issues and Confluence pages linked in the same paragraph. Every number gets one of four verdicts:

  • Verified: the same value (with units and sensible rounding) appears in the cited source, and Countercheck quotes the exact evidence.
  • Not in cited source: the paragraph cites a source, but the source says something else (for example the page says 3.5 hours, the ticket says 3.4).
  • No source cited: the number has no link next to it.
  • Source unreadable: the link is broken or the user cannot read it.

It runs two ways: as a Confluence content action (page menu, Apps, Countercheck numbers) and as a Rovo agent action, so you can ask "Which numbers on this page have no source?" in chat. The agent is instructed never to call a number verified unless the action returned a quote for it.

How we built it

Forge (Node 22 runtime) with a UI Kit frontend and a resolver. The core is a deterministic verifier: it parses the page storage format into paragraphs, extracts numeric claims with their units, filters out years, dates and version numbers, resolves each paragraph's links to Jira issue keys and Confluence page IDs, fetches them through the Forge product APIs as the current user, and matches values with unit normalisation ($1.2M equals 1,200,000 USD) and rounding tolerance. No model guesses the verdict; the Rovo agent only explains results the verifier returned.

Challenges we ran into

Numbers are messy: "down from 6 hours" has two claims in one sentence, "1,200,000 USD" and "$1.2M" are the same value, and version strings like 2.3.1 look like numbers. Getting zero false "verified" results mattered more than coverage.

Accomplishments that we're proud of

On the demo page it caught both planted errors (3.5 vs 3.4 hours, 91 vs 81 escalations) and the uncited CSAT figure, while verifying the four correct numbers with quoted evidence.

What we learned

Trust in a page comes from traceability, not from AI confidence. A small deterministic core plus an agent that can only report what the core proved is a better design than letting a model judge numbers.

What's next for Countercheck

Checking numbers against Jira JQL counts ("412 open tickets" against a live query), a page banner showing how many numbers are verified, and re-checking automatically when a page is published.

Share this project:

Updates

Submission history