# lema docs — Evaluation

We publish the grading sheet — misses included. Here is how faithful lema stayed to the source on three repos it had never seen.

- 96.7% faithful (58 of 60)
- 0 fabrications
- 30/30 ADR-derived

Each repo’s history was turned into a decision graph, then 20 claims per repo (a deterministic sample of ADR- and PR-derived claims) were blind-graded against their sources as faithful, distorted, or fabricated — on three repos we had never seen.

## By repo
- **mozilla/fxa** — 20 / 20 faithful. MADR-formatted ADRs, disciplined PR template.
- **backstage/backstage** — 18 / 20 faithful. 2 distorted (the misses below), 0 fabricated.
- **athensresearch/athens** — 20 / 20 faithful. Dormant repo; the product abstained rather than guess.

## The misses
- An open reviewer debate on backstage#34550 was read as “considered but not adopted.”
- A pull request closed in favor of consolidation was read as “this pull request was rejected.”

Both are one named class — resolution-state over-assertion on PR lifecycle — and both are addressed in the product (the resolution-state extraction rule and the vacuous-claim floor). The spot-test above has not been re-run against the fixed extractor.

## The abstain
On a dormant repo, lema scored 0.383 against its 0.38 confidence floor and declined to answer — surfacing the gap ledger instead of bluffing.

## Reproduce it
1. Read the per-claim faithfulness verdicts shown here, including the two that went against us.
2. Re-run the product yourself — `npx lema-mcp --repo github.com/mozilla/fxa` and ask the same questions. The inputs are public.
3. Our grading harness is internal — we don’t claim you can run our exact scorer, only that the inputs and outputs are public, so you can grade them yourself.