lema · docs
Evaluation
We publish the grading sheet — misses included. Here is how faithful lema stayed to the source on three repos it had never seen.
58 of 60 blind-graded claims held against their source. Each repo’s history was turned into a decision graph, then 20 claims per repo (a deterministic sample of ADR- and PR-derived claims) were blind-graded against their sources as faithful, distorted, or fabricated — on three repos we had never seen.
By repo
Three repos, never seen before
mozilla/fxa20 / 20 faithfulMADR-formatted ADRs, disciplined PR template.backstage/backstage18 / 20 faithful2 distorted (the misses below), 0 fabricated.athensresearch/athens20 / 20 faithfulDormant repo; the product abstained rather than guess.
The misses
The two that went against us
- An open reviewer debate on backstage#34550 was read as “considered but not adopted.”
- A pull request closed in favor of consolidation was read as “this pull request was rejected.”
Both are one named class — resolution-state over-assertion on PR lifecycle — and both are addressed in the product (the resolution-state extraction rule and the vacuous-claim floor). The spot-test above has not been re-run against the fixed extractor.
The abstain
When it declined to answer
On a dormant repo, lema scored 0.383 against its 0.38 confidence floor and declined to answer — surfacing the gap ledger instead of bluffing.
Reproduce it
Don’t take our word
- Read the per-claim faithfulness verdicts shown here, including the two that went against us.
- Re-run the product yourself — `npx lema-mcp --repo github.com/mozilla/fxa` and ask the same questions. The inputs are public.
- Our grading harness is internal — we don’t claim you can run our exact scorer, only that the inputs and outputs are public, so you can grade them yourself.