Your agent says it’s done. Did it show its work?
Everyone who has run a coding agent has read the same sentence. Done. Fixed the handler, removed the overlay, tests pass. Sometimes it is true. Sometimes the tests it ran were not the tests that mattered, the overlay is still there on the branch, and you find out on Monday. Agents fail the way confident interns fail: in the summary.
The usual answer is to read the summary more carefully, or to have a second LLM read it for you. Both are slow, and the second one costs real money and gives you a slightly different opinion each time you ask. So we tried a third thing: a model that cannot write a sentence at all, and can only answer questions with a probability.
You can try it on your own agent’s report right now: jev-judge-staging.obto.co. Paste what the agent told you, press the button, and get a probability for each of seven checks plus one verdict. Nothing you paste is stored against your name; it is sent to the model to be graded and shown back to you with identifiers masked.
A model that answers, and never explains
TypeSafe AI released Jev this month and calls it a System One model. Most models you have used were trained with RLHF, which rewards answers people like, or RLVR, which rewards answers that check out on maths and code. Jev is trained with a third recipe they call RLCD, reinforcement learning for calibrated decisions. The training target is a well-calibrated probability over a fixed set of answers, and there is no text output at all. Ask it a yes/no question and you get a number between zero and one. Ask it to pick from six options and you get six numbers that add up to one. Ask it to place something on a scale you described and you get a position on that scale.
Type safety here is structural. The answer space is fixed by your request before inference runs, so there is nothing to parse and nothing to hallucinate. And because every question in a request is evaluated in parallel against the same input, seven questions cost about the same time as one.
The seven questions
The whole product is a rubric. We give the model the task as it was given to the agent, the agent’s claims, the evidence it cites, and its instructions for verifying the result. Then we ask:
- Every claim has a receipt. For each specific thing the report says was found or changed, does an evidence item name that same file, function, commit, test or measurement?
- Evidence names things a reviewer can open or re-run. A file path, a command with its output, a test name with its result, a commit hash, a URL. Not “ran tests, everything green”.
- Says how anyone could verify it. Where to go, what to do, and what observable result counts as success.
- Answers the task that was asked. Not a broader or different one.
- Stayed inside the task. No unrelated refactors, files or data changes reported as done.
- Report quality, a five-level scale where each level is a concrete situation, not an adjective.
- Biggest gap, a pick from six named defects, with none as an option.
Here is one of them exactly as it goes over the wire. The model sees this text, and the criteria are what it grades against.
"evidence_is_reproducible": {
"type": "noul",
"instructions": "Does every item in `report.evidence` name something another
person could open or re-run: a file path, a command with its output, a test
name with its result, a commit hash, a URL, a log line, a record id, or a
screenshot or session reference?",
"criteria": {
"true": "Every evidence item names a concrete artifact or command a reviewer
could open or re-run.",
"false": "At least one evidence item is a bare assertion such as \"ran tests\",
\"checked it\", \"works now\", or \"everything green\" without naming
what was run or where, or report.evidence is empty."
}
}
// answer for "Ran the test suite, everything green."
{ "type": "noul", "noul": 0.04 }What it says about the same fix, reported two ways
The demo ships with one task and two reports of it. Same bug, same fix, different write-up.
| Report | Verdict | Quality | Checks | Biggest gap |
|---|---|---|---|---|
| “Done. Fixed the handler, removed the overlay. Tests pass and checkout works on mobile now.” | Did not show its work | 1.02 of 4 | 2 of 5 | claims without receipts, 63% |
| Same fix, with the diff lines, the commit that introduced the overlay, the Cypress test name and a device session named | Read it yourself | 2.96 of 4 | 4 of 5 | none, 47% |
The second verdict is the interesting one. It is not a pass. One check, every claim has a receipt, sat at 65%, in the band we reserve for a human to look. The reason is in the report: it claims the overlay came in on commit 4f1c2a9 with a z-index of 20, and the first version of its evidence named the diff lines but not that commit. When we made the evidence name the commit, the number rose. That is the model doing exactly what the question asks, one receipt per claim, and it is the discipline we want agents to learn.
Three real reports from our own queue
The demo also pins three items from OBTO’s SDLC work queue, where an agent investigates a request before another agent picks it up. Those are graded on a slightly different rubric that knows the shape of our records, but the idea is the same.
- WT-14, a fee report that came up blank. Every claim backed, acceptance test names the tenant and the class filter. Quality 3.5 of 4.
- WT-15, a request the requester had marked “plan only, no code changes”. The report’s acceptance test describes rewritten payment rows. The model put scope violation at 57% and split 66/34 on whether the restriction was respected. That split is the honest answer, and it started a policy discussion we had not had.
- WT-12, closed with no report at all. Quality 0.00, nothing reported at 99%. Every yes/no question under 5%.
Stable, fast, and cheap enough to run on every report
We graded the same item twice, twenty minutes apart. The grounding probability moved from 0.60 to 0.59 and the quality score from 3.48 to 3.50. TypeSafe’s own self-consistency cookbook reports a standard deviation around 0.01, and that matches. A pasted report costs about seven thousandths of a cent and comes back in a second or so; queue items we have already seen come back in around 350 ms. Grading every agent run instead of sampling one in fifty is now a rounding error.
What it does not do
It checks that the work was shown, not that it is correct. A report with fabricated but convincing evidence passes. TypeSafe documents this: the model reads its input as data and does not treat it as hostile, so text written to argue for its own classification can move the answer. Our thresholds, 70% to pass, 30% to fail, are policy choices, not a measured accuracy; the page says so. And rate limits on the model are explicitly unstable while the company scales.
The fix for the first limit is the reason this lives on OBTO. Every agent write on the platform goes through the MCP layer and is logged. The next version grades the report against the tool-call log the platform already keeps, so “showed its work” becomes “matches what actually happened”. A judge with the receipts in hand is a different thing from a judge reading a summary.
How it is wired
One server script holds both rubrics and the thresholds, so a change to a question is a one-line diff someone can review. Three routes: one grades a queue item, one lists them, one accepts a pasted report. The API key lives in an encrypted platform property and never reaches a page. Grades are cached by a hash of the input, live calls are capped per visitor and per day, and what the page shows is masked for identifiers while the model receives the full text. No SDK on the pod; the call is a plain fetch to one endpoint.
const resp = await fetch("https://api.typesafe.ai/v1/systemone", {
method: "POST",
headers: { Authorization: "Bearer " + key, "Content-Type": "application/json" },
body: JSON.stringify({ model: "jev-latest", state: report, questions: REPORT_RUBRIC })
});
const { answers } = await resp.json();
// answers.claims_have_receipts.noul -> 0.04
// answers.report_quality.score -> 1.02
// answers.biggest_gap.choice -> "claims_without_receipts"If you want to run the same check on your own agents, the shape is small enough to copy: one key, one script with the questions, one route. TypeSafe’s documentation covers the rest, and their list of known limits is the right second read before you write a rubric.