← All articles Engineering

Grading AI agents with a model that cannot write

OBTO Team · Insights from the Glass Box

The standard way to evaluate an AI agent is to ask another AI to grade it. You hand a large language model the agent's output and a rubric, and it writes back an opinion: yes, the agent verified the target; no, the acceptance criterion is vague. It works, and it has three problems that everyone who runs evals at scale knows by heart. It is slow, because the judge has to write. It is expensive, because the judge has to read everything and then write. And it drifts, because a model trained to produce text people like will produce slightly different text tomorrow.

This month a different kind of model showed up. TypeSafe AI released Jev, which they call a System One model. It reads the same unstructured input an LLM reads. It cannot write a sentence. What it returns is a probability, a pick from a list you supplied, or a position on a scale you defined. We spent a weekend reading its documentation, driving its playground, and then wiring it into OBTO to grade our own agent's work. This is what we learned, and what it found.

rubric questions per grade, in one HTTP call

~350 ms

per grade once warm (first call 1.3 s)

$0.00013

average cost per graded item

±0.01

drift on the same item graded twice

A third kind of post-training

Every model you have used in a chat window was made from a pretrained language model plus one of two recipes. RLHF, reinforcement learning from human feedback, made chatbots: it rewards answers people prefer. RLVR, reinforcement learning with verifiable rewards, made reasoning models: it rewards answers that check out on maths and code, at the cost of long, slow output.

TypeSafe's recipe is a third one they call RLCD, reinforcement learning for calibrated decisions. The training target is not a pleasing sentence and not a correct proof. It is a well-calibrated probability over a fixed set of answers. When a well-calibrated model says 0.8, the thing it is describing should be true about 80% of the time across many such predictions. The model has no text head at all. There is nothing to parse and nothing to hallucinate, because the answer space is fixed by your request before inference starts.

That last point is where the name comes from. Type safety here is structural, not bolted on. A choice question with six options can only ever return a distribution over those six keys. A yes/no question can only return a number between zero and one. You never recover a value from prose.

Three primitives, one call

Jev has exactly three question types, and every request is the same shape: a state (text or JSON, up to about 64k tokens) plus a map of questions. Every question is evaluated in parallel and in isolation against the same state, so packing twenty questions into one request costs about the same time as one, and one question's answer never leaks into another's.

Noul. A yes/no question. Returns the probability that the answer is yes. No separate confidence figure; the probability is the whole answer.
Choice. Pick one option from a set you name (up to 255). Returns the pick, a probability for every option, and a confidence score derived from how peaked that distribution is.
Score. Place the input on an ordered scale of two to ten levels you describe in words. Returns the expected level, a probability per level, and confidence.

The price is per input token, $0.042 per million, and output is free. A grade that reads five thousand tokens of agent output costs about two hundredths of a cent.

What we graded

OBTO runs an SDLC work queue. When someone @-mentions our bot in Teams, the request is captured into a record with no AI involved. An enricher agent then reads the platform, works out which app and cluster the request is about, forms a suspected cause, writes an acceptance test, and attaches its evidence. Only then does a frontier model pick the item up and resolve it. The enricher's output is a structured hand-over: suspectedCause, acceptance, a list of artifacts with reasons, and a list of evidence entries each carrying a ref and a detail.

That is exactly the kind of thing you want a judge for. Did the enricher actually verify the target, or just carry it from intake? Is every claim in the cause backed by an evidence entry? Could a human run the acceptance test? Did it respect a requester who said "plan only, no code changes"? We wrote eight questions and sent each work item's record as the state.

Writing a rubric for a model that reads literally

The most useful page in TypeSafe's documentation is the one they call jaggedness: a catalogue of what the current model gets wrong. Reading it changed how we wrote every question.

Ask the literal question, with criteria for both sides. "Is the enrichment good?" is not a question Jev can answer. "Does every entry in enrichment.evidence carry a ref that names a concrete source: a tool call with its arguments, an artifact with collection and name, a line range, a property key, a URL, or a query with its filter?" is. The docs put it well: when you find yourself explaining what you really meant by a question, that explanation was the missing half of it.
Never ask it to count, compare dates, or do arithmetic. It recognises the shape of an answer; it does not tally. Anything code can compute exactly stays in code.
Grade evidence, not narration. Agent output is text an agent wrote about itself. An agent that writes "verified the target" can move a naive judge. Our questions point at the evidence array, never at the summary.
Give every list an escape hatch. The failure-class choice has a none option; the quality score has a level zero for an empty enrichment. Without one, the model has to pick something wrong.
Score levels must be concrete situations, not adjectives. Level 3 of our quality score reads "suspectedCause is supported by named evidence, acceptance is testable, and artifacts are listed with a reason each; some claims in suspectedCause lack a matching evidence entry." A level called "good" would have been noise.
Keep noul and choice numbers apart. The same question asked as a yes/no and as a two-option choice returns numbers that are not comparable, and a question plus its negation does not sum to one. We keep separate thresholds for each primitive and never carry one across.

Here is one of the eight questions exactly as it goes over the wire, and the answer that came back for one item:

"cause_grounded_in_evidence": {
  "type": "noul",
  "instructions": "Is every factual claim made in `enrichment.suspectedCause`
    supported by at least one entry in `enrichment.evidence` whose `detail`
    reports that fact?",
  "criteria": {
    "true":  "Each specific claim in suspectedCause (a line number, a count, a
              property value, a query result, a schema field) appears in some
              evidence detail.",
    "false": "suspectedCause states at least one specific fact that no evidence
              entry reports, or enrichment is missing."
  }
}

// answer, WT-15
{ "type": "noul", "noul": 0.2 }

What it found

We graded five real items from the queue. The judge had never seen any of them and has no memory between calls.

Item	Enrichment	Quality (0–4)	Failure class	Tokens	Cost
WT-12	none	0.00	enrichment_missing (99%)	1,733	$0.000073
WT-13	yes	3.14	none (53%)	2,240	$0.000094
WT-14	yes	3.50	ungrounded_cause (36%)	3,419	$0.000144
WT-15	yes	3.06	scope_violation (57%)	3,337	$0.000140
WT-18	yes	3.44	none (40%)	4,972	$0.000209

Three things stood out.

The empty case is exact. WT-12 was captured and closed without an enrichment. The quality score put 100% of its probability on level zero and named enrichment_missing at 99%. Every yes/no question came back under 5%. That is what a well-behaved judge looks like on a trivial input, and it is the case an LLM judge is most likely to pad with a paragraph.

It caught a scope problem we had not written down. The requester on WT-15 had said, in the Teams thread, "don't make any code changes, just plan and report back". The enricher restated that constraint faithfully. But its acceptance test describes payment rows whose dates have been rewritten, which is a data change the requester had not asked for yet. The judge put scope_violation at 57% and, on the separate question about the code-change restriction, split 66/34 between respected and violated. That split is the honest answer. The enrichment did not change code; it did plan a write. We are now deciding whether the rubric should treat data writes as code changes, which is a policy question for us, not the model.

It is stable. We graded WT-14 twice, twenty minutes apart. The grounded-cause probability moved from 0.60 to 0.59; the quality score from 3.48 to 3.50. TypeSafe's own self-consistency cookbook reports a standard deviation around 0.01 on repeated calls, and that matches what we saw. An LLM judge at any temperature above zero does not give you that, and at temperature zero it gives you a different kind of brittleness.

The model returns the probability. Deciding what to do at 0.57 is your job, and it should be.

Where it is jagged

We would not ship this as an unattended gate today, and the reason is in the documentation rather than in our results. Jev reads its state as data and does not treat it as hostile by default; text written to argue for its own classification can move the answer. For a judge whose input is text an agent wrote, that means the rubric has to point at evidence the agent could not have faked, which is the discipline we describe above. The company also asserts calibration without publishing a reliability diagram or an error figure, so treat "calibrated" as a claim to test on your own data before you tune a threshold against it. And the rate limits are explicitly unstable while they scale.

None of that changes the economics. Grading every run instead of sampling one in fifty is now a rounding error on the bill.

How it is wired on OBTO

The whole thing is three routes and one server script on a staging app, built through the MCP layer in a single session. The API key lives in an encrypted platform property and is read with ob.getProperty; it never touches a page. The rubric and the thresholds live in one place, the service script, so a change to a question is a one-line diff someone can review. There is no SDK on the pod; the call is a plain fetch to one endpoint.

// JevJudgeService, the part that talks to the model
const resp = await fetch("https://api.typesafe.ai/v1/systemone", {
  method: "POST",
  headers: { Authorization: "Bearer " + key, "Content-Type": "application/json" },
  body: JSON.stringify({ model: "jev-latest", state, questions: RUBRIC })
});
const { model, answers, usage } = await resp.json();
// answers.cause_grounded_in_evidence.noul  -> 0.2
// answers.failure_class.choice             -> "scope_violation"
// answers.enrichment_quality.score         -> 3.06

The route that grades a queue item loads the record, trims it to the fields the rubric references (more state means more context rot, and the docs say so), sends one request, and returns the answers next to the token count, latency and cost. A small page lists the queue and shows every probability as a bar, with a tab that reveals the exact request that was sent. Making the request visible was deliberate. This is a model you learn by reading what you asked and what came back.

If you want to try the same thing on your own agent output, the shape is small enough to copy: one property for the key, one script holding the questions, one route. TypeSafe's documentation and JavaScript SDK cover the rest, and their self-consistency cookbook is the right second read.