Why do AI agents work in demos but break in production?

A demo runs one happy user against clean data on the happy path. Production adds concurrent users, multi-tenant data isolation, provider rate limits, messy edge-case inputs, cost that accumulates per run, and the need for an audit trail. Those conditions are absent from the demo, so the failure modes they cause never show up until real users arrive.

What is the hardest part of moving an AI agent to production?

Giving the agent write access to real systems safely. In a demo the agent only suggests; a human reviews and ships. In production, if the agent acts on live data, the human review-and-deploy gate is gone. Replacing that gate with something structural, rather than hoping the agent is always right, is the hard part.

Can an AI agent safely write to a production system?

Yes, if the write path enforces its own checks instead of trusting the agent. On OBTO a line-anchored patch is refused when the anchor text no longer matches what is live, and an upload is refused when the bytes do not hash to what was promised. Those structural checks stand in for the human code review and CI that a standard deploy pipeline relies on.

How is production-readiness for an agent different from a normal web app?

A normal app fails predictably; an agent can fail creatively. It can loop, retry, drift from the original request, or take an unexpected action on live data. Production-readiness therefore means bounding behavior: a cost ceiling per run, hard iteration limits, tenancy isolation, a safe write path, and a receipt of every action for audit.

Does OBTO make AI agents production-ready?

OBTO is built production-first: agents run on the live platform with a structural safe-write path, per-run cost and step tracing through the Glass Receipt, and tenancy handled by the platform rather than bolted on. It does not make an agent infallible, but it makes the failure modes bounded and visible instead of silent.

← All articles Insights

It works in the demo. Then production happens.

OBTO Team · Insights from the Glass Box

The demo goes perfectly. The agent reads the ticket, checks the record, drafts the reply, and closes the loop while everyone watches. Someone in the room says "ship it." Two weeks after launch, the support queue is full of runs that stalled, double-charged, or answered the wrong customer, and nobody can say which ones or why.

This is the most common story in AI right now, and it is not really about model quality. The demo and production are two different jobs. A demo has to work once, for one person, on a path you chose. Production has to work every time, for everyone, on paths you did not choose. Almost everything that makes agents hard lives in that gap.

The demo is a controlled environment

A demo succeeds because you quietly remove every hard variable. There is one user, so there is no concurrency and no question of whose data is whose. The input is clean, because you picked it. The path is the happy path, because you rehearsed it. Latency does not matter, cost does not matter, and if the agent does something slightly off, a human is standing right there to smooth it over.

None of those conditions survive contact with real usage. Removing them is not cheating — a demo is supposed to show the shape of the thing. The mistake is reading "it worked in the demo" as "it is ready," when the demo was engineered to hide exactly the parts that are hard.

What production actually adds

Production is not a bigger demo. It introduces categories of problem the demo never had.

Many tenants, one system. The moment you have more than one customer, every read and write has to be scoped to the right one. An agent that fetches "the latest invoice" has to fetch this account's latest invoice, never a neighbor's. Multi-tenant isolation is invisible in a single-user demo and unforgiving in production.

Concurrency and rate limits. Ten users at once means ten agent loops competing for the same model API, the same database, the same downstream services. Providers throttle. Queues back up. A workflow that ran in four seconds alone can time out under load, and a timeout mid-run is where half-finished actions come from.

Messy inputs and unhappy paths. Real tickets are misspelled, half-empty, in the wrong language, or describe two problems at once. Every branch you did not rehearse is a branch the agent will eventually take.

Drift between what was asked and what got built. Agents are probabilistic. The same request can produce a slightly different action twice in a row. Over thousands of runs, "slightly different" becomes a long tail of behavior no one specified — the gap between what the agent was asked to do and what it actually did.

Cost with no ceiling. A loop that retries a broken tool ten times still bills for ten tries. Without a per-run limit, one pathological run can cost more than a thousand healthy ones, and you find out on the invoice. We wrote separately about putting a number on this in AI agent cost tracking.

No memory of what happened. When a run goes wrong in production, "the model messed up" is not a diagnosis. You need to see the actual steps: which tool returned what, where it branched, what it wrote. If that record does not exist, every incident is unfalsifiable.

The gap nobody budgets for

Here is the part that separates a chat feature from a production agent. Most AI tooling is safe because the agent never actually touches anything. It suggests a reply, drafts a PR, proposes a query — and a human reviews it and ships it. The human is the safety gate. That is why an autocomplete in your editor can be wrong all day with no consequences: nothing it produces reaches production without a person in the loop.

An agent earns its keep when it stops suggesting and starts doing: resolving the ticket, updating the record, deploying the change. The instant it does that on live data, the human review-and-deploy gate is gone. This is the demo-to-production gap in one sentence — not "is the model good enough," but "what stands in for the review step now that no human is reading every action before it lands."

The common answer is to keep the agent out of production: it drafts, a person still ships. That is a legitimate choice, and for high-stakes changes it is the right one. But it also caps how much work the agent can actually take off your plate. If a human has to approve every action, you have built a very expensive suggestion box.

What actually closes it

The alternative is to let the agent write to production and make the write path enforce safety, rather than trusting the agent to be right. This is the design OBTO runs on, and we use our own site as the proof: the page you would be reading is itself a database record, edited by an agent with direct write access to the live system.

Two structural checks do the work a human reviewer used to do. A patch is line-anchored: it carries the text it expects to find, and if that anchor no longer matches what is live, the patch refuses to apply instead of editing the wrong place. An upload is checksum-gated: it carries a sha256 of the bytes it promised, and if the assembled file does not hash to that value, the write is rejected. Neither check asks whether the agent had good intentions. They ask whether the operation is still valid against reality, and they fail closed when it is not.

That substitution — structural integrity checks standing in for human code review and CI — is what lets an agent write to a live system without a person gating every change. It does not make the agent infallible. A stale patch still fails; it just fails loudly and harmlessly instead of corrupting the wrong record silently. Pair that with a per-run ledger — the Glass Receipt, which records every step, tool call, and token — and the two hardest production questions, "is this write safe" and "what did the agent actually do," both have concrete answers.

A short production-readiness check

Before you call an agent ready, walk the list the demo let you skip:

Tenancy: is every read and write scoped to the right customer, by the platform, not by hope?
Load: what happens to a run when the model API throttles or a downstream service is slow?
Cost ceiling: is there a hard limit per run so one pathological loop cannot run up the bill?
Safe writes: can a write land in the wrong place or on stale state, or does the write path refuse when reality has moved?
Audit trail: after a bad run, can you replay exactly what happened from a record, not a guess?
Real verification: is "done" defined by a real user or a real browser rendering the result, not by the agent reporting success?

These are the same instincts good engineers bring to any system going live. Agents just make them non-optional, because an agent can fail in more shapes than a deterministic app. If this framing is useful, our take on the broader pattern is in prototype to production, and the case for owning the whole stack is in self-hosted agent platform alternatives.

The demo earns you attention. Production earns you trust. The teams that make the jump are not the ones with the best demo — they are the ones who made the failure modes visible and bounded before the first real user showed up.