Ship an Agent Like Risk: Version, Gate, Canary, Roll Back

A support agent is handling your customers fine until someone edits the system prompt from "summarize the issue" to "summarize the issue concisely." An hour later it is truncating every response to two sentences, dropping context the escalation team needs, and the ticket backlog triples before anyone connects the dots. One word of prompt changed production behavior, and there was no way to roll it back without redeploying the entire service.¹

That is the moment teams realize an agent is not a service. A service has one artifact you know how to version, the code, and one definition of broken, an error or a five-nines violation. An agent is a bundle of code, prompt template, model version, tool contracts, and persistent state, and each piece changes independently and silently. This article is the sibling of two earlier ones: the model gateway is the runtime control plane that routes, caches, and fails over each request, and durable execution is how a long-running agent survives a crash. Both handle the agent while it runs; neither answers the question of how you promote a changed agent safely, which is what release engineering is for. A model provider bumps an inference version under an alias and the agent's tone shifts overnight. A teammate rewords a few-shot example and a downstream parser starts failing. The old playbooks, unit tests and deploy and watch, were built for deterministic software and they stop being enough the day the thing you ship reasons for a living.²

The fix is not better prompts. It is the same discipline mature infrastructure teams apply to any high-risk change, applied to a bundle that reasons. This article walks the full release pipeline for production agents: version the whole execution context as one immutable artifact, gate every promotion with regression evals on real traffic, roll out gradually through shadow mode and a canary with session-sticky routing and automated thresholds, and roll back in seconds by repointing a pointer instead of redeploying.³

Why an agent is not a service

The easy way to see the difference is to count how many independent things change when you ship. A traditional service has one input that is deterministic: send the same request, get the same response, and a test suite that passes reliably is a genuine signal. An agent breaks that assumption in four ways at once.¹

First, outputs are non-deterministic. The same input produces different output across runs, because inference arithmetic is not stable under parallel batch processing. Research has documented accuracy variations of up to fifteen percent across runs with identical inputs, even with temperature at zero. That one fact makes a unit test an unreliable signal: a suite that passes ninety-eight percent of the time might be fine or might be masking a regression, and you cannot tell which.⁴

Second, the thing you are versioning has at least four layers. Code, prompt template, model version, and tool contracts all sit in the same bundle, and a change to any one shifts behavior in ways the others do not. Third, agents are stateful. A user mid-conversation does not care that you just deployed v2.1, they expect continuity, and rolling back means deciding what to do with state that v2.1 created but v2.0 does not understand.¹

Fourth, agents cascade. Agent A calls agent B which calls a tool, and the contract between them is often unwritten. Roll back agent B and agent A breaks because it depends on a response format only the newer B produces. In a multi-agent pipeline every deployment is a coordination event, not a single-service swap. Each of these four is manageable on its own; all four at once is why "deploy and watch" fails for agents and works for code.¹

Version the whole bundle, not just the prompt

The foundational rule is to treat every prompt as an immutable, versioned artifact, but that is only the start. Versioning the text alone leaves you unable to reproduce or roll back the behavior a user actually experienced, because agent behavior depends on far more than the instructions. A reliable version captures the prompt, the model and its parameters, the tool definitions, and the retrieval configuration together.³

Think of it as four pinned layers. The code version is the application logic, orchestration, routing, and error handling, which most teams already track in Git. The prompt version is every template the agent uses, including system prompts, few-shot examples, and chain-of-thought instructions; mature teams keep prompts as versioned artifacts in a separate registry with their own release cycle so a prompt switch never requires a code deploy. The model version is the exact identifier, not the alias, because claude-sonnet-4-6-20250514 is not the same model as claude-sonnet-4-6-20250620. The tool contract version is the schemas, endpoints, and response formats of every tool the agent can call; when a vendor updates an API or an internal search index changes shape, that is a version change for the agent even if no agent code changed.¹

A deployment manifest ties the four layers together into one reproducible unit. If you cannot recreate the exact behavior from the manifest, you do not have versioning yet, you have a label. The strongest teams add to this a strict rule: an artifact ID is the hash of code, prompt, model config, and dependencies, and once a version ships it is never edited in place. You cannot change the prompt of v1 in production; you create v1.1. That immutability is what makes a later rollback safe, because the old version still exists exactly as it ran.⁵

Each environment pins to a specific version through a tag or alias, never a floating "latest" pointer. Dev points at the newest candidate for rapid experimentation, staging points at a release candidate under validation, and production pins to a reviewed, stable version. Tagging lets you develop and validate new versions in isolation, promote them without code changes, and switch back when performance declines. Production must never resolve to "latest," because that reintroduces exactly the drift and surprise you are trying to eliminate.³

Gate every promotion with evals

Prompt and model changes should never reach production blindly, and the tool that stops them is the eval gate: a regression suite run before a version moves up an environment. Because outputs are probabilistic, you cannot assert your way to confidence the way pytest asserts a sum. The prompt is a hyperparameter of behavior, and a minor tweak like "be more concise" can cause a massive regression in reasoning, an agent that stopped checking for errors because it was optimizing for brevity. That is exactly why agent CI needs an evaluation pipeline the way code CI needs unit tests.⁵

The practical shape is a golden dataset built from real interactions, replayed against the candidate. Instead of a handful of hand-tuned examples, a production-grade gate holds a few hundred curated inputs with known-good answers, then scores the candidate on the dimensions that matter for the use case, typically faithfulness, answer relevance, tool-usage accuracy, and latency. The pass criteria should be concrete and tied to requirements, such as faithfulness above 0.9, latency under a bound, and a regression threshold that blocks the build if the score drops more than a few percent below the main branch. The gate has to be a true gate, blocking merge, not a report sitting in a dashboard that nobody reads.⁵

Layout the thresholds per category, not as one blended number, because LLM quality does not reduce to a single metric. A model can be more accurate on factual questions, more verbose, and worse at tone all at once. A blended score averages those away and lets a real regression hide behind an improved other dimension. Per-category thresholds with different severities are harder to set up but produce far fewer false alarms, because they only trip when one dimension you explicitly value degrades. The goal is to make "the prompt looks better" become "the prompt measurably did not regress," which is a statement with a number behind it.⁴⁶

The strong pattern is to let the offline dataset grow from production. When a regression does slip through, capture the failing trace, add it to the dataset, build an evaluator for that specific failure mode, and validate the fix against the expanded set before redeploying. Over weeks the regression suite stops being a synthetic benchmark and becomes a log of your real failure history, which is the corpus most likely to catch the next regression too.⁶

Roll out gradually: shadow, then canary

An eval gate tells you a candidate is not obviously broken on your curated set. It cannot tell you how it behaves on the real distribution of prompts your users send, because staging datasets are static and production traffic is not. That gap is what gradual rollout exists to close, in two stages with a third optional one.²⁴

Shadow mode is the lowest-risk starting point for any significant change. Route all production traffic to the current version as normal, and have a background process duplicate the same requests to the candidate, whose responses are never shown to users. They go to a logging and evaluation system instead, where you compare the two versions offline, often with a judge model scoring both. Shadow mode is the step every team is tempted to skip and every team should do, because it gives you a large set of side-by-side comparisons under real load with zero user impact. Run it for at least a day before you trust a candidate for a big change, and compare tool-call patterns, response lengths, error rates, and latency, not just the final answer. The honest cost is that doubling the model roughly doubles inference spend for the evaluation window, which is why shadow is the right tool for major changes, model upgrades, prompt restructuring, new tool schemas, and not for a typo fix.¹⁴

Once shadow gives you confidence the candidate is not obviously broken, a canary moves the risk to real users at small scale. Route a small percentage of traffic, commonly one percent and sometimes as low as a tenth of a percent for high-stakes work, to the candidate while the rest stays on the baseline, then ramp through roughly five, twenty-five, fifty, and full rollout, with explicit metric gates at each step. The infrastructure requirement that most teams miss is consistent user routing: hash by user or session ID, not per request, so a user mid-conversation stays on one version for the whole rollout. Without it a conversational user could hit the stable model on one turn and the candidate on the next, producing an incoherent sequence.⁴²

The metrics you watch in a canary are not uptime alone. Error rate and latency catch outages, but a model can be flawless on both and still produce worse reasoning or shift tone. Watch latency percentiles rather than averages, cost per request because token counts change with model version, error and refusal rates, output length distribution, and user feedback signals like thumbs-down and session abandonment. Set explicit automated rollback thresholds, such as a latency percentile jumping more than forty percent, a refusal rate climbing more than five percent, or a cost-per-request delta beyond budget, and let the controller route all traffic back to baseline without a human at two in the morning. If you cannot roll back in under a minute, you do not have production AI yet, you have a demo.⁴⁶

Roll back by repointing a pointer, not redeploying

The payoff of all the immutability is a rollback that is a configuration change, not a release. Because old versions stay immutable and retrievable in the registry, and production references a tag rather than a hardcoded version, reverting is simply repointing the production tag from the bad version to the last known-good one. That takes seconds and requires no code deploy, and it is why the current stable version must always be kept live and one pointer change away.³

The trace requirement that makes rollback possible is logging. Every agent request should record the exact prompt, model, parameter, and tool configuration that produced it. When a regression appears, version-tagged traces turn debugging from guesswork into a lookup, and they give you the structured history you need to assemble the regression datasets that will feed your next eval gates. You cannot roll back what you cannot trace, and you cannot improve a system whose history is a blank.³

The part everyone acknowledges is hard is what rollback does to state. A user mid-conversation on v2.1 expects that conversation to continue even as you push traffic back to v2.0. The clean options are to drain existing conversations to completion before switching, which works when they are short, or to use sticky sessions that route existing conversations to the old environment until they finish while new conversations go to the new one. A state migration that exports v2.1 state, transforms it to v2.0's schema, and reimports it is the nuclear option, usually more trouble than it is worth. Match the state strategy to how long conversations live; a support chat that closes in minutes drains cleanly, a long-running workflow needs a plan.¹

In a multi-agent pipeline the rollback is also a coordination event. Because agents cascade, rolling one agent back can break the agents that now expect its newer response format. The pragmatic mitigation is to treat an agent's tool and response contracts as versioned interfaces that downstream consumers pin to, and to rehearse the rollback of the highest fan-in agent the way you would a database migration, in staging, before you ever need it under incident pressure.

The discipline is the differentiator

Adroit runs a delivery fleet in which many worker profiles ship config changes to their prompt, model routing, and tool sets on a weekly cadence, and we have been burned by exactly the failure this article describes: a config change that looked inert on review and silently shifted a worker's behavior in production, with no fast path back until we built the release discipline around it. The fix was not better prompting. It was version-tagged config promotion through an environment pipeline, regressions replayed against a captured dataset before promotion, and a rollback that repoints a tag and takes seconds. An external observer needs none of Adroit's internal names to apply the lesson, because the mechanism is the transferable part: when you can version a behavior bundle, gate it with evaluation on real data, and roll it back by pointer, you can upgrade an agent without betting the production experience on a review.

The organizations that get agentic systems to production reliably do not have a better model. They have a release pipeline that treats the model as one pinned layer of a larger bundle and applies the same rigor to every change that they already apply to a risky database migration. Version the whole execution context, gate every promotion with evals on real data, roll out through shadow and a session-sticky canary with automated thresholds, and roll back in seconds by repointing a tag. A support agent that cannot be rolled back in under a minute is not a deployment, it is a bet, and the bet gets expensive the day someone rephrases a prompt and nobody connected it to the ticket backlog until three days later.

Start by making one change and tracing its full path: bump a prompt version, run it against your golden dataset, watch it in shadow for a day, canary it at one percent with a latency and refusal guard, and if a metric wavers, repoint the tag. Once that single change rolls safely end to end, the pipeline, not the prompt, is what you take to the next change, the next agent, and the next system that reasons for a living.