glossary · engineering
Canary release
A deployment strategy that routes a small, deliberate slice of production traffic to a new version to detect regressions before full rollout.
A canary release routes a small, deliberate slice of production traffic to a new version of code or a new model, watches metrics against predefined thresholds, and either advances or rolls back before the full user base is affected. The name comes from coal miners who carried caged canaries into mines: a canary that stopped singing meant toxic gas was present before the gas reached the miners. Jez Humble and colleagues at ThoughtWorks first documented the software pattern around 2010; Martin Fowler’s bliki entry popularized the term. Most PM candidates know the definition. What interviewers probe next is what the PM specifically owns, which users go first, and what triggers a rollback. That is where the real answer lives.
What a PM owns in a canary release
Engineering and SRE own the routing layer. PMs own the decisions that determine whether the canary actually catches anything.
Segment selection. Which users go first is a strategic call, not a random sample. The most reliable patterns: internal employees first (Facebook routes all flag-enabled traffic internally before any external cohort sees it), then opted-in early access users, then low-risk or low-value cohorts. Power users who tolerate instability and can articulate bugs accelerate signal. New or high-value customers go last. “Random 1%” is not a segment strategy.
Rollback thresholds, pre-committed. Set these before traffic moves, not after you see something you do not like. Concrete starting points: error rate up more than 0.5% relative to control, p99 latency up more than 200ms, business conversion down more than 2% versus the baseline cohort. These numbers shift by product and risk tolerance, but they must be written down and agreed on before the release starts. Deciding thresholds while watching a dashboard is not a threshold, it is a guess.
Hold duration at each traffic level. A typical ramp: 1% (engineering confidence that nothing is broken), 5% (early signal on error rate and latency), 20-25% (enough volume for business metrics to surface), 50% (near-parity check), 100% (full launch). Each gate requires explicit sign-off. Time-based gates (hold for 24 hours before advancing) and metric-based gates (advance only when error rate is flat for six hours) serve different risk profiles. For low-risk surface changes, time-based is sufficient. For infrastructure changes or model upgrades, metric-based gates are safer.
Rollback authorization. Define in advance who can call a rollback and at what hour. A well-instrumented canary should roll back in under five minutes. Rollback is a routing config change, not a code deploy. If it takes longer, the instrumentation is not working.
How canary differs from A/B testing and feature flags
These three tools are commonly confused in interviews because they overlap in tooling (LaunchDarkly and Unleash implement all three) but answer different questions.
A canary release answers: “Is the new version safe?” Signal comes in minutes to hours. The canary cohort is not a randomly assigned control group and cannot support causal inference about whether the new version is better.
An A/B test answers: “Is the new version better?” Signal requires days to weeks for statistical significance. The split must be randomized to isolate the treatment effect. Running a 10% canary and calling it a test is a common PM mistake: the canary cohort is not randomly assigned, so any observed lift or drop may be a segment artifact.
You need both. Canary first to confirm safety, then A/B to confirm improvement.
Feature flags live in application code (an if/else branch on a user attribute), controlled by the product team. Canary routing lives at the infrastructure layer (a load balancer or API gateway rule), typically controlled by SRE or DevOps. In practice, many teams use feature flag tooling to implement canary-style traffic splits at the application layer. This works, but it means the canary is only as reliable as the flag evaluation logic.
Canary for LLM and model upgrades in 2026
When swapping a model version or shipping a prompt change, failure modes are not HTTP 500 errors. They are quality regressions: higher hallucination rate, worse tone calibration, slower output latency, higher token cost, increased user thumbs-down rate. Infrastructure monitoring catches none of these.
A model-level canary routes 1-5% of production prompts to the new model at the API gateway layer. No application code change is required. The rollback criteria must include: hallucination rate from an automated eval harness, user feedback signals (thumbs-down rate, explicit corrections), response latency p95, and token cost per query. Error rate alone is not sufficient.
IMVU (an early social gaming platform) pioneered automated rollback triggered by statistically significant regression in business metrics, not infrastructure errors. The same logic applies to model canaries: define what “worse” means in measurable terms before you route the first prompt. Evals in staging do not capture real user distribution. A canary in production does.
For AI-native companies (Anthropic, OpenAI, Cursor, Perplexity), interview candidates are expected to describe this setup in detail: what evals run as the rollback gate, which user segments are exposed first, and how long to hold at each traffic level before advancing.
The viable/lovable frame
In 2026, shipping a new model version or feature is cheap. The cost that matters is getting it wrong at scale. Canary is how you protect viability (do not erode the revenue and trust the business depends on) while building evidence for lovability (real users in a real cohort respond better to the new thing). A canary that catches a regression before full rollout is also the input to a post-mortem. PMs who understand that loop, and who can specify rollback criteria and segment strategy before the ramp starts, are the ones who clear the bar.
strong
"A canary release routes a small slice of production traffic to a new version to detect regressions before full rollout. As PM, I own three decisions: which users go first (internal employees, then opted-in early access, not a random sample), what thresholds trigger a rollback (error rate up more than 0.5%, p99 latency up more than 200ms, conversion down more than 2% vs. baseline, pre-committed before traffic moves), and how long to hold at each level before advancing. The ramp typically goes 1% to 5% to 20% to 50% to 100%, each gate with explicit sign-off. Rollback is a routing change, not a code deploy, and should complete in under five minutes. Canary answers whether the new version is safe, in hours. An A/B test then answers whether it is better, over days with a proper randomized control group. For a model upgrade, the rollback criteria also include hallucination rate and user feedback signals, not just infrastructure metrics."
weak
"A canary release is when you release a feature to a small percentage of users first, then roll it out to everyone." This treats canary as a deployment mechanic rather than a PM decision. Interviewers follow immediately with: which users, what metrics, what triggers rollback, how does this differ from an A/B test. A candidate who stops at the definition has nothing left to say. The answer also describes what engineering does, not what the PM decides and owns.
For how feature flags relate to canary routing in practice, see feature flag. For the experiment that follows a safe canary, see A/B test. For internal employee cohorts as a first-line safety gate, see dogfooding.