Shadow, Canary, A/B and Bandits: Rolling Out ML Models Safely

Say a team has spent months on a new machine-learning model, and offline evaluation looks great: better precision, better recall, fewer false positives and acceptable latency. The natural question is whether it can go to production. It probably can. A more important question follows, though: should it get 100% of production traffic right away? Usually it shouldn't. A model that shines on historical data can behave very differently once it meets real customers, real traffic, unexpected inputs, infrastructure failures and shifting data distributions.

Production ML teams therefore reach for four strategies, each with a different job:

• Shadow deployment lets you watch a new model without acting on it.
• A canary release exposes it to a small slice of live traffic.
• An A/B test compares it head to head with the old one.
• A multi-armed bandit shifts traffic toward whatever works best as results arrive.

The running example: fraud detection

A financial platform handles 1 million transactions per day, and each one passes through a fraud-detection model. The current production model is V1. The ML team has built V2.

Customer Transaction
        ↓
    Payment API
        ↓
  Feature Service
        ↓
 Fraud Model V1
        ↓
  Fraud Risk Score
        ↓
 ┌──────┼────────┐
 ↓      ↓        ↓
Approve Review  Decline

Offline numbers say V2 is better. Swapping it in for all one million transactions at once still adds needless risk, because the bigger the immediate rollout, the bigger the potential blast radius. So V2 gets introduced gradually.

Shadow deployment: observe before acting

Shadow deployment is among the safest ways to test a new model on production traffic. Real traffic goes to the new model, its predictions are logged, and customers see no effect. The architecture:

Production Transaction
                         ↓
               ┌─────────┴─────────┐
               ↓                   ↓
          🟢 Model V1         🔵 Model V2
           Production             Shadow
               ↓                   ↓
       Actual Decision        Log Prediction
               ↓
            Customer

Take a transaction like this:

Amount: $4,800
Location: Dallas
Device: New device
Transactions last 5 min: 4
Merchant risk: Medium

V1 gives it a fraud probability of 0.31 and approves it. V2 gives 0.82 and would send it to review. Because V2 runs in shadow mode, its answer doesn't touch the customer. It's only recorded. Over time the logs let you compare:

• Prediction differences
• Precision and recall
• False positives and false negatives
• Inference latency
• Feature availability
• Data drift
• Resource consumption

Shadow mode answers how V2 would behave in production without giving it control. Shadow means seeing before you switch.

Canary release: start small, grow with confidence

Suppose V2 looks healthy after shadow testing. Now it should make real decisions, but the move isn't 0% to 100%. It begins with a sliver of traffic:

Production Traffic
                 ↓
          ┌──────┴──────┐
          ↓             ↓
        99%             1%
          ↓             ↓
     🟢 Model V1   🟡 Model V2

Of a million daily transactions, roughly 990,000 still go to V1 and about 10,000 to V2, and V2's decisions now count. While the metrics hold, the share climbs in steps (1%, 5%, 10%, 25%, 50%, then 100%). Each stage should track money lost to fraud, legitimate customers declined, the approval rate, latency, errors and timeouts, how many cases land in manual review, and infrastructure load.

Now suppose false declines jump when V2 reaches 10%:

V2 reaches 10%
       ↓
🚨 Guardrail triggered
       ↓
Stop rollout
       ↓
Rollback
       ↓
100% → V1

A canary answers whether V2 is safe enough to release progressively. Canary means controlling the blast radius.

A/B testing: let the data decide

Once both models run stably, the question changes. It's no longer only whether V2 is safe, but whether it actually produces better business outcomes. Eligible traffic is split at random:

Eligible Traffic
                    ↓
            Random Assignment
              ↙           ↘
            50%           50%
             ↓             ↓
        🔵 Model V1   🟣 Model V2
          Group A       Group B

Then the results get compared. The values below are illustrative only.

Metric	Model V1	Model V2
Fraud detection	92.0%	95.1%
False declines	1.8%	1.2%
Average latency	42 ms	51 ms
Manual review	4.5%	3.7%

A lesson hides in that table: the most accurate model isn't automatically the best one in production. V2 might improve fraud detection while noticeably raising latency, or lift recall while sending too many legitimate customers to manual review. What you need to judge is the whole business outcome.

Don't confuse the two methods. A canary asks whether V2 can be released safely. An A/B test asks whether V2 is really better than V1. In the latter, let the data pick the winner.

Multi-armed bandits: learn while optimizing

An A/B test usually holds its traffic allocation fixed for the duration. Bandits differ because allocation can change as the system learns. Picture three recommendation models, starting with an even split:

Model A → 33%
Model B → 33%
Model C → 34%

After collecting outcomes:

A → 3.0% conversion
B → 4.2% conversion 🏆
C → 2.7% conversion

The system may then shift traffic gradually:

Model A → 20%
Model B → 70% 🚀
Model C → 10%

Why not send everything to B? Because the system still has to learn. That's the tension between exploration, trying alternatives to find out whether something better exists, and exploitation, sending more traffic to the option currently earning the best reward.

Explore
   ↓
Learn
   ↓
Update Expected Reward
   ↓
Exploit
   ↓
Continue Learning

Common algorithms include epsilon-greedy, UCB and Thompson sampling. Bandits fit recommendations, search ranking, advertising, offer selection, content personalization and notification optimization. In short: learn, adapt, optimize.

Bandits for fraud decisions?

This one needs care. When a recommender tries out a weaker suggestion, at worst a customer skips it. When a fraud system tries out a weaker decision policy, a fraudulent transaction may be approved. Exploration costs very different amounts in the two cases, so the deployment strategy has to account for business risk, not just model quality. High-impact systems call for tighter guardrails and eligibility rules, human review, compliance controls, good observability and a way to roll back.

Putting it all together

A mature production ML path can look like this:

🧠 Train Model V2
        ↓
📊 Offline Evaluation
        ↓
🔵 Shadow Deployment
        ↓
🔍 Compare V1 vs V2
        ↓
🟢 Canary Release
        ↓
   1% → 5% → 10%
        ↓
🟣 A/B Testing
        ↓
📈 Validate Business KPIs
        ↓
🚀 Full Rollout
        ↓
👀 Continuous Monitoring
        ↓
🔄 Retraining / Model V3

For suitable optimization problems, it continues with:

Continuous Monitoring
        ↓
🟠 Multi-Armed Bandit
        ↓
Learn + Optimize

When something goes wrong

Imagine V2 reaches 25% of traffic and monitoring reports latency up 45%, timeouts up 3× and false declines up 2.1%. The production system should respond on its own:

🚨 Threshold Exceeded
        ↓
🛑 Stop Rollout
        ↓
↩️ Route Traffic to V1
        ↓
🔔 Alert ML / SRE Teams
        ↓
🔍 Investigate

That's why observability belongs to ML deployment from the start and isn't an afterthought. Three groups of metrics deserve tracking:

• Model metrics: how good the predictions are (precision, recall, F1, AUC, both kinds of error) and drift in the data
• System metrics: how the service holds up, covering latency, throughput, errors, timeouts, CPU, memory and uptime
• Business metrics: what the outcome is worth (conversion, revenue, fraud loss, declined customers, manual reviews, engagement)

A model can be statistically excellent and still fail operationally or commercially.

The same ideas for e-commerce recommendations

In a lower-risk setting the strategies look different:

• Shadow: V2 generates recommendations, but customers keep seeing V1.
• Canary: 5% of customers begin seeing V2's recommendations.
• A/B test: 50% get V1 and 50% get V2, and you compare CTR, conversion, average order value, revenue per session and retention.
• Bandit: traffic shifts toward strategies earning better rewards, with controlled exploration continuing.

The four strategies side by side

Strategy	Main question	Customer impact	Traffic
Shadow	How will V2 behave?	No	Mirrored
Canary	Is V2 safe?	Yes	Gradually increased
A/B	Is V2 better?	Yes	Fixed experiment split
Bandit	Which option should get more traffic now?	Yes	Dynamic

A quick memory aid: shadow watches, canary releases carefully, A/B compares, and a bandit keeps re-optimizing.

Risk management

Talk about AI and ML usually centers on algorithms, features, training, accuracy and GPUs. Production brings in one more dimension: risk management. The real question goes beyond whether a model predicts well. It's whether you can introduce it safely, watch it, measure what it does, keep the blast radius small, catch failures and roll it back. A successful production ML platform is more than a model behind an API. It combines models, data, infrastructure, experimentation, observability, guardrails, rollback and business metrics. Training readies a model, and production engineering decides whether it deserves trust.