Blue/Green Deployments for ML Models: Zero-Downtime Retraining in Practice

Introduction

While building a music recommendation system (spotify-mlops, content-based filtering over Spotify's 114k-track dataset), I hit the problem every ML system eventually hits: the model needs to be retrained regularly, but users are querying it right now.

The naive approach, stop the server, load the new model, start again, means downtime on every retrain. The slightly-less-naive approach, reload the model file in place, risks serving requests mid-load with a half-initialized model. What I actually wanted was what infrastructure engineers have had for years: blue/green deployment, applied at the model level instead of the server level.

This post walks through the pattern I ended up with, the mistakes along the way, and when you should (and shouldn't) use it instead of a managed solution.

The Core Idea

Classic blue/green deployment runs two identical environments, "blue" (live) and "green" (idle), and switches traffic between them at the load balancer. For a single-service ML API, you can collapse the same idea into the process itself:

• The active model serves every prediction request.
• A candidate model is trained and loaded in the background, in the same process or a sidecar.
• When the candidate passes validation, you swap a single reference. New requests hit the new model; in-flight requests finish on the old one.
• The old model sticks around in memory until you're confident, making rollback instant.

The swap is the entire trick, and in Python it can be genuinely atomic because attribute assignment is a single bytecode operation:

class ModelRegistry:
    """Holds the active model behind a single swappable reference."""

    def __init__(self, initial_model):
        self.active = initial_model     # blue
        self.previous = None            # rollback target

    def promote(self, candidate):
        """Candidate becomes active; old active is kept for rollback."""
        self.previous = self.active
        self.active = candidate         # the atomic swap

    def rollback(self):
        if self.previous is None:
            raise RuntimeError("No previous model to roll back to")
        self.active, self.previous = self.previous, self.active

Request handlers only ever read registry.active, so they see either the old model or the new one, never an in-between state:

from fastapi import FastAPI

app = FastAPI()

@app.post("/recommend")
def recommend(query: RecommendationQuery):
    model = registry.active          # grab the reference once per request
    return model.recommend(query.track_features, k=query.k)

One subtle rule: grab the reference once at the top of the request and use that local variable throughout. If you read registry.active twice in one request, a swap between the two reads can hand you features from one model and scores from another.

Retraining in the Background

The retrain job runs on a schedule (or on demand) and never touches the active model:

import threading

def retrain_and_promote(registry, feedback_path="data/feedback_data.csv"):
    # 1. Build the candidate completely off to the side
    dataset = load_base_dataset()
    feedback = load_feedback(feedback_path)     # user likes since last train
    candidate = train_knn_model(dataset, feedback)

    # 2. Warm-up validation BEFORE it sees traffic
    checks = validate(candidate)
    if not checks.passed:
        log.error("Candidate rejected: %s", checks.failures)
        return

    # 3. Atomic promotion
    registry.promote(candidate)
    log.info("Candidate promoted to active")

threading.Thread(target=retrain_and_promote, args=(registry,), daemon=True).start()

The feedback loop matters here: user "likes" get appended to a simple CSV during serving, and each retrain folds them in. It's the least glamorous possible feature store, and for a project of this size it's exactly enough.

Validation: The Step I Skipped First (and Regretted)

My first version promoted every candidate that trained without raising an exception. Then one retrain ran against a partially-written feedback file, produced a model with a subtly different feature ordering, and started returning recommendations that were valid-looking but wrong, high-confidence nonsense, served with zero errors in the logs.

Now the warm-up gate checks three things before any swap:

• Contract: the candidate answers a fixed set of smoke-test queries with the right shapes, dtypes, and value ranges.
• Sanity: for a handful of known tracks, the top-N recommendations overlap meaningfully with the previous model's (wild divergence usually means a data bug, not a better model).
• Latency: p95 over the smoke set stays within budget, a model that's 10× slower is a regression even if it's more accurate.

def validate(candidate) -> ValidationResult:
    failures = []
    for query, expectation in SMOKE_QUERIES:
        result = candidate.recommend(query, k=10)
        if len(result) != 10:
            failures.append(f"wrong result count for {query!r}")
        if any(score < 0 or score > 1 for _, score in result):
            failures.append(f"score out of range for {query!r}")
    return ValidationResult(passed=not failures, failures=failures)

None of this measures whether the new model is better, that's what offline evaluation and A/B tests are for. The gate's only job is to guarantee the new model isn't broken. Those are different questions, and conflating them is how broken models reach production while teams debate metrics.

Rollback Is a Feature, Not an Apology

Because the previous model stays in memory, rollback is the same one-line swap in reverse, no artifact download, no restart, no redeploy. I wired it to an admin endpoint:

@app.post("/admin/rollback")
def rollback():
    registry.rollback()
    return {"active_model": registry.active.version}

The first time a bad candidate slipped past validation, recovery took the time it took to curl that endpoint. Compare that with "re-run the deploy pipeline with the old artifact", the difference is minutes versus seconds, and at 2 a.m. that difference is everything.

The cost is memory: you're holding two models. For a KNN index over 114k tracks that's trivial. For a multi-gigabyte transformer it isn't, and that's one of the signals you've outgrown this pattern.

When to Use This, and When Not To

In-process blue/green is the right size when:

• You run a single service (or a few replicas) serving one model
• Models are small enough to hold two copies in memory
• Retraining is frequent enough that restart-per-deploy hurts
• You want instant rollback without extra infrastructure

You've outgrown it when traffic needs gradual shifting (canary percentages), when models are too big to double-load, or when multiple services consume the same model. At that point the same concept lives one level up: SageMaker endpoints support blue/green traffic shifting with configurable guardrails, Kubernetes gives you canary rollouts, and a model registry replaces the in-process reference. The mental model transfers unchanged (active, candidate, validate, promote, rollback); you're just renting the machinery instead of writing it.

Takeaways

• Serve every request through one swappable reference, read once per request.
• Train candidates completely off the serving path; promotion is a single atomic assignment.
• Gate promotion on a warm-up validation that checks the model is functional, separate from whether it's better.
• Keep the previous model in memory so rollback is instant and boring.
• Reach for managed blue/green (SageMaker, canary rollouts) when models get big or traffic needs gradual shifting, the pattern survives the migration.

The whole implementation is a few dozen lines on top of FastAPI and scikit-learn, and it turned retraining from a scheduled maintenance event into a non-event. That's the goal: deployments so uneventful nobody notices them.