From Notebook to Production: Deploying ML Models on AWS SageMaker

Introduction

Training a model is the part everyone talks about. Getting that model in front of real traffic, reliably, cheaply, and observably, is the part that decides whether the project ships.

This guide covers the deployment half of the ML lifecycle on AWS SageMaker: how to choose between the four inference modes, how to package a model correctly, how to promote it through a model registry, and how to configure autoscaling and monitoring so the endpoint survives contact with production. Everything here maps to what the AWS Machine Learning Engineer - Associate exam calls "deployment and orchestration," but the focus is practical, not exam trivia.

The Four Inference Modes (and How to Choose)

SageMaker gives you four ways to serve predictions, and picking the wrong one is the most expensive mistake you can make early.

Real-Time Endpoints

A managed HTTPS endpoint backed by instances you choose, always on, answering synchronous requests in milliseconds. This is the default choice for user-facing features, recommendations, fraud checks, search ranking.

• Latency: single-digit to low hundreds of milliseconds
• Cost model: you pay for the instances 24/7, whether or not traffic arrives
• Payload limit: 6 MB per request, 60-second timeout

Serverless Inference

The same synchronous API, but AWS manages capacity and scales to zero. You pay per request and per compute-second. This is the right default for spiky or low-volume traffic, internal tools, early-stage products, anything under a few requests per second on average.

The trade-off is cold starts: after idle periods, the first request can take seconds while a container spins up. If your p99 latency budget can't absorb that, provision concurrency or go back to real-time instances.

Asynchronous Inference

Requests go into an internal queue; responses land in S3, with an optional SNS notification. Payloads can be up to 1 GB and processing can take up to one hour. Use it for large inputs (video, long documents) or slow models where the caller doesn't need an answer within the HTTP request. Async endpoints can also scale to zero between bursts.

Batch Transform

No endpoint at all. You point SageMaker at a dataset in S3, it spins up instances, scores everything, writes results back to S3, and tears the fleet down. If predictions are consumed on a schedule, nightly propensity scores, weekly churn lists, batch is almost always the cheapest and simplest answer.

A useful decision rule: start from the consumer of the prediction, not the model. If nobody reads the prediction within seconds of it being made, you probably don't need a real-time endpoint.

Packaging the Model

Whatever mode you choose, SageMaker expects the same artifact: a model.tar.gz in S3 containing the model weights and, for framework containers, an inference script.

model.tar.gz
├── model.joblib          # your serialized model
└── code/
    ├── inference.py      # how to load it and handle requests
    └── requirements.txt  # extra pip dependencies

The inference script implements up to four functions. For the scikit-learn, PyTorch, and XGBoost containers the contract looks like this:

# code/inference.py
import json
import joblib
import os

def model_fn(model_dir):
    """Called once when the container starts. Load and return the model."""
    return joblib.load(os.path.join(model_dir, "model.joblib"))

def input_fn(request_body, request_content_type):
    """Deserialize the request payload into model input."""
    if request_content_type == "application/json":
        payload = json.loads(request_body)
        return payload["instances"]
    raise ValueError(f"Unsupported content type: {request_content_type}")

def predict_fn(input_data, model):
    """Run inference."""
    return model.predict_proba(input_data)[:, 1].tolist()

def output_fn(prediction, accept):
    """Serialize the response."""
    return json.dumps({"scores": prediction}), "application/json"

Two hard-won rules here:

• Do heavy work in model_fn, never in predict_fn. model_fn runs once per container; predict_fn runs on every request.
• Validate inputs in input_fn and fail loudly. A model that silently receives features in the wrong order will happily return confident nonsense for months.

Deploying with the SageMaker Python SDK

With the artifact in S3, deployment is a few lines:

from sagemaker.sklearn import SKLearnModel

model = SKLearnModel(
    model_data="s3://my-bucket/models/churn/model.tar.gz",
    role=execution_role_arn,
    entry_point="inference.py",
    source_dir="code",
    framework_version="1.2-1",
)

predictor = model.deploy(
    initial_instance_count=2,          # two instances = two AZs = no single point of failure
    instance_type="ml.c6i.xlarge",
    endpoint_name="churn-scorer-prod",
)

Note initial_instance_count=2. A single-instance endpoint has no availability story: an instance failure or an AZ event takes the whole endpoint down. Two instances get spread across Availability Zones automatically.

For serverless, swap the config:

from sagemaker.serverless import ServerlessInferenceConfig

predictor = model.deploy(
    serverless_inference_config=ServerlessInferenceConfig(
        memory_size_in_mb=3072,
        max_concurrency=20,
    ),
    endpoint_name="churn-scorer-serverless",
)

The Model Registry: Stop Deploying from Notebooks

The single biggest MLOps upgrade for a small team is refusing to deploy anything that isn't registered. The SageMaker Model Registry gives you versioned model packages with an approval status, which becomes the gate between "a data scientist liked this run" and "this is in production."

model_package = model.register(
    model_package_group_name="churn-scorer",
    content_types=["application/json"],
    response_types=["application/json"],
    inference_instances=["ml.c6i.xlarge"],
    approval_status="PendingManualApproval",
)

The production deployment pipeline then only ever deploys the latest package whose status is Approved. Flipping a bad model back to Rejected and re-running the pipeline is your rollback mechanism, no notebook required, fully auditable.

This pairs naturally with a CI/CD flow: a training pipeline registers candidates, a human (or an automated evaluation against a golden dataset) approves, and an EventBridge rule triggers the deployment pipeline on approval-status change.

Autoscaling That Actually Works

Real-time endpoints scale with Application Auto Scaling on the InvocationsPerInstance metric. The pattern:

import boto3

client = boto3.client("application-autoscaling")
resource_id = "endpoint/churn-scorer-prod/variant/AllTraffic"

client.register_scalable_target(
    ServiceNamespace="sagemaker",
    ResourceId=resource_id,
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    MinCapacity=2,
    MaxCapacity=10,
)

client.put_scaling_policy(
    PolicyName="invocations-target-tracking",
    ServiceNamespace="sagemaker",
    ResourceId=resource_id,
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    PolicyType="TargetTrackingScaling",
    TargetTrackingScalingPolicyConfiguration={
        "TargetValue": 700.0,   # invocations per instance per minute
        "PredefinedMetricSpecification": {
            "PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"
        },
        "ScaleInCooldown": 300,
        "ScaleOutCooldown": 60,
    },
)

How to pick TargetValue: load-test one instance, find the invocations-per-minute where latency starts degrading, and set the target at 50-70% of that. Scale out fast (short cooldown), scale in slow, flapping instance counts hurt more than a few idle minutes.

Monitoring: The Part Everyone Skips

An ML endpoint can be perfectly healthy at the infrastructure level and completely wrong at the prediction level. You need both layers.

Infrastructure layer, CloudWatch gives you ModelLatency, OverheadLatency, Invocation4XXErrors, Invocation5XXErrors, and instance utilization out of the box. Alarm on p99 ModelLatency and on any sustained 5XX rate.

Prediction layer, enable Data Capture so the endpoint logs requests and responses to S3:

from sagemaker.model_monitor import DataCaptureConfig

predictor = model.deploy(
    initial_instance_count=2,
    instance_type="ml.c6i.xlarge",
    endpoint_name="churn-scorer-prod",
    data_capture_config=DataCaptureConfig(
        enable_capture=True,
        sampling_percentage=20,
        destination_s3_uri="s3://my-bucket/datacapture/churn-scorer",
    ),
)

On top of captured data, SageMaker Model Monitor can run scheduled jobs that compare live feature distributions against a training-time baseline and raise violations on drift. Even if you never automate a response, a weekly drift report is the difference between discovering a broken upstream feature in days versus quarters.

A Deployment Checklist

Before you call an endpoint "production":

• At least 2 instances (real-time) or tested cold-start behavior (serverless)
• Model artifact deployed from the registry, not from a notebook
• Autoscaling configured and load-tested past expected peak
• CloudWatch alarms on p99 latency and 5XX errors, wired to a human
• Data Capture enabled with a drift baseline
• A rollback path you have actually rehearsed
• Cost review: would batch or serverless serve this traffic for less?

Closing Thoughts

The gap between a model that works and a model that ships is mostly engineering, and SageMaker's value is that each piece, packaging, registry, scaling, monitoring, is a managed primitive you compose rather than infrastructure you build. Start with the consumer of the prediction, pick the cheapest inference mode that meets that consumer's latency needs, and put the registry between your notebooks and your users. Your future self, paged at 3 a.m., will thank you.