Batch vs. Streaming: Choosing a Data Pipeline Architecture You Won't Regret

Introduction

Every data platform discussion eventually produces the sentence "we need this in real time." Sometimes it's true. More often, "real time" means "the dashboard felt stale once," and acting on it unexamined commits the team to the most operationally expensive architecture in data engineering.

Having built both, Airflow-orchestrated batch pipelines and streaming ingestion on AWS, the question I now force before any architecture diagram: what is the actual freshness SLA, and who pays for missing it?

Start From Freshness, Not Technology

Put every consumer of the data on this scale:

• Sub-second to seconds: fraud checks, live personalization, operational alerting. Genuine streaming territory: the value of the data decays in seconds.
• Minutes: "operational dashboards," most alerting on business metrics. Micro-batch (a job every 1-5 minutes) usually serves this at a fraction of streaming's cost.
• Hours to daily: reporting, ML training sets, finance, anything a human reads in a meeting. Batch. Full stop.

The mistake pattern is uniform: one genuinely-streaming use case appears, and suddenly everything migrates to the streaming stack "for consistency", including the daily revenue report that now flows through Kafka, Flink, and a lakehouse merge to arrive exactly as often as a human looks at it.

What Batch Buys You

Batch pipelines (files in, transform, tables out, on a schedule) have virtues that only become visible after you've operated their absence:

• Retryability is trivial. A failed run re-runs. Airflow retries the task; the input still exists, immutable, exactly as it was.
• Backfills are first-class. Reprocess last quarter with a date-range parameter. In streaming, "reprocess history" is a project, not a parameter.
• Correctness is checkable. Row counts, sums, dbt tests, you validate a completed dataset before publishing it. A stream is never complete, so every check is provisional.
• Cost is bounded and visible. Compute spins up, runs, stops. Streaming infrastructure runs, and bills, around the clock, matched to peak.

The cost, obviously, is latency: data is at best as fresh as the schedule.

What Streaming Actually Costs

The streaming tax isn't writing consumer code, Kafka/Kinesis client libraries are fine. It's the semantics you inherit:

• Late and out-of-order events. An event about 09:59 can arrive at 10:07. Every windowed computation now needs watermarks and a lateness policy, and "the numbers changed after we reported them" becomes a conversation with stakeholders.
• Exactly-once is a discipline, not a checkbox. Delivery guarantees are at-least-once by default; deduplication has to be designed, idempotent writes keyed on event IDs, transactional sinks, or offset-coupled commits. (The same idempotency patterns that protect APIs protect sinks.)
• State is your problem. Aggregations hold state; state needs checkpointing; checkpoint recovery needs testing. A stateful stream job is a distributed database wearing a job's clothing.
• Schema evolution under fire. Producers deploy a new event shape while consumers are mid-stream; without a schema registry and compatibility rules, you find out in production.
• Operational surface. Consumer lag monitoring, partition rebalancing, poison-pill messages, replay procedures. This is a 24/7 system with paging semantics.

None of this is a reason to avoid streaming, it's the price list, and genuine sub-minute use cases are worth paying it for. The failure mode is paying it for a daily report.

The Hybrid That Usually Wins

Most mature platforms converge on the same shape:

1. Ingest as a stream, land immutably. Events flow through Kafka/Kinesis into cheap immutable storage (S3, a lakehouse table) via a dumb, stateless lander, Kinesis Firehose being the canonical zero-ops example. Streaming ingestion, batch semantics downstream.
2. Serve the few genuinely-real-time consumers from the stream, the fraud model, the alerting rule, as narrow, purpose-built consumers with explicit SLAs.
3. Everything else reads the landed tables in batch, Airflow/dbt on minutes-to-daily schedules, with tests, backfills, and bounded cost.

This gets stream-grade capture (you never lose the ability to be real-time later, because the raw events are all there) without stream-grade processing everywhere. Late data self-heals in the next batch window. And when a new "we need it in real time" request arrives, the marginal cost is one new consumer, after the freshness interrogation, which usually reveals that a 5-minute micro-batch was the actual requirement.

On AWS specifically: Kinesis Data Streams + Firehose → S3/Iceberg + Athena/Glue for the batch side is a remarkably low-ops version of this whole diagram; Kafka (MSK) earns its extra operational weight when you need its ecosystem (connect, streams, exactly-once transactions) or multi-consumer replay at scale.

A Decision Checklist

Before committing to streaming for a use case, get written answers to:

• What is the freshness SLA, in numbers, and what breaks if it's missed by 10×?
• Who owns late/out-of-order handling, and what does "the numbers moved" mean for this consumer?
• What's the replay story when (not if) a consumer bug corrupts a week of output?
• What does this cost per month at 2 a.m. traffic levels, and who gets paged?

If the answers are shrugs, land the events and run a batch job. You can always tighten a 5-minute batch. Untangling an unnecessary streaming architecture is much harder.

Takeaways

• Classify every consumer by freshness SLA; architect per consumer, not per fashion.
• Batch's superpowers, trivial retries, first-class backfills, checkable correctness, are worth protecting.
• Streaming's real cost is semantic (lateness, exactly-once, state), not syntactic.
• Ingest-as-stream + land-immutably + process-mostly-in-batch serves most platforms best, and keeps the door open for true real-time consumers later.