Firing the Firehose: replacing a managed AWS service with a stateful Lambda

Let's dive in deep

1This post is part of In Deep, a series where our engineers walk through a project in technical detail. 
2
3It's for:
4/Backend and cloud engineers working with AWS
5/Anyone with a streaming pipeline and a growing bill
6/People who like a good edge case

What Firehose was doing for us

The setup is simple. IoT devices send ZSTD-compressed data to Amazon Kinesis Data Streams. Firehose reads from the stream, calls a Lambda to decompress each batch, and writes the result to Amazon S3. From there it goes to an external data lake. A second Lambda handles retries when a device fails to put a record on the stream.

Firehose does more than move data to S3. It batched small records into large files, delivered them reliably, and ran without anyone looking after it.

The limits we had to work within were fixed. A Kinesis shard handles at most 1,000 records or 1 MB per second. Traffic runs around the clock, so there's no room for backpressure. And zero data loss was not up for discussion.

Without Firehose, we'd have to handle all three of those jobs ourselves.

The obvious solution

Write a Lambda that reads a batch from the stream, decompresses the records and writes a file to S3. That's about fifteen lines, one function, stateless and cheap.

The problem is the files. A stateless function writes whatever it has and then forgets about it. Millions of invocations produce millions of tiny files.

The small-files problem

S3 charges $0.005 per 1,000 PUT requests. That's $5 per million, and at our volume we'd hit a million in roughly a minute. It's also a performance problem. Query engines slow down on tiny files because they spend more time on metadata than on data. Listing, partitioning and lifecycle rules get messy across millions of objects.

This brings back the problem Firehose was solving for us. To buffer thousands of records over many invocations into one large file, the function needs to remember things. Lambdas don't.

Making a stateless Lambda stateful

The usual answer is to add an external store like DynamoDB: read state at the start of each invocation and write it back at the end. We decided against it. It's another resource to provision, secure, pay for and monitor. It puts two network calls on the hot path, thousands of times a second. We'd have to manage keys and clean up stale state. And we'd bring back locking and race conditions we would rather avoid.

Lambda has a built-in option for this: tumbling windows.

Tumbling windows

When Lambda reads from Kinesis through an event source mapping (the config that wires a stream to a function), you can configure a tumbling window. Within that window, each invocation can return a state object. We shape it:: whatever JSON we hand back in the response, Lambda stores and passes unchanged into the next invocation for that same shard, as input. It's the only thing that survives between invocations in the same window. We used

60 seconds.

Each invocation receives up to 1,000 records, or whatever arrived in the last five seconds. It does its work and hands the state forward. When the window closes, Lambda calls the function one last time with no records. That's the signal to close the file. The next window starts fresh.

Store a pointer, not the payload

The obvious next idea is to keep the records in the state. That doesn't work. State is capped at 1 MB, and each window moves far more data than that. If the state goes over 1 MB, AWS ends the window early.

So the state holds a reference to the data instead of the data itself. The data itself goes into an open S3 multipart upload (a way of building one large file out of several separately-uploaded chunks, explained in detail below) and the state holds just a few hundred bytes describing where that upload is:

• Bucket: where the file lives
• Key: the filename, fixed when the upload opens
• UploadId: the handle for this multipart upload
• PartNumber: a counter, because S3 needs unique, ordered parts
• CompletedParts: the ETag and number of every part uploaded so far

This is serialised as JSON and passed forward in event.State. If no upload is in progress, the state is empty.

One file, many uploads

S3 multipart uploads fit this pattern well. CreateMultipartUpload opens the file and returns an upload ID. Each invocation in the window calls UploadPart with its own chunk. On the final invocation, CompleteMultipartUpload joins all parts into a single object.

‍

There's one constraint: every part must be at least 5 MB, except the last one. The function therefore writes either a direct upload or a multipart upload, depending on how much data a window holds.

The real architecture is eight lines of config

Most of the behaviour is set in the event source mapping, not in the code. These are the eight settings that matter:

# Batching and windowing
tumbling_window_in_seconds          = 60      # a 60-second memory across invocations
batch_size                          = 1000    # up to 1,000 records per invocation
maximum_batching_window_in_seconds  = 5       # don't wait more than 5s to fill a batch
parallelization_factor              = 1       # one Lambda per shard, no conflicts

# Durability and retries
maximum_retry_attempts         = -1       # never give up on a failed batch
maximum_record_age_in_seconds  = -1       # never drop a record for being old
function_response_types        = ["ReportBatchItemFailures"]  # retry only what failed
starting_position              = "LATEST"  # on first start, begin at the live edge

Never leave a file half-open

A window can close in two ways. In a normal close, the window ends, the empty final invocation arrives and we complete the upload. In a forced close, the state goes over 1 MB, AWS resets it, and we complete the upload straight away.

A multipart upload that is never completed or aborted becomes an orphan. It stays in S3, you can't see it in a normal listing, and you keep paying for it.

Zero data loss

The checkpoint only moves when the data is safe

Kinesis delivers at least once. We never let the checkpoint move past a record until that record is stored durably in S3. When something fails, we report the failed items, and Lambda retries from that point in a new invocation. Retries are unlimited and no record expires, so it keeps retrying for as long as it takes.

Failing cleanly

Lambda has a hard timeout of 300 seconds. If it kills the process while an S3 write is in progress, we can't be sure what happened. So we stop ourselves 500 ms early (400 ms reserved for flushing telemetry, 100 ms for returning the response), so any S3 calls still running fail cleanly and get retried instead of being cut off.

A Go recover() catches unexpected panics and returns a batch failure. Every abnormal exit ends up on the same path: fail cleanly and let the event source mapping retry.

Skipping a record

Not every failure should be retried. A corrupt record that can never be decoded would block the shard forever. We skip these records so one bad message doesn't block the whole shard. They're logged with their sequence number and partition key, so none of them disappear and each one can be investigated.

Transient errors such as S3 throttling or network problems are different. Those fail the batch and get retried indefinitely.

When retrying forever is the wrong answer

A third type of failure needs a different approach. Sometimes the state points to a multipart upload that no longer exists. If you retry with the same state, the shard gets stuck in a loop.

‍

In practice:

• NoSuchUpload on UploadPart: the upload is gone, so we drop the pointer and process the batch as new.
• NoSuchUpload on CompleteMultipartUpload: an earlier complete call already succeeded, so we treat it as a success.
• InvalidPart on CompleteMultipartUpload: here it gets more interesting. The part list we sent doesn't match what S3 actually holds for that upload (either a part never made it, or its ETag disagrees with what we remember). Rather than guess, we ask S3 directly: a call to ListParts returns its authoritative list. If every part number we expected is present, even under a different ETag, the bytes are safe, so we retry CompleteMultipartUpload using S3's own list instead of ours. If a part number is genuinely missing, those bytes were never durably stored and this invocation no longer has them, so it falls back to the same retry-forever path as a transient error: state preserved, batch retried.

‍

NoSuchUpload and InvalidPart happen in production sometimes. When they do, the function recovers by itself, without data loss and without paging anyone. The deciding move in both cases is the same: never guess what's true in S3, ask it."

The decisions that surprised us

Two things we planned to build turned out to be unnecessary.

We didn't need a graceful shutdown handler

A shutdown handler would protect us in two situations, and both were already covered. When a deploy or environment recycle happens, the upload state lives in the event source mapping, so the new environment carries on with it. When the process is killed mid-invocation, the 500 ms deadline already gets us out cleanly. SIGTERM also only fires between invocations, never during one.

A handler would also have caused problems. It would complete the upload while the event source mapping still refers to it, which leads to a NoSuchUpload error that gets retried forever.

The state is a database schema

The state is serialised JSON that survives a deploy. That makes it a wire format, and changing it needs the same care as a database migration. Adding a field is fine, because old state just gets a zero value. Removing a field is fine, because the old value is ignored. Renaming a field is not: rename UploadId to UploadID and every upload in progress loses its pointer and fails with NoSuchUpload. We considered adding a version field to the state for this reason.

Did it work?

The pipeline is now shorter. Devices send data to Kinesis Data Streams. A single Lambda, the S3 sink, reads the compressed messages, decompresses them and writes JSONL to S3 through a direct or multipart upload. The data lake gets the same large files it always did.

Closing

Replacing a managed service makes sense when your volume is high and steady enough for per-unit pricing to dominate the bill. It also helps if the service's features are simpler than they look, so you can solve its hidden problems yourself, and if your team is willing to own durability and edge cases. If your traffic is spiky or low, the managed service will usually win. The same goes for cases where the hidden problems are harder than the cost savings justify.

Firehose did three jobs for us, not one: it buffered small records into large files, it delivered them reliably, and it operated itself without anyone looking after it. We had to solve all three before the switch paid off.

‍