Keeping a Real-Time Processing Pipeline Understandable

On a security platform, I helped build processing flows for a continuous stream of real-time events. The architecture needed some complexity. Events had to be received, transformed, enriched with other information, saved and made available to other features. We used processing workers, Apache Flink, AWS services, databases and caching for different parts of the flow. There was no realistic version of this system with one component and no moving parts. The useful question was how many moving parts we could clearly justify.

The Cost of Each Stage

Adding another processing step looks simple in a diagram. In production, that new box needs configuration, monitoring, capacity and a recovery strategy. The connections also need contracts and a plan for delayed, duplicated or unavailable communication. If the stage doesn’t create a clear capability or boundary, it may add more failure modes than value. We tried to keep transformations close to the part of the pipeline that owned them and avoid moving the same event through unnecessary formats. When data crossed a boundary, the reason needed to be understandable. This made the flow easier to trace and reduced the number of places where an event could become stuck without a clear owner.

We removed one intermediate transformation after finding that it only changed field names before another stage converted the data again. Each conversion had looked small when it was added, but together they made it harder to compare a processed event with the original one. Keeping one clear internal shape through more of the pipeline reduced code and made production examples easier to follow.

We also avoided creating a separate service only because a processing function was complex. In some cases, a module inside an existing worker gave us enough separation without another deployment and queue. We introduced a new component when it needed different scaling, ownership or failure behavior rather than for a cleaner box in the diagram.

Timing Across the Pipeline

Real-time processing made me more cautious about abstractions that make work look immediate when it’s not. A method called processEvent may publish a message that waits in a queue, triggers enrichment, retries a dependency, and updates a result later. The method name is simple. The behavior is distributed across time. If the application hides all of that, engineers can make incorrect assumptions about completion. One service may treat the event as finished while another has only accepted it. We needed names and states that reflected what the system had actually guaranteed: received, scheduled, enriched, completed, or failed. We needed the code and the states to show that the workflow was asynchronous instead of making it look like a local function call.

Those states were used outside the engineering team as well. Support needed to know if an event was still moving normally or if it had reached a point that required attention. We kept the user-facing states smaller than the technical pipeline, but they were connected to real processing stages. Otherwise, a generic “processing” status could remain on the screen long after the backend had stopped trying.

Event schemas also had to change without assuming that every producer and consumer would deploy together. We added fields in compatible ways, kept consumers tolerant during the transition and only removed old behavior after the pending events and older producers were no longer using it. This added some temporary code, but it was easier to understand than a deployment where the same event changed meaning halfway through the pipeline.

When the Backlog Started Growing

When processing is healthy, even a complicated pipeline can look simple. Events go in, results come out and nobody asks many questions. When one dependency slows down, work starts accumulating, memory or connection usage changes and retries add more load. At that point we need to find which stage owns the backlog, whether processing can continue without the unavailable enrichment and whether adding capacity would only overload the next dependency. Fewer hidden interactions make those questions easier to answer. We still used specialized tools where they made sense. Apache Flink handled event transformation and orchestration, PostgreSQL and DynamoDB served different data needs, and Redis/Valkey helped with fast access and coordination. Each one had a specific job in the flow.

During one slowdown, increasing worker capacity would have moved more requests into a dependency that was already taking longer to respond. We limited the work entering that stage, allowed the previous queue to absorb part of the traffic and watched the age of the events while the dependency recovered. The pipeline processed more slowly for a period, but it stayed understandable and didn’t turn the slowdown into a wider failure.

After incidents like that, we reviewed both the capacity and the path an engineer had to follow to explain what happened. If several dashboards and log searches were needed only to discover which stage owned the backlog, we improved the identifiers or metrics around that boundary. Reducing investigation time was part of keeping the system manageable even when the number of components couldn’t be reduced.

The difficult moments came when one stage slowed down and events continued arriving. We needed to know who owned the backlog, what that stage was waiting for and whether increasing capacity would only move the problem to the next dependency.