Keeping a Real-Time Processing Pipeline Understandable

On a security platform, I helped build processing flows for a continuous stream of real-time events. The architecture needed some complexity. Events had to be received, transformed, enriched with other information, saved and made available to other features. We used processing workers, Apache Flink, AWS services, databases and caching for different parts of the flow. There was no realistic version of this system with one component and no moving parts. The useful question was how many moving parts we could clearly justify.

The Cost of Each Stage

Adding another processing step looks simple in a diagram. In production, that new box needs configuration, monitoring, capacity and a recovery strategy. The connections also need contracts and a plan for delayed, duplicated or unavailable communication. If the stage doesn’t create a clear capability or boundary, it may add more failure modes than value. We tried to keep transformations close to the part of the pipeline that owned them and avoid moving the same event through unnecessary formats. When data crossed a boundary, the reason needed to be understandable. This made the flow easier to trace and reduced the number of places where an event could become stuck without a clear owner.

Timing Across the Pipeline

Real-time processing made me more cautious about abstractions that make work look immediate when it’s not. A method called processEvent may publish a message that waits in a queue, triggers enrichment, retries a dependency, and updates a result later. The method name is simple. The behavior is distributed across time. If the application hides all of that, engineers can make incorrect assumptions about completion. One service may treat the event as finished while another has only accepted it. We needed names and states that reflected what the system had actually guaranteed: received, scheduled, enriched, completed, or failed. We needed the code and the states to show that the workflow was asynchronous instead of making it look like a local function call.

When the Backlog Started Growing

When processing is healthy, even a complicated pipeline can look simple. Events go in, results come out and nobody asks many questions. When one dependency slows down, the internal relationships become visible. Work starts accumulating. Memory or connection usage changes. Retries create additional load. A slow stage affects everything behind it. A simpler flow is easier to reason about in that moment. Which stage owns the backlog? Can processing continue without the unavailable enrichment? How old is the oldest event? Can we increase capacity safely, or would that overload the next dependency? The fewer hidden interactions the system has, the faster the team can answer those questions. Sometimes the clearest design uses specialized tools. Apache Flink was appropriate for event transformation and orchestration in our platform. PostgreSQL and DynamoDB served different data needs. Redis/Valkey helped with fast access and coordination. Simplicity didn’t mean forcing every workload into one technology. It meant giving each component a specific responsibility and resisting extra layers that didn’t improve the system’s behavior.

The number of services wasn’t really what made the pipeline simple or complex for us. The difficult moments came when one stage slowed down and events continued arriving. Then we had to find who owned the backlog, what the stage was waiting for and whether increasing capacity would only move the problem to the next dependency. I still like modularity and specialized tools, but this project made me much more interested in whether the team could explain the complete flow when it wasn’t moving normally.