Processing 100,000+ Security Events per Day

I worked on a security platform that processed more than 100,000 events every day. That’s a useful number on a résumé, but what changed my thinking were the small failures the volume made impossible to ignore. A dependency will eventually respond slowly. A message may arrive more than once. An event can be delayed or reach a worker in an unexpected order. A deployment can happen while processing is still in progress. None of this was unusual. It was just what a busy system looked like on a normal day.

The Happy Path

The platform received security events, enriched them with additional information, applied processing rules, and made the results available to other parts of the product. The main path was easy to describe: receive an event, process it, save the result, and continue. The difficult questions appeared around that path. What should happen if enrichment data is temporarily unavailable? Can the event be processed again safely? How do we distinguish work that’s delayed from work that’s genuinely stuck? If one stage succeeds and the next fails, where should processing resume? We spent more time on these questions than on the happy path, and that made sense for this kind of system.

Retries

Retries are useful, but an uncontrolled retry can make an incident worse. If a dependency is already struggling, immediately sending the same request repeatedly adds more pressure. If an operation isn’t safe to repeat, retrying can create duplicate effects. If retries happen without enough context, the team may know that something failed but not which stage completed successfully. We treated recovery as part of the processing flow. That meant making important operations safe to repeat, limiting retries, preserving enough state to understand progress, and exposing failures that required attention. The flow used .NET workers, Apache Flink, AWS services and several data stores. These tools moved and stored the events, but we still had to define what safe processing meant for our system. In particular, we had to decide what “processed” meant when only part of the work succeeded.

Logs and Metrics

Distributed processing also changed what I wanted from logs and metrics. A log saying “event processing failed” was rarely enough. We needed to connect activity across components and answer more practical questions:

• Which event was affected?
• Which stages had already completed?
• Was this the first attempt or a retry?
• Was the failure isolated or part of a wider pattern?
• Could the system recover automatically?

Good observability didn’t prevent failures, but it stopped us from spending the first part of an investigation trying to work out which event everyone was talking about. Earlier in my career, I thought about retries, validation and logging after the main flow was working. On this platform, these things were mixed with the main work every day. Events kept arriving while we investigated, a small failure could create a backlog and some of the cases we called rare happened several times before the day was over. A lot of our discussions were simply about which failures the platform could recover from alone and which ones needed somebody to look at them. At that volume, this wasn’t a special reliability project. It was normal backend work for the platform.