Processing 100,000+ Security Events per Day

I currently work on a security platform that processes more than 100,000 events every day. It’s a good number to put on a résumé, but the volume became more real to me through the small problems that happened while the events were moving across the system. A dependency responded slowly, a message arrived more than once, an event took longer than expected in one stage or a deployment happened while there was still work being processed.

None of these situations happened all the time, but with this amount of events they happened often enough that we couldn’t treat them as very unusual cases. The system continued receiving new work while we investigated the previous one, so even a small failure could create a backlog if the platform didn’t know how to recover from it.

The Normal Flow

The platform received security events, enriched them with additional information, applied processing rules and made the results available to other parts of the product. The normal flow was easy to explain, but an event could go through several components before it was finished. We used .NET workers, Apache Flink, AWS services and different data stores, and each part had its own idea of what had already been processed.

A lot of our discussions happened when one of these stages completed and the next one didn’t. We needed to know if the event could start again, if it should continue from a specific point or if retrying would duplicate something that had already happened. “Processed” wasn’t always a yes or no state because the event could be valid and partially enriched while still waiting for another dependency.

The order also mattered for some types of events. Two messages about the same asset could arrive close together, and the newest information wasn’t necessarily the one that reached every worker first. We couldn’t make the whole platform wait for perfect ordering, so the processing rules needed to know which information could be applied independently and which updates had to be compared with a version or timestamp. This wasn’t a problem in every flow, but where it mattered, ignoring it produced results that looked valid until somebody compared them with the original events.

Another part that only became obvious under volume was backpressure. If one enrichment service became slower, the events didn’t stop arriving. The queue between the stages grew, processing time increased and a temporary dependency problem could continue affecting the platform after that dependency had recovered. We watched both failures and delay because a pipeline could be technically processing every message while still falling far behind the current traffic.

Retries and Delayed Events

We used retries, but simply trying everything again wasn’t safe. If a dependency was already having problems, sending the same request several times could make it worse. We had to limit the attempts, keep enough information about the progress and make the important operations safe to execute more than once. There were also events that weren’t failing but were taking longer than normal, and we needed to avoid treating every delay as an incident while still noticing when processing had really stopped.

This part of the work involved many decisions that were specific to our platform. We discussed which failures could recover without anyone doing anything, how long the system should wait and when somebody needed to look at an event manually. Sometimes we changed the processing logic, and other times we found that the real problem was that we didn’t have enough information to understand what the system had done.

We didn’t use the same retry policy everywhere. A short network timeout was different from invalid event data, and both were different from a response saying that a resource didn’t exist. Retrying invalid data several times only delayed the moment when we had to inspect it, while immediately giving up on a temporary timeout created manual work the platform could have resolved by itself. We classified these situations as part of each integration instead of placing one generic retry around the complete pipeline.

When the automatic attempts ended, the event still needed somewhere to go. We kept failed work visible and retained enough context to process it again after the cause was fixed. This was useful during incidents because we didn’t need to reconstruct the original event from logs or ask another system to send it again. It also meant replaying events had to be treated as a normal operation, with the same concern about duplicate effects as the original processing.

Following an Event

Our logs and metrics needed to help us follow the same event through different components and attempts. A message saying that event processing had failed didn’t help much if we couldn’t see which event it was, what stages had completed and whether the current execution was the first attempt or a retry. We spent time adding this context because otherwise the beginning of an investigation was mostly people trying to find the same operation in several places.

Metrics gave us the wider picture while the event history gave us the specific one. We watched the number of events entering and leaving important stages, the age of the pending work and the proportion of operations being retried. A growing queue could be more useful than an error count because it showed that processing was losing pace before the workers stopped completely. When we found a problem, we still needed the event identifier and the path it had followed to understand why only some messages were affected.

Deployments also needed more thought than they did in a request-based application. Workers could be handling old messages while a new version was starting, and changing the format or meaning of an event without considering that overlap could make the deployment itself create failures. We kept compatibility during transitions and avoided assuming that every message in the system had been produced by the version currently running.

I used to think more about retries and observability after the main flow was already working. On this platform, they’re part of the conversations while we build the flow because the failures aren’t separate from the normal work. The volume just gives us more opportunities to see them, sometimes several times in the same day.

There are still days when an event reaches a state we didn’t expect or a dependency fails in a new way. When that happens, we can usually find the event, see each attempt and understand how far it went. At more than 100,000 events a day, I don’t expect to remove every failure. I expect us to contain it, recover the affected work and keep processing everything else.