Failure Rates at 100,000+ Events per Day

On a security platform I worked on, we processed more than 100,000 events per day. A condition that happened once in every ten thousand events could appear around ten times every day. We might never see it in a small test, but the production system would keep finding it. The percentage was small, but the number of operations needing attention wasn’t.

What Production Found

We tested expected inputs, failures, and important boundary conditions. Production still produced combinations that were difficult to reproduce in a controlled environment. Events could arrive close together, repeat, appear later than expected, or depend on data that was changing at the same time. A downstream service could be healthy for most requests and slow for a small subset. A retry could overlap with work that was still finishing. None of these situations was surprising by itself. The volume made sure the platform encountered them regularly. This changed how I think about edge cases. The percentage was only part of the discussion. We also needed to look at the volume and the impact. A rare formatting problem with a harmless fallback may remain low priority. A rare duplicate that can produce a conflicting security result deserves a different response.

We used production examples to improve the tests instead of trying to imagine every possible combination before release. When an event reached a state we hadn’t covered, we kept a safe version of the relevant input, understood why the existing rule failed and added that case to the suite. Over time, the suite included more of the real behavior the platform had encountered and fewer assumptions from the first implementation.

The Failure Queue

At high volume, even a strong success rate can leave meaningful operational work behind. If failures require manual investigation, a small percentage can become a queue that grows every day. If retries happen without limits, a small problem can generate additional traffic and make recovery slower. If failed events are difficult to trace, the time spent understanding them can exceed the time spent fixing them. We designed the processing flow so expected failures had an expected path. Recoverable problems could retry safely. Invalid events were separated from temporary dependency failures. Operations that needed attention remained visible instead of disappearing into a generic error log. We expected some events to need another attempt. What we didn’t want was normal production behavior creating work that nobody could see or manage.

The age of that queue mattered as much as its size. A temporary spike could create many failed attempts that recovered a few minutes later, while a small group of events waiting for days meant there was a condition the automatic process would never solve. We exposed both so the team could distinguish a busy period from work that had been forgotten.

Logs, Metrics and Volume

Logging everything isn’t the same as understanding the system. With a continuous event stream, unstructured logs can become another large dataset that hides the useful signal. We needed identifiers that followed an event across components, clear processing states, and metrics that showed changes in behavior before a backlog became serious. Counts also needed context. Ten failed events may be alarming in a system that normally processes twenty. The same number may represent a small variation in a system processing thousands during that period. Ratios, trends, latency, and backlog age often told a more useful story than a raw total. Volume didn’t mean that every imagined edge case deserved an elaborate solution. We still asked how often a situation could occur, what its impact would be, whether recovery could be automatic, and whether we’d notice it. At scale, a small weakness can become daily operational work. A rare race condition can become a routine production case. A tiny percentage of untraceable failures can consume hours of engineering time.

Alerts needed the same context. Alerting on one failed event would have created noise, but waiting for a very high error rate could hide a serious problem affecting a specific type of security event. We combined rates, duration and the affected stage, then adjusted the thresholds after seeing the normal variation of the platform. The first version of an alert was rarely the final one.

I still use percentages, but on that platform I translated them back into real events. Less than one percent could leave several operations every day for somebody to trace through logs and decide whether they were safe to process again.