Failure Rates at 100,000+ Events per Day

On a security platform I worked on, we processed more than 100,000 events per day. A condition that happened once in every ten thousand events could appear around ten times every day. We might never see it in a small test, but the production system would keep finding it. The percentage was small, but the number of operations needing attention wasn’t.

What Production Found

We tested expected inputs, failures, and important boundary conditions. Production still produced combinations that were difficult to reproduce in a controlled environment. Events could arrive close together, repeat, appear later than expected, or depend on data that was changing at the same time. A downstream service could be healthy for most requests and slow for a small subset. A retry could overlap with work that was still finishing. None of these situations was surprising by itself. The volume made sure the platform encountered them regularly. This changed how I think about edge cases. The percentage was only part of the discussion. We also needed to look at the volume and the impact. A rare formatting problem with a harmless fallback may remain low priority. A rare duplicate that can produce a conflicting security result deserves a different response.

The Failure Queue

At high volume, even a strong success rate can leave meaningful operational work behind. If failures require manual investigation, a small percentage can become a queue that grows every day. If retries happen without limits, a small problem can generate additional traffic and make recovery slower. If failed events are difficult to trace, the time spent understanding them can exceed the time spent fixing them. We designed the processing flow so expected failures had an expected path. Recoverable problems could retry safely. Invalid events were separated from temporary dependency failures. Operations that needed attention remained visible instead of disappearing into a generic error log. We expected some events to need another attempt. What we didn’t want was normal production behavior creating work that nobody could see or manage.

Logs, Metrics and Volume

Logging everything isn’t the same as understanding the system. With a continuous event stream, unstructured logs can become another large dataset that hides the useful signal. We needed identifiers that followed an event across components, clear processing states, and metrics that showed changes in behavior before a backlog became serious. Counts also needed context. Ten failed events may be alarming in a system that normally processes twenty. The same number may represent a small variation in a system processing thousands during that period. Ratios, trends, latency, and backlog age often told a more useful story than a raw total. Volume didn’t mean that every imagined edge case deserved an elaborate solution. We still asked how often a situation could occur, what its impact would be, whether recovery could be automatic, and whether we’d notice it. At scale, a small weakness can become daily operational work. A rare race condition can become a routine production case. A tiny percentage of untraceable failures can consume hours of engineering time.

I still use percentages, but on that platform I always translated them back into real events. Less than one percent could still mean several operations every day, and each one could leave somebody searching through logs or deciding if a retry was safe. The percentage looked small on a dashboard. The work behind it didn’t always feel small.