Following One Operation Across Services and Workers

One user operation could leave traces in an API, a worker, a database, and an external integration. Each component logged something. Reconstructing the complete operation was another matter. That was the reality of several backend workflows I worked on at Softcraft. Locally, a developer could execute the main path and see a successful result. In production, the same operation might continue after the API responded, move between components, wait for a dependency, retry, and finish later. When something went wrong, reproducing the code locally was only part of the investigation. We needed to understand what had happened to one real operation across the entire system.

Connecting the Logs

We had a lot of logs. The problem was finding which ones belonged to the same operation. One service logged a request identifier. A worker logged a message identifier. The external integration used another reference and the database record had its own key. Every log was correct, but we still had to connect them manually. During production support, that translation cost time. We improved the propagation of operation identifiers so the same business workflow could be followed across boundaries. We also changed important logs to record the decision being made, not only the method being executed. “Calling external service” was less useful than knowing which operation was being sent, on which attempt, and which state would follow.

Choosing What to Log

My first reaction was to add more logs. That can easily make a system harder to investigate. Repeated messages create noise, sensitive data can be exposed accidentally, and important events disappear inside implementation details. Large log volume also has an operational cost. We focused on transitions and boundaries: when work was accepted, when it moved to another stage, when a dependency responded unexpectedly, when a retry was scheduled, and when the operation reached a final state. These points described the lifecycle of the workflow. Debug-level implementation details still had a place, but the default production story needed to remain readable.

Patterns in the Metrics

Logs helped investigate individual operations. Metrics helped us see the system changing. Error rate, latency, retry volume, queue depth, and backlog age could show that a dependency or processing stage was degrading before users reported a complete failure. The relationship between metrics mattered. A growing queue with stable input volume suggested a processing problem. Increased retries alongside dependency latency suggested a different cause. A healthy average could hide a slow tail affecting a smaller group of operations. No dashboard explained everything, but the right signals gave us a smaller place to start the investigation. Infrastructure metrics are necessary, but users don’t experience CPU or memory directly. They experience whether an operation was accepted, completed, delayed, or failed. We connected technical signals to those outcomes where possible. Instead of only counting exceptions, we tracked the state of important workflows. This helped the team distinguish a harmless internal retry from work that was no longer making progress.

It also made conversations with non-technical people easier. “The queue has 2,000 messages” needed another explanation, while “operations are completing with a ten-minute delay” described something they could connect to the user experience. Before this work, I mostly thought about observability as logs we added for later. After it, we could start an investigation with an operation identifier instead of a timestamp and a guess. That was a very practical difference during support, especially when the same operation had passed through four different components.