Following One Operation Across Services and Workers

One user operation could leave traces in an API, a worker, a database, and an external integration. Each component logged something. Reconstructing the complete operation was another matter. That was the reality of several backend workflows I worked on at Softcraft. Locally, a developer could execute the main path and see a successful result. In production, the same operation might continue after the API responded, move between components, wait for a dependency, retry, and finish later. When something went wrong, reproducing the code locally was only part of the investigation. We needed to understand what had happened to one real operation across the entire system.

Connecting the Logs

We had a lot of logs. The problem was finding which ones belonged to the same operation. One service logged a request identifier. A worker logged a message identifier. The external integration used another reference and the database record had its own key. Every log was correct, but we still had to connect them manually. During production support, that translation cost time. We improved the propagation of operation identifiers so the same business workflow could be followed across boundaries. We also changed important logs to record the decision being made instead of only the method being executed. “Calling external service” was less useful than knowing which operation was being sent, on which attempt, and which state would follow.

The API created the operation identifier when the work first entered the platform. We stored it with the business record, included it in queued messages and passed it to integrations where the contract allowed an external reference. The worker still had its own job identifier and the database still had primary keys, but the operation identifier gave us one value to begin the search. When a new attempt was created, it kept the business identity and added the attempt number instead of becoming unrelated work.

Not every component adopted this at the same time. During the transition, we logged both the old local identifier and the new operation identifier at the boundary between them. This allowed support to cross the older parts of the workflow without requiring a large observability change across every service in one release.

Choosing What to Log

My first reaction was to add more logs. That can easily make a system harder to investigate. Repeated messages create noise, sensitive data can be exposed accidentally, and important events disappear inside implementation details. Large log volume also has an operational cost. We focused on transitions and boundaries: when work was accepted, when it moved to another stage, when a dependency responded unexpectedly, when a retry was scheduled, and when the operation reached a final state. These points described the lifecycle of the workflow. Debug-level implementation details still had a place, but the default production story needed to remain readable.

We reviewed which fields were safe to record because some workflows contained personal or sensitive information. Logging the complete incoming payload would have made debugging convenient while creating a different problem around access and retention. We preferred stable identifiers, status changes and selected context that explained the decision without copying the business data into another system.

The log level also reflected whether somebody needed to act. A temporary failure that the worker would retry normally didn’t need to produce the same alert as an operation that had exhausted every attempt. We still counted the retry in metrics, but keeping every recoverable event at a high severity made the actual unresolved failures harder to notice.

Patterns in the Metrics

Logs helped investigate individual operations. Metrics helped us see the system changing. Error rate, latency, retry volume, queue depth, and backlog age could show that a dependency or processing stage was degrading before users reported a complete failure. The relationship between metrics mattered. A growing queue with stable input volume suggested a processing problem. Increased retries alongside dependency latency suggested a different cause. A healthy average could hide a slow tail affecting a smaller group of operations. No dashboard explained everything, but the right signals gave us a smaller place to start the investigation. Infrastructure metrics are necessary, but users don’t experience CPU or memory directly. They experience whether an operation was accepted, completed, delayed, or failed. We connected technical signals to those outcomes where possible. Instead of only counting exceptions, we tracked the state of important workflows. This helped the team distinguish a harmless internal retry from work that was no longer making progress.

We built dashboards around what we checked during support: how many operations entered and completed each stage, the age of the oldest pending item, where retries were concentrated and whether the workers could catch up with the incoming rate. This kept the dashboard smaller than a collection of every metric the tools could expose.

Alerts were adjusted after seeing normal traffic. The first thresholds produced noise during expected peaks, and a few broad averages didn’t alert when a smaller group of operations was delayed for too long. We combined duration, rate and backlog age depending on the workflow. The thresholds were part of operating the system and changed as the volume changed.

It also made conversations with non-technical people easier. “The queue has 2,000 messages” needed another explanation, while “operations are completing with a ten-minute delay” described something they could connect to the user experience. Before this work, I mostly thought about observability as logs we added for later. After it, we could start an investigation with an operation identifier instead of a timestamp and a guess. That was a very practical difference during support, especially when the same operation had passed through four different components.

We still reproduced bugs locally when we could. Observability didn’t replace debugging or tests. It gave us enough information to know which behavior to reproduce and which data conditions mattered. Instead of starting with a generic report that something had failed, we could see the stage, attempt and dependency involved, then build a smaller case around what production had actually done.