Reviewing Reliability in an Integration Workflow

A pull request for an integration workflow looked complete. Validation worked, the external call returned the expected response, and the result was stored correctly. During the review, we checked what the system would do after a timeout, a repeated client request or a failure that happened after the database update. We also tried to follow the operation from the information support would have in production. The code worked for the successful response but left these other states unclear.

The Assumptions in the Happy Path

The first implementation had assumptions that weren’t written down. We expected the dependency to answer inside the timeout, the request to arrive once and a repeated operation to be safe. Some of this was reasonable, but we hadn’t checked what happened when two assumptions failed at the same time. During the review we walked through the operation step by step and found the places where the state became uncertain. That discussion changed the design before the workflow reached production.

We wrote down the sequence in the pull request and marked the points where an effect had already happened. Before the external call, another attempt was simple. After the call was sent but before its response was stored, we no longer had the same certainty. Looking at the flow this way was more useful than reviewing the controller line by line because it showed where a technical error changed the business state.

Incomplete Operations

Retries, timeouts and logging looked like infrastructure around the business logic until the first operation stopped halfway through. The model needed to represent work that had been accepted but not completed, was waiting for another system, was retrying after a temporary failure or had stopped because the input required review. With only success and error, those situations were indistinguishable. We added states for the workflow and made their transitions explicit, which gave the application enough information to decide what it could retry.

We avoided exposing every technical step as a user-facing status. The application needed enough detail to recover and support needed to see where processing stopped, but users mainly needed to know whether the operation was still moving, had completed or required an action from them. Keeping those two views connected without making them identical made the workflow easier to explain.

Recovery and Testing

Waiting until after launch to consider recovery usually produces emergency scripts and manual database updates. Those tools may be necessary during an incident, but they shouldn’t be the normal recovery path for expected failures. For this workflow, we considered safe repetition, controlled retries, useful operation identifiers, and how unresolved work would be exposed. We also made sure logs recorded business context rather than only a stack trace. We focused on failures created directly by our design. The tests repeated the same request without duplicating its effect, preserved enough information after a temporary failure, stopped invalid work instead of retrying forever and changed state only after the required step completed.

One test simulated the external service completing an operation while our client received a timeout. It forced us to decide what the next attempt should do instead of simply verifying that an exception was thrown. Another repeated the same incoming request with the same operation identity. These cases took more setup than the success response, but they protected the behavior we would depend on during an integration problem.

Those tests also protected recovery behavior that users didn’t see directly. The amount of work depended on the risk, of course; we didn’t treat a small internal tool like a critical transaction. But for this integration, the timeout, the repeated request and the partially completed operation were already part of the feature discussion. We didn’t put them into a future “hardening” phase because the first version of the workflow already depended on a network call that could fail.