Reviewing Reliability in an Integration Workflow

A pull request for an integration workflow looked complete. Validation worked, the external call returned the expected response, and the result was stored correctly. During the review, we started looking at everything around the successful response. What happened if the external call timed out? Could the client send the request again? What if our database update succeeded but the next step failed? How would support determine the current state of the operation? The code wasn’t wrong, but it didn’t answer all the questions we’d have in production.

The Assumptions in the Happy Path

The first implementation had assumptions that weren’t written down. We expected the dependency to answer inside the timeout, the request to arrive once and a repeated operation to be safe. Some of this was reasonable, but we hadn’t checked what happened when two assumptions failed at the same time. During the review we walked through the operation step by step and found the places where the state became uncertain. That discussion changed the design before the workflow reached production.

Incomplete Operations

It’s easy to treat retries, timeouts, and logging as infrastructure around business logic. Sometimes the business model itself needs to represent incomplete work. An operation may be accepted but not completed. It may be waiting for another system, retrying after a temporary failure, or stopped because the input requires review. If the model includes only success and error, several meaningful situations become indistinguishable. We added states that represented the workflow and made the transitions clear. This helped the application decide what it could retry and helped the team understand what had happened. The implementation had more details, but the behavior was easier to understand.

Recovery and Testing

Waiting until after launch to consider recovery usually produces emergency scripts and manual database updates. Those tools may be necessary during an incident, but they shouldn’t be the normal recovery path for expected failures. For this workflow, we considered safe repetition, controlled retries, useful operation identifiers, and how unresolved work would be exposed. We also made sure logs recorded business context rather than only a stack trace. We weren’t trying to imagine every possible failure. We focused on the ones created directly by our design. If the feature depends on a network call, network uncertainty isn’t an unexpected edge case. The easiest tests followed the happy path. The more valuable tests exercised repetition and partial progress. Could the same request be received twice without duplicating the effect? Did a temporary failure preserve enough information for another attempt? Would an invalid request stop instead of retrying forever? Did a state transition occur only after the required work completed? These tests described the guarantees the feature made in production.

Those tests also protected recovery behavior that users didn’t see directly. The amount of work depended on the risk, of course; we didn’t treat a small internal tool like a critical transaction. But for this integration, the timeout, the repeated request and the partially completed operation were already part of the feature discussion. We didn’t put them into a future “hardening” phase because the first version of the workflow already depended on a network call that could fail.