Building and Operating My Own Production Systems

In freelance projects, I often owned more than the implementation. I designed the backend, modeled the data, built features, deployed the application, and responded when production did something unexpected. Sometimes the first alert was a client message rather than a dashboard. That changed my relationship with the code. A vague error message was no longer a small detail someone else would handle. A difficult deployment was a process I’d repeat. A hidden background failure could become a client asking why an operation hadn’t completed. I knew I’d need to understand these small decisions again when something happened in production, probably at an inconvenient time.

Error Messages in Production

During development, a stack trace and local debugger can provide enough context. In production, the useful question is often which user operation failed and what the system had already done. I started giving important workflows identifiers that appeared across logs and persisted state. Errors distinguished invalid input from a temporary dependency failure. Logs recorded relevant decisions without exposing sensitive data. This didn’t make incidents easy, but I spent less time searching unrelated logs while somebody waited for an answer. Instead of searching by timestamp and guessing which messages belonged together, I could follow the operation through the application.

One client reported that an integration result hadn’t appeared. The application had several log entries around the same time, but none connected the API request to the later background job. I eventually found the failed work through database timestamps, which was enough to resolve the problem but not a process I wanted to repeat. After that, the operation identity was stored when the request was created and carried into the job and external call.

I also changed error messages with support in mind. A message didn’t need to expose internal details to the user, but it needed a reference that I could search and a distinction between something they could correct and something the system would retry. Before that, several errors simply asked the user to try again, even when another attempt would produce the same invalid request.

Background Job States

Several products included scheduled tasks, notifications, integrations, or other background jobs. In code, it was easy to think of them as functions that continued outside the user-facing request. Then a notification went missing or an integration stopped halfway through, and I needed to know whether the job had been scheduled, started or completed, whether it was safe to retry and if the failure was visible anywhere. I started storing meaningful processing states and enough information to understand incomplete work. Otherwise, somebody would only find the failure later as a support problem. Owning deployment also made me more careful about changes that crossed code and data. A database migration needed to work with the application version active during rollout. A new required field couldn’t assume every existing record already had a value. A long-running job might still be using the previous behavior while the new version started. These situations encouraged smaller, compatible steps instead of one change that required everything to switch at exactly the same moment.

Retries were limited and depended on the failure. A temporary provider timeout could run again, while an invalid destination or missing configuration needed attention. I kept failed jobs visible after the automatic attempts ended and added a controlled way to run them again after the cause was fixed. This saved me from recreating the operation manually or changing a database status without knowing which steps had already happened.

For deployments, I started separating schema changes from the code that depended on them. An optional field could be added first, existing records could be updated and only then would a later version require it. The process took more steps but each one was easier to reverse. That mattered when I was the person checking the release and deciding whether an unexpected result was safe to continue.

I also invested more in automated builds, tests, and deployment pipelines. I had repeated enough manual releases to know that familiarity wasn’t the same as reliability.

The pipeline built the same artifact for each environment, ran tests and made configuration differences explicit. I still checked important releases, but I wasn’t rebuilding the procedure from notes or terminal history. When a deployment failed, the next attempt started from a known state instead of depending on which commands had completed on my machine.

Keeping the System Small

When I operated what I built, every additional component had a very visible cost. A separate service meant another deployment and more configuration. A new database meant backups and another set of access patterns to understand. For smaller products I usually stayed with a modular application and a few dependencies I knew well, partly because there wasn’t an operations team behind me. I was the person who would receive the client message, open the logs and try to remember why the code behaved that way. Knowing that changed small details in the implementation: better operation identifiers, clearer errors, visible job states and deployment steps I could repeat. I was mostly leaving information for my future self, but it also helped the next engineer who needed to understand the project.

I added queues, caches or separate workers when a measured workload needed them. Every new component also needed a plan for backup, monitoring, updates and recovery because those tasks would return to me later. A simpler architecture wasn’t always faster to build, but it usually gave me fewer places to search when a client was waiting.

Owning production didn’t make me predict every incident. I still found errors through client messages and had deployments that behaved differently from local testing. It changed what I considered part of the feature. The operation identifier, failed-job view and repeatable release weren’t work around the application anymore. They were how I made the application possible to support after I had moved on to the next feature.