Improving API Response Times by 60%

At Softcraft, I worked on a government case management platform that supported thousands of operations every day. As the platform grew, one of its most used workflows started getting slower. It was still working, but users had to wait longer for information they needed several times during the day, and the difference became more visible when the platform was under heavier use.

My first thought was that maybe the application needed more resources. We also discussed caching and whether we had reached a point where part of the stack needed to change. These weren’t unreasonable ideas, but at that moment they were mostly guesses. Before adding something new, we decided to follow the requests and see where the time was actually being spent.

The Investigation

The application was built with Java, Spring Boot and PostgreSQL. We followed the main requests from the API, through the application and down to the queries executed in the database. The endpoint looked simple from the outside, but it was doing more work than the response suggested. Some queries loaded information that the screen didn’t use, related records were fetched through several additional calls and a few filters prevented PostgreSQL from using the indexes as we expected.

Several smaller problems had become expensive as the amount of data increased; no single query accounted for the complete delay. That also explained why the issue had appeared gradually instead of after a specific deployment. The same code that had worked well with less data wasn’t behaving the same way anymore.

Reproducing that behavior outside production was part of the investigation. With a small local database, almost every version of the query looked fast, so changing the code based only on local tests would have given us a false sense of improvement. We used data volumes closer to the real environment and compared the complete request instead of measuring an isolated method. I also looked at the query plans because two queries that returned the same result could make PostgreSQL do very different amounts of work. In a few cases, an index existed but the way we filtered or joined the data meant it wasn’t being used as we assumed.

We also needed to separate database time from the time spent in the application. Some of the delay came after the query, when the service loaded related entities and converted much more data than the response needed. This was important because optimizing only the slowest query would have improved one part of the request while leaving the rest of the workflow almost the same for the user.

The Changes

We adjusted the heaviest queries, removed information that didn’t need to be part of the response and changed how some of the related records were loaded. We also added indexes based on the searches the application was really making rather than the access patterns we had imagined when the tables were first designed. After each relevant change, we measured the workflow again because some improvements that looked good in the code made almost no difference in the complete request.

These changes improved response times by more than 60% in the critical workloads we were working on, and the platform also became more consistent during peak periods. We didn’t need to replace the framework, introduce a cache or increase the infrastructure for this problem. Those options were still available if the measurements pointed in that direction, but they didn’t.

We released the changes in smaller parts and watched the same requests after each one. Query changes can look safe and still behave differently with real data, especially when they affect ordering, pagination or optional filters, so we kept the previous behavior covered and checked the results as well as the response time. The improvement wasn’t useful if a faster endpoint returned incomplete information or changed the order users expected on the screen.

After the Improvement

What stayed with me from this work was how normal the final changes looked compared with the discussions we had at the beginning. A few query changes, less unnecessary data and better indexes don’t sound like a large architecture project, but they solved the problem the users were actually having. It also made me more careful when a performance issue first appears, because the first solution that comes to mind is usually influenced by the part of the system I can already see.

I still consider infrastructure, caching or a different architecture when they make sense, but I’m more comfortable spending time with the current behavior before choosing one of them. In this case, that time helped us make the workflow more than 60% faster without giving the team another component to operate.

The platform continued growing after this change, so I don’t see the 60% as a permanent end state. New data and new features can change the access pattern again. What we had after the work was a faster workflow and a much clearer idea of how it behaved, which made the next performance discussion less speculative than the first one.