In my role as a Java Technical Architect, I am forced to wear different thinking hats. Some times I am a Solution Architect, Subject Matter Expert on some technology and the last few days I have been wearing the hat of performance tuning expert. Today I am going to pen down some thoughts around what a person should focus on once he or she is by design or default selected to tune an application.
Rule No. 1: Focus on data and not on people.
The first thing that happens whenever there is a performance issue is that there is a lot of people talking about the issue and adding their colored perception and inferences without supporting evidence. Such people will come in all shapes and sizes, project managers, delivery owners, programmers and any one who wants to feel important by putting down his or her opinion. It is your job to separate out the noise from the sound. Do not let these people’s opinions cloud your perspective. The first thing to do is respectfully enquire if anyone has taken the effort to gather any data around the performance issue. If not, start gathering the same yourself. This is the first and the most essential step in your arsenal, which you would revert to time and again to back your observations or performance improvement suggestions.
So, how do you gather data? There is a lot of sophisticated tooling like profilers and performance monitoring equipment around, however I always use my humble file logger to capture information. Here’s a sample.
long startTime = System.nanoTime(); // Some method invocation of a significant process long endTime = System.nanoTime(); logger.debug("Time spent in invoking significant method: " + (endTime-startTime)/1000000 + " msec.");
Identify possible bottleneck areas and put in logger statements like what I have written above. Profilers will freeze, but the humble logger will always log unabated, which moves us to rule no. 2.
Rule No. 2: Focus on basics of performance tuning.
Before jumping into the application code and theorizing on the possible areas to fix look at the obvious, check if your JVM has enough memory allocated, the CPU has enough free time, in case of database queries check out if they have been tuned. Seems the obvious choices, unfortunately projects suffer because the obvious is overlooked for the exotic. Just recently I was called because a criteria based UI search was taking a lot of time, specifically 30 to 90 seconds. After some digging it became apparent that the database procedure fetching the search results was slow and was consuming 95% of the total process time. I spoke to the PL/SQL developer and inquired what made the query slow. His conclusion was the query was on a single table containing around 75 million records. On my pestering he asked the DBA to run statistics which apparently also tunes up the table and presto the search results appeared in 3 secs. Conclusion, do not overlook the obvious.
Rule No. 3: Do performance testing on the target machine in the target environment.
A few years ago I was asked to look into a memory leak. The target system was a Unix machine in the customer environment and all developer machines were on Windows. We spent around 2 weeks working on the developer boxes trying to replicate the memory leak issue. It never appeared. So we changed tact and got monitoring enabled on the target machine. Another week of testing and the conclusion was that Tomcat 4.1 on Unix had a memory leak. The Windows version did not have the same problem. So lesson for the day, always ensure that you do the testing on the same machine where the failure is reported.
Rule No. 4: Watch out for framework specific idiosyncrasies.
Each framework, tooling or component that you utilize in your application has its own characteristics. Typically each framework has one or a limited set of objects which hold all the configuration information. For example, JAXBContext in JAXB, SessionFactory in hibernate or EJBHome in EJBs. These are heavy objects whose creation imposes a performance penalty. Care should be taken in design to ensure that one or a limited set of objects are created thus avoiding the performance penalty.
One of the integration projects my organization was working on was facing some performance issues. The project used a lot of web services and concerned people were speculating that the issue was due to excess use of web services. Although the statement was true, that was just half truth. Due to the large number of Web Services, there was a considerable marshalling and unmarshalling happening. JAXB was used for the data transformation which is not a problem except that on each and every marshalling and unmarshalling operation, a new JAXBContext was created, this was consuming @5 seconds for a single object creation, the entire operation was taking around 90 seconds. On changing the implementation to creating the JAXBContext once in the application lifetime and reusing the same object the response time was down to 3 secs. Such objects are typically threadsafe by design. It’s just up to you to know that and make a judicious decision in implementation.
Rule No. 5: Observe the performance metrics for all moving parts/components.
A web application consists of many moving parts/components, the presentation tier, integration tier, data access tier and other third-party components like document generation, bar code generator utility etc. As a performance tuning consultant you need to observe the performance characteristics of these parts under variety of business scenarios. At times a component takes time only under certain business flow. To capture this behavior load test the application and run the same scenario over and over again. This is an extremely useful technique in capturing slow forming memory leaks and also identifying or ruling out components that might have performance problems.
Rule No. 6: Remember the 80-20 principle.
Application code is written by developers of varying experience and expertise levels. This translates into poor design or non-implementation of design patterns, reusable code etc. However modern-day JVMs are robust enough not to take a performance hit or take a limited performance hit due to poor coding standards. Do not waste your time in doing static code analysis and suggesting code implementation as per standards. Remember most of the performance problems have one or two root causes. Fix these and the bottlenecks disappear. Do your 20% by identifying these problem areas and get 80% improved throughput.
Finally, my rules/tips are simple, obvious and will be apparent to people with common sense. However time and again in practice, I have seen these been overlooked. After all common sense is a sense not found in common people.
Thats all from performance tuning help desk.