Application Performance Tuning – A forensic view point

I have been looking at application performance issues in various shapes and forms since I started doing software development around 15 years back. As I moved up the technology value chain, I started getting called into crisis assignments. Basically there was an application in crisis; it was experiencing performance issues. Could I come over and help address the issue?

Over the years I could see a pattern emerge around how I went about solving the performance issue. The solution approach is akin to becoming a detective, looking at the forensic evidence, conceptualizing a theory which addressed the observations and based on theory proposing a solution. In case the problem is reproducible, replaying the scenario and proving your approach’s validity.

I couldn’t help but draw parallels with Sherlock Holmes or his medical counterpart Dr Gregory House. Their quotes provide insights in the approach to take for performance tuning.

The first quote to start with is Dr House’s.

Everybody lies.

You are called into this crisis meeting. The client has expressed concern about the application’s performance. To address and alleviate client concerns, your delivery leadership creates a crack team to identify the root cause and propose remedial measures. You are tasked with leading this activity and an unreasonable deadline is thrust on your shoulders. Your first urge is to get hold of available information and start working from there. This is your first mistake. Information provided to you can be erroneous, someone’s perspective or viewpoint. It is not that people are lying purposely but misinformation happens.

Now we move on to the second quote.

There is nothing like first-hand evidence.
– A Study in Scarlet

Start with gathering data independently. Assess the application flow in trouble and determine checkpoints around which you need pertinent performance data to initiate your analysis. Dispassionate, relevant data is the first stepping stone in application tuning. Do not let deadlines or any other distractions sidetrack your focus on getting data.

The world is full of obvious things which nobody by any chance ever observes.
– The Hound of the Baskervilles
Chapter 3: The Problem

While gathering data, focus on everything. Pay attention to detail and consider all information elements however big or small. The biggest failure in tuning occurs because we did not gather all information.

“Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.”
-The Adventure of the Copper Beeches

Information is the king. Work towards acquiring more than adequate data points. You cannot build castle in air or theory without adequate data points.

Once the data starts flowing in, you need to separate the wheat from the chaff. This is where the next quote comes in.

“You see, but you do not observe. The distinction is clear.”
Sherlock Holmes Quote
-A Scandal in Bohemia

Take all the data points received and put them under a microscope. Determine which is relevant and which is not. This is a vital step. Too much data will leave you overwhelmed with information overload; too little will make it difficult to theorize.

Having gathered these facts, Watson, I smoked several pipes over them, trying to separate those which were crucial from others which were merely incidental.
-The Crooked Man

Now that you have gathered data, determined what is relevant, its time to theorize. Start weaving stories which align to the facts observed. Eliminate the ones which do not cover all facts until you find one or more theories meeting all factual information. Multiple matching theories could point to multiple problem areas or you need to gather more information to finalize on one.

I ought to know by this time that when a fact appears to be opposed to a long train of deductions it invariably proves to be capable of bearing some other interpretation.
– A Study in Scarlet
Chapter 3: Light in the Darkness

Eliminate all other factors, and the one which remains must be the truth.
– The Sign of Four
Chapter 1: The Science of Deduction

…when you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.
– The Blanched Soldier

New facts / data points can emerge. If they do not fit into your current theory, there is bound to be a different interpretation which we have not considered. Some times what emerges can be completely unexpected. So do not keep a narrow focused mindset.

The emotional qualities are antagonistic to clear reasoning.
– The Sign of Four

I never guess. It is a shocking habit,—destructive to the logical faculty.
– The Sign of Four
Chapter 2: The Science of Deduction

Sometimes in spite of our best efforts, we are not able to come up with an appropriate solution or identify the root cause. We try to compensate by guessing. Please avoid.

It is the nature of medicine that you are going to screw up.

In spite of our best efforts we will fail. We will make mistakes, start from the beginning, revisit data and proceed on the path mentioned above.

I consider that a man’s brain originally is like a little empty attic, and you have to stock it with such furniture as you choose. A fool takes in all the lumber of every sort that he comes across, so that the knowledge which might be useful to him gets crowded out, or at best is jumbled up with a lot of other things, so that he has a difficulty in laying his hands on it. Now the skillful workman is very careful indeed as to what he takes into his brain-attic. He will have nothing but the tools which may help him in doing his work, but of these he has a large assortment, and all in the most perfect order. It is a mistake to think that little room has elastic walls and can distend to any extent. Depend upon it – there comes a time when for every addition of knowledge you forget something that you knew before. It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones.
– A Study in Scarlet
Chapter 2: The Science of Deduction

A key asset in performance tuning is your knowledge around the architecture of the software assets you are working on. Sherlock Holmes or House were very good at their jobs because of the tremendous amount of knowledge they gathered on varied things which a normal person would neglect. To be a good performance tuner, start gathering knowledge.

Happy performance tuning!

Featured image courtesy pixabay.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s