Often performance testing starts with a statement like “we need to test the performance of X”. This statement is often devoid of any details on how to do such a thing or what you hope to discover.

Start from the end

My simple tip is to start from the end. The first thing you should do is to design the report that you want when the effort is complete.

Any good performance report has graphs, so draw them. Label the x and y axes, figure out what each series represents, label the units. The x axis (and series) tells you what you want to vary. The y axis tells you what you want to measure. Anything not on the graph should be fixed. You must determine concrete answers for each fixed value.

Sometimes you’ll discover that one of the fixed values is really interesting and it would be great to explore what happens when that value varies. Great! Draw another graph and put that variable on the x axis. Repeat.

Let’s try it

Let’s try a concrete example: I need to test the performance of multi-threaded Maps. (See how vague and useless that goal is?) Instead let’s focus on what graph we might want out of the answer.

Most performance graphs are going to have either response time (latency) or throughput as the y axis measurement. Let’s say throughput. We want to test multi-threaded performance, so we probably want to know how throughput changes as the number of threads changes. That’s our x axis – # of threads. Pick some x axis hash marks – those tell you concrete values to test with. Should they go up linearly, or a factor?

Throughput is measured in “actions per time”. What actions is the test performing? Often it’s useful to shove another dimension into the graph by varying something else and drawing one series per each discrete value in the extra dimension. In this case, it’s probably useful to vary what kind of actions the test is performing. I’ll make a big leap and suggest that a mixture of reads and writes is probably interesting. So we should pick some discrete mixes to test and each one becomes a series (draw the legend).

So what’s left? Well, we don’t know what kind of Map we’re testing. Let’s say we’re going to test ConcurrentHashMap. And we need to determine the size of the initial map, what kinds of keys and values should be used, etc.

Follow your nose

Gee, we had to pick the Map implementation in the last graph. Sure would be interesting to know how ConcurrentHashMap compares to ConcurrentSkipListMap and a synchronized HashMap, wouldn’t it? We could do another graph with that as a variable. It’s discrete, so that usually maps better to the series variable. We could leave our threads as the x axis. Then we’d have to fix the read/write ratio. What’s interesting to you?

Hopefully that gives you the idea of the thought process. Usually, 20-30 minutes of brainstorming can give you 2 or 3 things you really want to know. Now you have not only a concrete plan of what to test, but you know you’re done when you can draw the graphs.