I’m doing a little research right now on distributed testing frameworks and thought I could blog as I go. You may be asking, “But Alex, you work at Terracotta, making a distributed computing product, surely you have a distributed testing framework?” And sure enough we do have one. Well, actually 2 or 3, all with their pros and cons.

But I have some new needs and ideas and I’m doing a little poking to see if it makes sense to extend an existing external framework rather than an existing internal framework or writing something new. I’m also wondering whether integrating an existing framework with Terracotta would be a useful thing for a lot of others as well. In particular, using Terracotta can make it a lot easier to coordinate test actions and phases and collect results.

Here are some good meta pages I’ve found that seem to list every project that came to the top of my mind plus a lot of others:

• opensourcetesting.org performance tools
• manageability.org automated test tools page

I don’t really want to bore you with the detailed requirements I have but things I would expect to get out of a distributed test framework would include the ability to start up multiple nodes, walk those nodes through a series of phases (not just a single load), start/stop nodes as part of the test, use some kind of scripting for some or all of the actions taken in each phase, and collect and aggregate results.

I’d really like to write the scripts in something simpler than Java like Groovy, Jython, Ruby, Javascript, etc. If you mention XML, you have to go sit in the corner and wear a funny hat until you apologize.

Ones that I find to be leading choices at the moment:

• JMeter – this was the first thing to came to mind and something I’ve used in the past with success. I think it would probably hit 90% of my needs. The scripts are a little clunky for my taste but certainly usable. Plus, you can plug in bean shell scripts, so I think I could do something reasonable.
• The Grinder – seems to have advanced quite a bit since the last time I looked at it. Scripts are in Jython which makes them both flexible and readable, which I like a lot. I’ve worked with people that used this successfully and liked it.

Ones that I looked at but am not currently considering (nothing negative implied here, just not a good match for my reqs):

• Testare – seems to focus on J2EE testing and container testing
• Java Test Runner – doesn’t seem like a good match, doesn’t seem to be a community around it

Am I missing anything stupidly obvious and important for consideration? I’d be happy to use something that was written or scripted in a language other than Java as long as I could launch Java stuff efficiently. Any distributed JRuby test framework I should be aware of?

Searching just on my top finds so far yielded a shoot-out report of JMeter, The Grinder, and Mercurcy LoadRunner. He ultimately picked The Grinder and liked some of the same things I like about it. My own general impressions of Mercury tools in general is that they are expensive and some of the lowest quality software I’ve ever used (on any dimension), which doesn’t seem good for tools intended to improve your own quality.