This week the whole Terracotta engineering gang from all over the world is in San Francisco for Terracotta dev week. We try to do this a couple times a year and it’s pretty intense. The activities officially go from 8 am to 8 pm every day but it’s not uncommon for people to be at the office late into the night.

One activity we usually do is a performance tuning competition. Being all about clustering apps and making them scale, we think it’s important for everyone on the team to be a guru in performance tuning. We break everyone up into cross-functional teams and give them all the same app to work on. In this case, we were working on an app built by our field guys to simulate single-sign on that was using MySQL and Hibernate for the backing store of authenticated credentials and every team got a 6-node cluster to work on (we’ve got a big perf lab :).

For the next Terracotta release (3.1) we’re working on building the most kick-ass clustered Hibernate second-level cache in existence. It will be the Chuck Norris of Hibernate second-level caches. Anyhow, the test had two phases – a load phase (to fill the db and warm the cache) and a query phase. The baselines for phase 1 were somewhere around 45 TPS and no one actually could baseline phase 2 because the app would run out of memory and crash. Lots of room for improvement.

By the end of the competition, we had fixed some bugs in the app, made a bunch of configuration tuning changes for Hibernate and Terracotta, and even found a bunch of awesome improvements at various levels of Terracotta and the our cache implementation. Most teams were seeing 3,000-4,000 TPS in phase 1, were able to load the entire 3 million cache entries without memory problems and were seeing in the region of 60,000 TPS in phase 2. Pretty sweet. Some of the memory-related optimizations we found are completely generic and should allow us to be much more efficient in all Terracotta apps.

At the moment, we are in the second day of a product improvement competition. Generally we turn people loose and let them work on whatever they want with some focusing guidelines. Last time we did this we had people try some crazy stuff but we didn’t put a high premium on actually getting something fully productized because it was a shorter competition. This time, we put some more emphasis on making things that actually can go into the product right away and also a focus on usability and Hibernate users.

The competition’s not over yet but there is some awesome stuff in work. I suspect we’ll see even better Hibernate tooling come out of this (since several people are working on things for that) and several other usability and performance improvements. It’s definitely a good thing to have the performance tuning competition first so people get a taste of where some bottlenecks are and where the tooling wasn’t helpful enough.

All of this is scored by our judges (from other parts of the company) and we give out some prizes to those that have made the biggest impact. We’ll finish out tomorrow with a bunch of brainstorming and meetings. All in all, it’s a great team exercise and a lot of fun. It’s also completely exhausting. :)