3xP: Predicting production performance pt. 1

I've recently worked a bit on performance tuning several applications that have been in production for several years. They range from 70 users to 3000 users, but have one thing in common: None of them have a performance management scheme.

Of course, it comes down to a question of cost. To be able to make, say, 99% correct predictions of how a system will perform, you need to run a bunch of tests in a sensible environment. There are two cost drivers: creating and managing the tests, and acquiring and maintaining the environment.

Creating load tests for a huge custom made application in three weeks is not possible. There are so many tweaks and processes and different behavior, that you'd need a load of domain experts just to tell you about all the things you can do. If the load tests were written in parallel with development, it would at least be possible to get somewhat complete coverage. But that's a huge overhead that cannot be added to the vendor budget. And the customer didn't specify inferior performance, so why should they take the cost?

I guess the lesson learned is that "good" performance may be governed by a tedious technical project management. Great performance has to be a concern even wider than the project scope, also including pre- and post-processes.

The second cost driver - environment - requires the first driver to be implemented. No need for realistic testing if you don't have a clue of what to test. But if you do - having a faulty or incorrectly scaled environment gives you either a number of falsely positive performance issues to solve, or a number of excuses for bad testing results. None of which is a good thing.

So, if you got your tests - do go the extra mile and ensure that the environment in which they are running is enabling you to do realistic testing and get correct predictions.