Performance Tests and Sqlite3

Performance "unit tests" present a bit of problem to the usual QA tools since it's not the usual "pass/fail" so typically you'll have to write your own harness and store the results and apply some extra logic to determine if the build was ok or not

I can't help you write the harness, but I can help with where to store the results.

RRDtool is great. However it has a lot of options and I've found unless the engineer is already familiar with it, it's a bit of overkill for a performance database. It also works best when the test is run regularly. (I should actually write up a HOWTO for this). MySQL also excellent, but it requires setting up a server, and permissions, and all that.

sqlite is perfect. No server. No config. One File. Standard SQL. Here's the SQL for a sample metrics db:

DROP TABLE IF EXISTS metric;
CREATE TABLE metric (
       date       TEXT NOT NULL DEFAULT CURRENT_DATETIME,
       name      TEXT NOT NULL,
       value       REAL NOT NULL 
);

DROP INDEX IF EXISTS datename;
CREATE UNIQUE INDEX datename ON metric (date,key);

You might want to jazz up this table and add a build number, SVN/CVS id or product version. The 'name' field is just the name of the test.

To create the database, just do sqite3 DBNAME < FILE, where FILE contains the SQL above

python and php are now shipping with sqlite3 out of the box and you can use the fancy APIs, however you can use sqlite3 directly: just add the sql statement as the last argument, sqlite3 db 'sql':

# python example
import os
os.spawnlp(os.P_WAIT, 'sqlite3', databasename, 
                 "INSERT INTO metric SET name='%s', value=%f' % (name,value')

In php, see exec, in perl see exec.

Sleazy, yes. By all means, actually use the real API. But it's not available, this works. It also means you can use sqite3 via bash.

Now go make your intern make pretty graphs for you and do alerts if the last run is 20% slower than the last.

Whoops, I forgot the optimization flag

So it turns out if you use macports on Mac OS 10.5, python2.5 is compiled without any optimization. This makes it run at least 2x slower than the version in /usr/bin (the bug report is here).

I only noticed this since I'm a nerd and tuned on verbose/debug flags on macports and then thought it was weird that I didn't see the usual -O2. I then wrote a minor benchmark to confirm that indeed, the macports version is slower.

Which reminds me of major screwups I have done involving pushing debug code live instead of the normal optimized version (in that case it was C++ code). In this case I think it took 6 hours of many people's time to "undo" since all the servers were on fire and end customers were pissed off.

Besides turning on al compiler warnings, besides running your unit tests, besides running your integration tests, you also need to have some type of minor performance test to catch these problems. Even the simplest test will catch these type of problems. And it's not just for C++: even if you are writing using a scripting language, the packager of the language can screw up, or use a different optimization between builds, or use a different compile or just not use the best optimization possible.

In the macport example, it appears that some tweeks needed to be done to make python compile on the new OS. In the process of fixing that, the optimizer flags were ignored silently. Nothing was "removed". I've done the same thing fiddling with Makefiles. I didn't remove optimization, I just screwed up some rules, so the wrong flags were used. In other words, it's really easy to this.

Have a performance smoke test