2010/12/13
Release and test
Whew, finally I put 0.9.1 out of the door. There are several loose ends I couldn't managed to tie; for example, the new interface of rfc.http wasn't made public since I couldn't tested it enough. We still had enough stuff to make a long release notes, though.
The bottleneck of release cycle is testing. Yes we have unit tests, but they tests mostly the internal consistency---it tests that whatever you do inside Gauche you won't get wrong results.
The kind of tests before releases are focused on the external consistency---how Gauche interacts with the external world. Does it behave the same on different OSes? Does it work consistently when combined with other programs, and communicating over the network?
This time I happened to find a bug in gauche-config program
right after the release. The bug would affect where extension
packages are installed, and fixing things after many extension
packages are placed in wrong directory would be messy. So I
replaced the release with the fix.
How can I prevent problems like this, and ensure checking
other stuff that interacts with the outside world?
I had added some unit tests for utility scripts gauche-install,
gauche-config and gauche-package, but they were not enough to
catch the error I had.
One idea is to have a script that automates packaging, installing, and checking integration of the external world. It should be automated, since the release test is taking longer and longer as more external programs interact with Gauche. I'm curious how other projects manage the release testing.
Post a comment