Collaborative testing of Common Lisp libraries.

Test Results | Reported Issues: launchpad issue tracker; (old issue registry: docs/issues-found.org).

(Note: this README file is a work-in-progress)

The Goal

Improve stability of the Common Lisp ecosystem by performing automated tests on as wide set of envionments as possible. Environments vary mainly in 3 dimensions:

Common Lisp implementations. They have incompatibilities, sometimes allowed by the standard (features specified as implementation-dependent), somtimes due to bugs, and sometimes in non-standartized features (threading, sockets, FFI, etc.). Moreover, impementations are evolving over the time, we should care about different versions of the same implementation.
Growing number of CL libraries (which are also evolving over the time).
Compatibility between library versions - libraries depend on other libraries and work correctly only with particular versions of the dependencies.

Therefore, when we run test suite of some library, we can speak about success or failure only in context of given Common Lisp implementation, and versions of all the dependency libraries.

Lets call the set of libraries with specified versions a “lib-world”. Important example of lib-world are Quicklisp distros.

It is hoped that constantly running tests on wide variety of environments will help the CL community by:

Fast response to the library authors in case new changes cause bugs on implementations not available to the author. Otherwise author may receive notification about the bug years after he made the change, and thus the cost of fixing the bug may be much higher than fixing it week or two after the change.
The same benefit for CL implementors - when they release new version, run test suites of large number of libraries and quickly detect possible regressions in the new release.
Help to discover and maintain compatible set of library versions (e.g. Quicklisp distros).

Limitations.

Of course, we should understand that test sute success does not always mean the library is workable - there might be bugs which are not covered by the tests. And the other way around - failed tests not always means the library is broken - it may be just a bug in the tests themselves. Reducing this gap increases the utility of automated testing.

The Implementaion Idea

Everyone can run a simple command which will run tests of Common Lisp libraries and upload results to the central server.

That way, instead of setting up a central build farm with all the possible hardware/OS/Lisp implementation combinations, we provide a way for Common Lisp users to contribute test results from their systems, and collectively monitor the CL world. (Of course, if anyone whould want to setup a test farm, cl-test-grid simplifies this task too, by providing required building blocks).

Status

We have function test-grid::run-libtests wich runs tests of the CL libraries. Quicklisp is used to download the libraries. The test results are uploaded to the central server and published as reports here: http://common-lisp.net/project/cl-test-grid/. Clicking library test status (OK/FAIL) in any report refers to the library test logs with the failure details.

We have considered 113 libraries. 56 of these libraries have appropriate test suites (fully automated, no user interaction is needed) and are added to the cl-test-grid.

File run.sh.sample shows what parameters to use for particular CL implementations so that it bootstraps quicklisp in a local directory and updates it to the recent version (if necessary), and then preforms test-grid::run-libtests using this local quicklisp installation. The CL implementation parameters suppress loading the standard init files of the implementaion (to avoid interference with the quicklisp installation user might have on his computer).

Participation

Running tests

The most appreciated way to participate.

The first time you will need do the following:

git clone git://github.com/cl-test-grid/cl-test-grid.git
cd cl-test-grid
cp run.sh.sample run.sh; chmod +x run.sh
Edit the run.sh (you may need to change paths to your CL implementation(s)).
./run.sh

Next time all you need is to just start ./run.sh. It will update the cl-test-grid from git, update quicklisp to the latest distro (and of course run tests and upload the results).

We are looking for contributors who would agree to run this command periodically, for example once a month.

Note also, that this is not 100% automated system. For example uploading results uses drakma. But if drakma is broken on your planform and the upload fails, you will be asked to send us the resuls manually. Or if some test sute hangs, you will need to kill the lisp process.

If you decided to give this project a try and run tests, feel free contact us, we will be glad to assist you (actually it is even recommended, because the project is in the early stage of the development and problems are possible; and also we are interested to see how it goes and how to make test running more convenient).

Discussing the project

Feedback, discussions of the apporach and suggestesion for the open problems are very welcome.

Everyone interested is invited to the “mailing list” - http://groups.google.com/group/cl-test-grid.

Examples of the problems which need solution:

Currently we run tests only on the quicklisp release. But it is very desirable to run tests on the latest library versions from the source control too. For example if we found a bug and the library author has fixed it, he might want to issue a request to cl-test-grid to run tests of the recent version of his library on all the platforms available. This feature would also help to ensure quicklisp distro quality before releasing the distro.
Protecting from hanging tests or crashes.
All the test sutes run in a single Lisp session. If some test hangs, or crases unrecoverably (core dump for example), it will block other tests.

Possible solution would be to run every test as a separate process, but it would require a portable way to start lisp process and making it run particular lisp code.
Distributing the work. Currently we considered only 50 libraries and 23 of them have test suites appropriate to be added to the cl-test-grid. Running these 23 test sutes (including download and compilation) takes 6-10 minutes on a single CL implementation.
Quicklisp today contains almost 650 libraries. Based on this, we may estimate that running all the test suites available in quicklisp may take around 1.5 hours on a single implementation.

This work might be distributed between several contributors - everyone runs part of the tests.

[Today this feature has quite low priority]
…

Adding more libraries

It is quite easy - few lines of code.

Look how the library tests are started in the asdf:perform method for asdf:test-op defined in the library .asd file. Then use the same approach to define a method test-grid::libtest eql specilized for that library and send us this code.

See examples for the already added libraries in the test-grid.lisp.

Today adding more libraries has low priority because we are testing how cl-test-grid works, polishing corner cases and reporting already found bugs to the libraries/CL implementation mainteners. After we stabilize these libaries, we can add more.

Lot of things may be done in this project. But the project has no independent value, it is only useful if it helps to improve the CL ecosystem quality. Fixing bugs in the CL libraries, writing more tests is the most important.

fix in this README:

Terminology - I say “quicklisp distro”, but if be precise, quicklisp calls it “quicklisp distro version”. But if I say “lib-world is a set of libraries with specified versions. An example of lib-world is a quicklisp distro version” the word “version” is repeated twice with diffirent sense - confusing.

d35h/cl-test-grid