Run CWL conformance tests
Closed this issue · 4 comments
CWL provides a suite of conformance tests to demonstrate how an implementation conforms to the spec: https://github.com/common-workflow-language/cwltool/blob/master/cwltool/schemas/CONFORMANCE_TESTS.md
I've started to explore how to run these:
python -m venv venv
source venv/bin/activate
pip install calrissian cwltest
cd cwltool/cwltool/schemas
git clone git@github.com:common-workflow-language/cwltool.git
./run_test.sh RUNNER=calrissian -n1 --verbose --EXTRA="--max-ram 16G --max-cores 8"
Above attempts to run the first test, but fails:
$ ./run_test.sh RUNNER=calrissian -n1 --verbose --EXTRA="--max-ram 16G --max-cores 8"
./run_test.sh: line 69: eval: --: invalid option
eval: usage: eval [arg ...]
--- Running conformance test v1.0 on /Users/dcl9/Code/python/calrissian-conformance/venv/bin/calrissian ---
calrissian 0.5.0 (cwltool 1.0.20181217162649)
cwltest --tool /Users/dcl9/Code/python/calrissian-conformance/venv/bin/calrissian --test=conformance_test_v1.0.yaml -n1 --verbose --basedir /Users/dcl9/Code/python/calrissian-conformance/cwltool/cwltool/schemas/v1.0 --
Test [1/128] General test of command line generation
Test 1 failed: /Users/dcl9/Code/python/calrissian-conformance/venv/bin/calrissian --outdir=/var/folders/7t/q8bb2np92p5b26pk_h5lr0lh0000gn/T/tmp3sqc2xvb --quiet v1.0/bwa-mem-tool.cwl v1.0/bwa-mem-job.json
General test of command line generation
Returned non-zero
0 tests passed, 1 failures, 0 unsupported features
1 tool tests failed
Notes:
- The first failure is due to the
EXTRAoption. I'm not sure if it's being used correctly - Tests likely need to run inside a kubernetes cluster so that local path lookups are correct and data is available
- Should look to other CI examples that run these conformance tests for how they are run automatically.
No -- in front of EXTRA
@mr-c Thanks for the tip. I guess I didn't make the same mistake the second time 😄 :
The initial exploration is done and I can run the tests (passes 122/128 for 1.0). Now I'm looking at how to run in a CI environment and automate to run the other verisons.
@dleehr 🙂 I presumed you figured it out, but I thought it would be good to make it clear here as well
The v1.1 tests are at https://github.com/common-workflow-language/cwl-v1.1