exercism/v3-launch

Build Test Runner

ErikSchierboom opened this issue · 12 comments

ErikSchierboom commented

In Exercism v3, one of the biggest changes is that we'll automatically check if a submitted solution passes all the tests.

We'll check this via a new, track-specific tool: the Test Runner. Each test runner is track-specific. When a new solution is submitted, we run the track's test runner, which outputs a JSON file that describes the test results.

The test runner must be able to run the tests suites of both Concept Exercises and Practice Exercises. Depending on the test runner implementation, this could mean having to update the Practice Exercises to the format expected by the test runner.

Each track must build a test runner according to the spec. For tracks building a test runner from scratch, we have a starting guide and a generic test runner that can be used as the base for the new test runner.

ErikSchierboom commented
neenjaw commented

PHP and Elixir have working test runners. Elixir is to full spec, php is to reduced spec

NobbZ commented

@ErikSchierboom can you please check the version from the WIP PR exercism/erlang-test-runner#7?

I know that it currently can not run some of the tests (4 of the practice exercises are failing) but I hope I'll be able to tackle that during the weekend.

ErikSchierboom commented

@NobbZ I currently can't test Erlang due to it not syncing. I've opened a PR to fix that: exercism/erlang#464

NobbZ commented

I have merged the PR.

ErikSchierboom commented

@NobbZ The Dockerfile would not build on my machine, which was due to the image aliases being in uppercase. After changing FROM ERLANG as BUILDER to FROM erlang as builder (and the other image aliases too) I could build it. Running the test runner on the hello-world exercise fails with:

Exercise: hello-world
Solution: /mnt/exercism-iteration/
Output:   /mnt/exercism-iteration/
escript: exception error: bad argument
  in function  binary_to_existing_atom/2
     called as binary_to_existing_atom(<<"example_tests">>,utf8)
  in call from etr_runner:get_test_module_from/1 (/app/src/etr_runner.erl, line 27)
  in call from etr_runner:run/1 (/app/src/etr_runner.erl, line 6)
  in call from erlang_test_runner:main/1 (/app/src/erlang_test_runner.erl, line 15)
  in call from escript:run/2 (escript.erl, line 758)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_em/1 
  in call from init:do_boot/3

Is hello-world one of the 4 failing practice exercises?

BTW If you like to easily test the Docker container, you could add a script like this: https://github.com/exercism/vimscript-test-runner/blob/main/bin/run-in-docker.sh It will allow you to do ./bin/run-in-docker.sh <slug> <input-dir> <output-dir>

NobbZ commented

Weird that you need the stage-names/aliases in lowercase… I was under the impression to have read some document saying the should be uppercase… Can't find it anymore, though good to know that this might fail.

So far I only observed podman acting weird on casing…

Re the error you are facing now, if I understand my code correctly, this means that there is an src/example.erl in the input and the runner searches for the corresponding test module, alternatively, you have an src/hello_world.erl which first line is -module(example). (or something semantically equivalent).

I really should wrap some error handling (later) that will give some proper error message on unexpected file/module names.

The currently known to fail are:

  • robot-simulator
  • nucleotide-count
  • rotational-cipher
  • sublist

They all use some lazy test generation mechanism, which I have not yet implemented in the testrunner.

And thanks for the vim-script-runner link. I'm already doing something similar in my test-suite:

https://github.com/NobbZ/erlang-test-runner/blob/main/test/runs_in_docker_SUITE.erl#L213-L224

Sole thing that is missing is the explicit type=bind and the global --read-only as well as the tmpfs mount.

Therefore I really think that the docker itself is working (except for the stage-names).

ErikSchierboom commented

@NobbZ Let's continue this discussion in exercism/erlang-test-runner#7

neenjaw commented

@ErikSchierboom I'm playing around with making a crystal test-runner. :)

ErikSchierboom commented

@neenjaw Brilliant! Let me know if you need any help.

neenjaw commented

Will do, it's going alright so far!

  • 👍1