exercism/haskell

What if we could have multiple example files?

petertseng opened this issue · 4 comments

petertseng commented

When reviewing #393 I realized I wish that we could test that multiple signatures would be accepted by the test suite. But of course we have to pick a certain signature in our example file... or do we?

What about a crazy idea - what if we supported having multiple example files?

This is already supported on x-api - any file that has example in its name will be excluded, so we could have Example2.hs, Example3.hs, etc.

Then we would have to make our .travis.yml support it in a reasonable way.

The idea is that Travis CI would test each example file individually (separately from the other examples) and makes sure that the test suite passes for each.

Potential use cases I can think of:

  • Verify that the anagram test suite can accept multiple signatures.
  • (If we also have a way to specify that a certain example should fail the test suite) We can test the quality of our test suite, making sure that it catches various cases we want it to catch. In accumulate, we have an inefficient example noted at https://github.com/exercism/xhaskell/blob/master/exercises/accumulate/src/Example.hs#L15 and we say that it should fail the test suite. But this was never automatically verified and indeed for the longest time that example did not even compile until #209

However, I hope that this doesn't make our tests take too long to run.

Is this idea worth it? Should I try it out to see what happens?

rbasso commented

... If we also have a way to specify that a certain example should fail the test suite

Humm.... I like that!

Is this idea worth it? Should I try it out to see what happens?

Certainly!

Possible implementation

Add a example folder, containing sub-folders for each example solution:

  • `example/someName/ModuleName.hs
  • `example/someName/package.yaml
  • `example/otherName/ModuleName.hs
  • `example/otherName/package.yaml

When testing, Travis could loop the folders inside example and do the following:

  • Move example/someName/package.yaml. to src/.
  • Move example/someName/ModuleName.hs to test/.

I believe that this will not be problematic.

This will only work if the site excludes everything that has example in the full path, not only in the filename. I don't know how it was implemented... 😕

Do you know anything about that?

I still don't know yet how to check for failing tests. Checking it now...

However, I hope that this doesn't make our tests take too long to run.

If the examples use packages already in Stack's cache this will probably not be a problem.

petertseng commented

I still don't know yet how to check for failing tests. Checking it now...

This will be tricky because we have set -e on, and in the case of a should-fail example we want to invert it. I will welcome ideas on making this work.

petertseng commented

This will only work if the site excludes everything that has example in the full path, not only in the filename

We will be OK: Evidence in two places:

  1. x-api source:

https://github.com/exercism/x-api/blob/aa4869e999443e3736d12e77436ac195fccf7e6b/lib/xapi/implementation.rb#L8
https://github.com/exercism/x-api/blob/aa4869e999443e3736d12e77436ac195fccf7e6b/lib/xapi/file_bundle.rb#L24

file bundle ignores using the whole path (despite its name, file is a full path, or at least the entire path starting with the exercise dir, so it will at least include anagram/example/blahblah, tested this)

  1. other tracks

They have directories named example.

  • 🎉1
petertseng commented

I'm not sure we want to move every single exercise's example to this structure. The examples that would benefit are the ones that need a package.yaml and the exercises where we want to have multiple examples. So at least at first I will support both the simple src/Example.hs and the examples directory, I think.

  • 👍1