Automated Testing / CI
Closed this issue · 11 comments
We should add some automated testing and build up a collection of test cases.
I've been thinking about this for a while. I think it would be nice to keep the tests and the expected results in one place, i.e. in the same file or under same directory, and run tests on all cases by traversing a directory (optionally flatten it beforehand).
Meanwhile, it's possible to add a github workflow to test once pushed to CI. There can even be a post-commit check.
This is really two tasks:
- Get
cargo testto do something - Get the CI set up
Am I missing something or is there no simple way to make cargo test just run the executable against a list of example files? If that's the case it might be easier to virtualize the evaluator (WIP locally) so that we can make them normal integration tests that just check for non-zero exit code.
On my previous attempt I used walkdir crate to traverse at runtime in tests. To utilize cargo test, we may need to find a way to traverse the file system at compile time and generate the tests IMO. In the worst case, we may need to add the paths by hand.
Sorry what does walkdir have to do with it? I'm fine with hard-coding the names of the files.
What I'm saying is that all I wanted cargo test to do is for each test to run zydeco run FILENAME and considers it a pass if the exit code is 0 and a fail otherwise. But it seems like integration tests don't support that easily, because you would need to start a new process for each test, which would require you know the executable is built and where it is etc, which would be trivial to write as a separate script but I would prefer to integrate it so that cargo test works nicely and handles that kind of stuff automatically for us.'
Oh I was thinking about just including each zydeco source file and feed them to a buffer and call the run function directly, but then we can't handle exit for us.
It seems that we are not the first ones to deal with the issue: rust-lang/rust#47506. I suggest that we can try rusty_fork to spawn the processes and call run in it. A script should also work, I wouldn't have opinion on using one.
And since the issue in rust repo is still there, I guess there is indeed no easy way to just call our binary.
Too bad. I feel like pulling in an external dependency like rusty_fork and figuring it out is probably nearly as much work as virtualizing the evaluator and less useful so I'll just work on virtualizing the evaluator today instead.
We'll just stick with using a script and spawn a new process in every test then?
No I'm saying we'll virtualize the evlauator so it doesn't actually exit so we can mock the tests instead.
Oh I see... Yeah that's the ideal approach.