fir-lang/fir

Start testing the interpreter

Closed this issue · 4 comments

osa1 commented

We should start testing the interpreter.

The tests should be written in a way that is possible to run in the bootstrapped compiler as well.

My current thinking is that we should have everything needed to evaluate output of a program in the same source as the program. The test runner should parse the comments in the file for: exit code, stderr and stdout outputs.

In the interpreter, we can probably use golden-tests or something similar.

In the bootstrapped compiler we can keep using the same library, or port it to Fir.

osa1 commented

I've started adding some tests in https://github.com/fir-lang/fir/tree/main/tests, using golden-tests.

We should add more.

osa1 commented

Two things that I think we should add to golden-tests:

  1. We should make it possible to somehow process stdout/stderr before comparing, e.g. maybe with a line of shell.

    This allows checking stderr in case of type error without comparing the stack trace printed by the error.

    Of course type errors shouldn't be panics, but (1) preprocessing the program output before comparing would still be useful filter out unimportant details (2) panicking may be acceptable in the interpreter, which we want to keep simple.

  2. We should make it possible to run a program multiple times, passing different inputs. For example:

    fn main(input: Str) = ...
    
    # input:
    # ...
    # expected stdout:
    # ...
    
    # input:
    # ...
    # expected stdout:
    # ...
    

    Here the program should be run twice, with the given inputs and expected outputs.

    This is useful in cases like:

    1. We can test example programs this way. Example programs will have a main and print stuff.

      We could add a test mode and another entry for testing, but the program would have to be refactored to return values and we would have to check those values, instead of checking stdout.

    2. When testing crashes/errors, we can't use a test function with assertions as the program will crash after an error.

      Also, we don't have error/exception handling right now and it may take some time before we have the features and standard library functions in place.

osa1 commented

It would also be helpful if we could run the tests without spawning a Fir process for each test, to be able to generate coverage.

osa1 commented

We have a lot of tests in the tests dir now. We also test examples.