ImperialCollegeLondon/Visual2

Add testing framework

Closed this issue · 3 comments

tomcl commented

There are three jobs here:

  1. Implement F# single instruction .Net tests for the emulator against Visual. Mostly random input but with random numbers chosen to exercise corner cases.

  2. Add code to write test inputs/outputs from 1. to a test file and read them under the renderer project in FABLE - redoing the testing to make sure that FABLE number issues have not changed anything.

  3. Implement some whole program ad-hoc integration tests to check ExecuteTop

tomcl commented

I've integrated VisualRandomTestGen. This generates files of pseudo-random assembler instructions with Visual simulation inputs/outputs. these can be checked against Visual2 simulation using the Visual2 GUI help-> run emulator tests.

The testing is not yet complete - but adding new random tests is easy by modifying random.fs code in VisualrandomTestGen. Thus far DP instructions are well tested.

Testing is particularly needed because FABLE causes uint32 code to fail whenever uint32 arithmetic leads to overflow with 64 bit words. These overflows are in some (most?) circumstances kept as part of the uint32 type and can cause equality tets etc to fail.

See wiki Testing page for more information.

tomcl commented

The random test generation project VisualRandomTestGen has now been refactored as a test generation DSL, and the DSL at least vaguely documented in the wiki.

Memory (single register) tests have been added, and debugged using Visual2 however there are still a lot of failing memory tests due to Visual2 lack of scaled register access. Some work to do here, which you can see by running the list of standard tests.

Memory (multiple register) tests not yet added.

The set of tests can be found under ./test-data and help -> run emulator tests will run them all. You can see summary results on console and detailed results under ./test-results

So the testing is in good shape but more work both generating tests and debugging Visual2 against them would be worthwhile.

tomcl commented

More tests added, and the framework enhanced to do this and also make test results more informative. I'll close this now because the issue is no longer test framework, but what to test....