justinmeiners/lc3-vm

Tests for testing correct implementation?

madig opened this issue · 6 comments

madig commented

Hi!
First, thanks for the guide, I enjoyed it 😃 I am wondering if there are test-images or something that extensively test if things are implemented correctly? The code I came up with does not match your example code exactly and I'd like to see if would make a difference in a hypothetical "compliance test suite".

justinmeiners commented

Other than the two games we don't have a test suite. It is a good idea though since there are a few instructions and trap calls we do not use.

One thought I have about writing such a suite is that it needs to be careful about what it decides to enforce. My LC-3 does not follow the LC-3 specification. For example trap codes are not implemented in assembly, and so they behave internally differently. One could write a program that depended on this behavior and it would not work.

We want to encourage flexibility of implementation, with a reasonable expectation of program compatibility. With this in mind I think the test suite should just verify the basic operations of the trap codes and instructions.

@rpendleton are you interested in this?

  • 👍3
rpendleton commented

I'm interested in this. I don't know exactly how we'd want to implement these tests (comparing output? or writing a test runner in LC-3?) but I'll look into it.

justinmeiners commented

I now have an opinion on implementing tests.
The tests should just be an LC-3 program which tries to use each instruction and mode and outputs some results. We can then write a simple shell script to verify output is correct. Of course, failure to pass tests will probably result in things like infinite loops.

Is anyone interested in writing an LC-3 program like this? Perhaps there is one out there?

justinmeiners commented

We want to do this. Here is what I want

  • Create a src/test.asm program in lc3 assembly which tests basic operations of each instruction and variant based on the specification included in the supplies.
  • At each test step make logs to stdout using Puts which demonstrate that the test is working.
    For example, addition might add 2 numbers, and print the resulting sum.
  • Create a phony makefile target which runs each unix variant of the lc3 program and uses
    cmp(1) to test that stdout matches a previously verified implementation.
  • Add some comments to the makefile explaining that this test does.
  • Change the end of the tutorial to mention that the test is available.
fizaan commented

"..trap codes are not implemented in assembly, and so they behave internally differently. One could write a program that depended on this behavior and it would not work."

Hmm..this program when run on the official LC-3 simulator will never HALT (unless R7 is saved first then restored). But it does halt when tested in this implementation of the lc3-vm.

.orig x3000

jsr print
halt

print
 ; st r7, temp
 ld r0, char 
 out
 ; ld r7, temp
 ret

char   .stringz "5"
temp   .fill x0000
.end
rpendleton commented

Hmm..this program when run on the official LC-3 simulator will never HALT (unless R7 is saved first then restored). But it does halt when tested in this implementation of the lc3-vm.

@fizaan Good catch. I've created #52 to fix this and will have a PR up soon.

Even though we're implementing the TRAPs directly in the VM (as opposed to relying on the assembly-based implementations), we should still be changing R7 as part of the TRAP instruction since that's part of the specification. The fact that R7 doesn't change in our implementation is a pretty obvious difference in behavior and, as you mentioned, leads to programs that work in the article's implementation of the VM but not the official one. So I think it's important to fix.

  • 👍1