Testing with synchk unreliable
tremby opened this issue · 4 comments
As mentioned in #18, testing with synchk isn't robust.
Highlighting is different in two successive commits, the synchk files and test cases haven't changed, but it says tests pass on each one. I've tried with my full vim open too, so it's not to do with how I'm invoking it in the test.sh script.
I can't spend time on it just now, but this ticket is to look into possibly improving synchk or looking for an alternative testing solution.
I emailed the author back when I threw it in a repository on Github and never received a reply, so I doubt it's possible to get him to help find a solution.
Have a theory on how to achieve accurate testing. Can use the ToHTML() vim function we spoke about previously and then store the SHA-1 hash of the file. The test suite would then just compare the current current hash value against the stored hash value.
The thing that worries me about that is whether you and I (and others) will get the same output, given different terminals and settings. Maybe it's all the same as long as colour scheme is set to default? We'll find out...
Started poking around with it. There is a way to set force vim to use a specific .vimrc file from the command line, which would be helpful as it would allow to set up a test environment. Problem I immediately ran into though is that TOhtml() writes info about the vim client (version, etc) into the output file. Which would cause differences if different versions of vim are being used.