rpc-compat: Run `speccheck` on non-deterministic tests
lightclient opened this issue · 4 comments
Some methods are not deterministic across clients. For example, eth_gasPrice returns as gasPrice in wei that users could use for a transaction to be included quickly. This is subjective based on client so a static test case doesn't make sense. However, we know what the shape the request and response should look like. So it would make sense to make the request anyways and verify the round trip against the spec.
I've written a tool speccheck which more-or-less does this. However it doesn't yet have the ability to verify round-trips that aren't written to file in the format of the test fixtures. So it would need to be librarified to a degree and then imported into rpc-compat and run against certain non-deterministic endpoints (maybe even all of them!).
I'm adding a speconly flag in ethereum/execution-apis#520 so we can at least mark the tests where this kind of checking should occur.
This is a really nice idea, I am happy to begin using it.
I'm working on it right now!
Parsing of the flag is now implemented in the simulator, but it's unclear to me how to proceed with adding speccheck. I don't want to depend on rpctestgen from the simulator, it defeats the purpose of having the generator and test output separated.