laike9m/Cyberbrain

Should only create gold data files if cyberbrain ran successfully

Opened this issue · 3 comments

Currently, even if cyberbrain fails, new json files will be created.

Relevant code: https://github.com/laike9m/Cyberbrain/blob/master/test/conftest.py#L128-L141

How do we know whether cyberbrain ran successfully?
if tracer_events==golden_filepath returns True indicates it ran successfully?

@victorjzsun is supposed to improve this, and can provide more context

yield will return whether the test ran successfully or unsuccessfully, we then immediately create/compare golden files with tracer_events.You'd need to find a way to check if it ran successfully, for example, using a hook or doing something like pytest-golden plugin and use a separate command for updating. I haven't done too much investigation regarding this but it might be hard to ensure every test passes before writing all of the golden files