(maybe) put hypothesis back
Opened this issue · 4 comments
#586 removed hypothesis, but left the door open for re-integrating it if we can address a couple of issues:
- remove random generation from gating PR CI
- ideally, run any random stuff in a timed action of some kind, that we can peruse to discover new tests
- fix the bug in Hyperlink's strategies that generates bogus URLs python-hyper/hyperlink#153
- reduce the selection of sample data being used so that its doesn't double the amount of time we spend in CI. A little overhead might be worth it for the functionality that Hypothesis is giving us, but not multiple minutes per run.
fix the bug in Hyperlink's strategies that generates bogus URLs python-hyper/hyperlink#153
I fixed this almost a year ago with absolutely no response from the maintainers. Any ideas how to actually get any changes in to Hyperlink?
👋 I can't do much about the hyperlink issue, but happy to help you find a hypothesis workflow you like. It sounds like this suggestion from @nelhage might work well:
- use Hypothesis' settings to run only the
explicitphase, i.e. inputs in@example(...)decorators. This is fully deterministic, and basically equivalent to@pytest.mark.parametrize(). - have a separate cronjob in CI which runs just the Hypothesis tests on
trunkwith all phases, to find new failures which you can add to those explicit decorators.- see
GitHubArtifactDatabasefor easy local replay - since last week, Hypothesis can dump patch files to auto-add failing examples to your code - this is automatic but requires optional deps to be installed
- or try HypoFuzz to get coverage-guided fuzzing in this step - patching in covering
@example()s is planned (but no eta)
- see
Alternatively, using e.g. max_examples=20, derandomize=True in your CI settings might be sufficient to meet your goals.
fix the bug in Hyperlink's strategies that generates bogus URLs python-hyper/hyperlink#153
I fixed this almost a year ago with absolutely no response from the maintainers. Any ideas how to actually get any changes in to Hyperlink?
Hmm. Apparently I'm a maintainer so maybe I can do something about that. Are you referring to this? python-hyper/hyperlink#178 Was it ever passing CI?
fix the bug in Hyperlink's strategies that generates bogus URLs python-hyper/hyperlink#153
I fixed this almost a year ago with absolutely no response from the maintainers. Any ideas how to actually get any changes in to Hyperlink?
Hmm. Apparently I'm a maintainer so maybe I can do something about that. Are you referring to this? python-hyper/hyperlink#178 Was it ever passing CI?
That's the one, yes. I don't remember any of the details.