Screenshot testing
wKich opened this issue · 4 comments
Hi. I see you using hermione to visual testing. I can suggest to try use Creevey. It's tightly integrated with storybook, so you don't need write tests explicitly, also it's faster and have better UI.
it's faster
any proofs? can't find any benchmarks in docs
It's pretty obviously, Hermione as the Gemini for each test case load story by URL from scratch, so browser load all static assets and initialize whole storybook bundle every time. So if you use BrowserStack or SauceLabs as we used in react-ui and if you don't have low latency and good bandwidth, it's be significant difference. For us with ~2000 tests with Gemini + SauceLabs we run all tests around an hour, I did small hack for Gemini, to don't refresh browser page for next test case and it goes to ~30 minutes. But after we switched into Creevey all tests runs in around 10 minutes.
It's not simple to get right benchmarks, because for Hermione you need write a lot boilerplate tests code. Or I could try to integrate Creevey into yandex-ui. And we could compare real numbers in real project.
Hermione good tool for e2e visual tests, but if you want to get unit-like visual tests like test your ui-kit with huge amount small test cases, you need to allow switch between fast as possible and don't waste time to write/maintenance test code. So Creevey helps with that.
Hermione as the Gemini for each test case load story by URL from scratch
this ensures test independence. In your case you can't give such a guarantee
so browser load all static assets
it's wrong - browser can reuse already downloaded static. just change your testsPerSession option
For us with ~2000 tests with Gemini + SauceLabs we run all tests around an hour
we can run about 30k tests for 5-10 minutes. It depends on your concurrency settings. Maybe you should do something with your storybook performance? I would like to see your hermione/gemini settings
But after we switched into Creevey all tests runs in around 10 minutes.
Your tool is aimed at a specific case for which you decided to write a separate tool. You claim that your tool is faster, but in fact you just removed the part (in my opinion, very important) of the logic that runs in hermione. But I think that you can do the same from hermione plugin ;)
Hermione good tool for e2e visual tests
hermione is not only about visual testing - you can write any types of browser-based testing
this ensures test independence. In your case you can't give such a guarantee
For React components you could simply unmount/mount components with fresh state. If your components has side effects and you need reload browser page, maybe you have something wrong with your components. If you manually open storybook and go through stories, you can see browser page not reloading every time then you switch story.
it's wrong - browser can reuse already downloaded static. just change your testsPerSession option
I didn't say, that new selenium session is starting for each test case, I said about your open new url every time then you start new test case. So yeah, browser could cache assets, but it compile js anyway.
It depends on your concurrency settings
Yeah, sure. I forgot to say, we have 6 parallel sessions.
But I think that you can do the same from hermione plugin ;)
What plugin you mean? It might be cool, if some plugin could generate tests in runtime from stories. So you don't need write all of that stuff explicitly.
you can write any types of browser-based testing
Creevey could do that as well :)