Code coverage reporting with other unit tests
mjmaix opened this issue · 24 comments
Hi,
I want to view the code coverage of my tests with my regular unit tests. I already use jest code coverage reporting in my unit tests and iI would like to merge add the coverage by detox.
Not sure how we can add coverage by Detox since the tests don't run your JS code in node. Instead, it commands the device and the app to do stuff, and then asserts that something has changed in the native view hierarchy on the device/simulator.
I believe this will never be possible.
Can I just chip in here as a developer evaluating Detox for a company project?
- Code coverage is a huge selling point for a testing solution.
- Measuring coverage in JS is a matter of either using engine features (as e.g. recently introduced in V8) or running an instrumented version of the code (as done by the
istanbulfamily of tools) to record data during execution, then communicate it back to a reporting tool. - It is in fact easy to conceive of Detox driving an instrumented build running on a device/simulator, retrieving the recorded data and feeding it to a standard coverage reporting tool. This would of course come with the standard caveats of instrumentation (mainly reduced performance) but is far from impossible IMHO.
1 - I strongly agree
2/3 - I am still not sure of how to even approach this. Detox 'tester' runs on node, communicating via websocket with the the device ('testee'), a purely native iOS/Android (C, objC, Java) engine that runs on the device.
The react-native javascript code is being executed inside a JScore engine as a part of the application.
The only thing I can think of is somehow calculate how much of the parsed bundle has actually been executed, not sure how accurate a measurement of this kind can be, and I'm even less sure that there are any tools that could be utilized for that.
Do you have an idea of how to approach this ? I'd be happy to an elaborate idea to understand the feasibility of this kind of feature.
@rotemmiz Yes, I think I can help formulate an approach.
Using gotwarlost/istanbul#16 (comment) as a template:
So it looks like you have figured out how to send instrumented code to the browser when running tests.
- In our case we'd need to slip
babel-plugin-istanbulinto the build and ensure correct source maps are generated for the final JS bundle - this is fairly standard JS tooling stuff, 100% doable. (For instance usingBABEL_ENV=e2eand conditionally adding the plugin in the userland.babelrc; once the basic workflow is in place it would be worth looking at minimising the config burden on Detox users - maybe with a custom Babel preset that wires everything up correctly out of the box, or building this into the Detox repackager)
When the code runs on the browser it will update a
window.__coverage__object with stats
- This will be
global.__coverage__in our case.
Once your tests are run, you need to send (POST) this object back to a server that can collect this information.
- Right before test cleanup, we'll need to serialise the contents of
global.__coverage__on the testee (using the local JSCore API to get it) and send it back over a socket connection (or similar) to the Detox Node client. This is the part I know the least about, but I think I'm making safe assumptions about Detox's architecture here.
The rest pretty much applies as-is - The Detox Node client will be able to output coverage files and pretty reports the same way e.g. Jest does it, using libraries in the istanbul ecosystem. Then as a final step one could use a postprocessing tool (or a service like Codecov) to merge coverage reports from the different test runners.
Let's say that your back-end handler writes a coverage.json file every time it gets a call (i.e. coverage1.json for the first set of tests, coverage2.json for the second set of tests and so on)
After all your tests have run you have a bunch of coverage*.json files sitting somewhere
Use the following command line to get reports out of it.
$ istanbul report --root /path/to/dir/containing/coverage/files --dir /path/to/output/dir
Type istanbul help report for more info
Note that there is also a way to use istanbul as a library so that you can keep accumulating the coverage information in memory without writing files etc. I'm just keeping it simple for now.
@rotemmiz Would you like to reopen this issue? I'd be happy to start tinkering with the code with an eye to eventually contributing a PR.
Sure, do it! It sounds amazing if it's doable!
I'm looking for a way to eval() some JS from DetoxManager (be it the iOS or Android one for now). Or another way to access the value of a JS global var (__coverage__), JSON-encode it and send it via the socket. Any pointers?
Why not bundle it with the packager ?
We need to specifically read this variable at cleanup time, after everything has run. I'm starting to think of an approach like react-native-eval's - emitting an event through one of the existing native modules and getting the return value via a callback from JS: https://github.com/artemyarulin/react-native-eval/blob/master/android/src/main/java/com/evaluator/react/RNMEvaluator.java#L168
That would indeed require us to inject some JS code through the packager. Is this possible to automate in Detox?
Not currently, only with repackager
Hmm. Alternatively, maybe I can put together a standalone, pure JS solution for collecting coverage from a React Native app, and we can tackle Detox integration as a second step?
OK, based on my realisation above that most of what's needed here isn't actually Detox-specific (or even React Native-specific), I've started a project I'm tentatively calling bosphorus, which provides tools for serialising IstanbulJS coverage data from running apps over a socket connection. The architecture is, unsurprisingly, inspired by Detox itself 😄. Here's an example of reporting coverage from a basic Expo app running in the Simulator.
No Detox in there yet but I'll be looking into that next - help welcome!
Now, what's needed on the Detox side for a seamless developer experience? Here's what I think is essential:
- Spawn
bosphorus-serverfor the duration of the test run. - Run the equivalent of
bosphorus-cli save-coverageright before cleanup (maybe indetox/src/client/Client.js?)
And then for bonus points:
- Automate adding
babel-plugin-istanbulto the build. - Automate importing and starting
bosphorus-clientin the testee.
Wow, this is amazing! Keeping an eye on that project.
Might interest you guys as a potential solution but I'm getting coverage in-process via my bridge module i'm working on.
TL;DR bridge extends on detox and allows react native code to run inside the node process alongside the tests. Main goal is to provide an easier way for React Native module developers to test their native modules - without hacky solutions or writing a ton of separate native android/ios test code. But it could have other uses cases.
Here's a gif of the core features test suite running, as you can see it's rapid:
And here's a gist of the test suite shown above - gives you an idea of what it's doing and what you have access to - no limits e.g. here's an output coverage report of our react-native-firebase library that uses the earlier mentioned istanbul babel plugin.
Ooh, nice @Salakar! Is that conceptually/technically similar to how the "debug JS remotely" feature works? This is pretty awesome.
@motiz88 yep pretty much that except it automatically enables that for you and takes over as the debugger instead of chrome - even across reloads/relaunches
There's no visible lag or performance hit in doing so that I've noticed either - a quick FPS test I ran the other day maintained around ~55 FPS - so it is lag free and just as performant as the chrome debugger if not more.
Am hoping to have a version published by the end of the weekend.
@motiz88 I have just pushed up my playground app project with this all in if you'd like to take a look: https://github.com/invertase/react-native-firebase/tree/bridge-detox/tests-new
Includes nyc/Istanbul coverage all configured. Bridge code is is bridge dir, I just need to move it out to own repo, document it and publish to npm, but sleep first 😩
(Android only, I still need to configure the iOS project)
EDIT: now with automatic source mapping of stack traces:
I'll say it again @Salakar - that is awesome stuff! I will definitely be tinkering with bridge once it's released.
On the Bosphorus front, still early days (day 2 to be exact!) but I have gotten it to work with Jest + Detox in what I think is a pretty viable way - no changes to Detox required: https://github.com/motiz88/bosphorus/tree/master/examples/expo-app#readme
I'll continue working on the reliability & ease of setup of this solution, and give it a go with Android soon. An npm release will follow when I'm reasonably confident the thing will really work on machines other than mine 😅
@motiz88 cool will keep an eye on your project as well. This might be a useful thing for you; nyc/instanbul can wrap directly around detox cli commands, so as long as you can populate global.__coverage__ it will pick it up - it's pretty much what I do between tests: I collate the debugger context's __coverage__ and merge into global's __coverage__ using istanbul-lib-coverage
Links for reference:
https://github.com/invertase/react-native-firebase/blob/bridge-detox/tests-new/bridge/env/node/coverage.js#L6
https://github.com/invertase/react-native-firebase/blob/bridge-detox/tests-new/package.json#L12
https://github.com/invertase/react-native-firebase/blob/bridge-detox/tests-new/package.json#L51
https://github.com/invertase/react-native-firebase/blob/bridge-detox/tests-new/.babelrc#L8
Well, I just published an alpha to NPM 🙈 https://github.com/invertase/jet - need to document it fully but it's working fully on both platforms.
Wondering if there was ever a solution to getting some metrics associated with detox and code coverage?
Closing this issue. This should come from the community—we don't have the capacity to create or support a full blown solution.