json file rewrites when run tests in few browsers simuntaneosly
maximkoev opened this issue · 39 comments
Hi guys, you created really good tool, I like it. Could u please help me with my problem. When I run tests in few browsers or run tests again, roport.json deletes and creates each time. how can I avoid it?
Hi @maximkoev, thank you for your feedback. It is possible to have a report filename that is different at each run. The question is: how to do you want to show the report: should it be the merge of each report files, should it be the last generated report file?
Hi @hdorgeval in the best case I want to see report with history. So I want to know when case passes and when failed.
Hi @hdorgeval
Maybe to give you an example of this issue consider the following scenario(I am facing a similar situation):
- Using a test case runner class to perform parallel testing with BrowserStack.
- If you instance different web browsers in the same runner, it ONLY displays the first browser result ( JSON only have that result). Therefore the rest of the tests are missed in the final report.
*Note: Maybe he might be thinking about the possibility of having a different report per browser or have an HTML report with all the results.
Attaching a test runner example of that particular case.
Hi @JoanEsquivel
I am using this report and running on browserstack .
The issue is it doesn't show the browser name and device name properly which was Edge and Windows.
Would be great you can help me on this
Hi @vasantverma-LM , if you can provide the generated report.json file, it will help me to solve your issue concerning the browser.
Hi @hdorgeval Thanks for your response.
Regarding the screenshot I will create a sample project and would the report.json file in some time.
For this I will create a separate issue.
@hdorgeval
Do you have any solution for getting the test results in the report when multiple browsers are running simultaneously.
As mentioned by @JoanEsquivel only the report only displays the test results of the first browser
Hi @vasantverma-LM : yes I plan to solve this issue as soon as possible :). I will start the dev this week-end.
Hello guys, thank you for your feedbacks and your patience!
I have published a v6.0.0 version to the npm registry that should solve this issue.
Please read the CHANGELOG, as this release is a breaking change.
It would be awesome if you could provide some feedback on this new release.
@hdorgeval Thanks for the latest release.
I am currently using it in my sample project. Till now its working fine.
I have encountered one issue in the latest version.
When viewing the error log displayed in the report, there is a symbol which repeated frequently. Due to this the error log is not readable.
Attached are the screenshots of version 5.1.0 and latest version 6.0.0.
Hi @vasantverma-LM , thanks a lot for your feedback!
This is a side effect caused by the removal of the report.json file declaration together with the reporter. If you put back the file declaration after the reporter, it should work.
This problem is due to the fact that before version 6, the json file was 'cleaned' and written by testcafe itself.
From version 6, the file is entirely managed and written by the reporter itself and not by testcafe.
I fix this issue asap.
Hi @vasantverma-LM, I have published a v6.0.1 that should solve the issue.
Hi @hdorgeval the error log issue in the report is resolved in the v6.0.1 but the enhancement added in the v6.0.0 is not working.
I tested it on v6.0.0 and v6.0.1.
In v6.0.0 I am able to run at least three browsers simultaneously and the report was generated accordingly but with the same scenario in v6.0.1 browsers are not getting closed after successfull execution.
Hi @vasantverma-LM, I cannot reproduce locally on my mac your issue. Which browsers are you using? Did you try to rimraf the node_modules and re-install everything?
Hi @hdorgeval I have tested it again and it seems that the issue that I have mentioned earlier is intermittent and occurs only when we are running tests on three or more browsers simultaneously.
Hi @hdorgeval ,
Thanks for your update.
There is one query related to the showing the screenshots in the report.
Suppose I have executed a test on chrome, edge, and firefox simultaneously. JSON files for these three browsers are created.
When viewing the screenshots of the test in the reports for a particular, it shows screenshots of all other browsers also.
I have specified below mention naming techniques for screenshots.
Is there any way to show screenshots of each browser separately in the screenshots sections of the report.
Hi @vasantverma-LM , I have pushed a v6.0.2 version that maybe could resolve that browser does not close at the end of the tests.
Concerning the screenshots, I need some more time on my side.
Thanks!
Hi @vasantverma-LM, I am working on your issue. I will release a new version in a few days.
Hi @vasantverma-LM , Thank's a lot for your feedback, as it helped me to significantly improve the reporter.
I have pushed a v6.0.3 version on npm. This version should correctly handle error messages and screenshots for each browser when TestCafe runs with multiple browsers simultaneously.
There is still some improvements to implement:
- insert TestCafé warning messages when a screenshot could not be taken
- If you do not see a screenshot in a failed report, have a look inside the json file at a section called
runInfo: you should have a warning sub sections that will explain why the screenshot could not be taken
- If you do not see a screenshot in a failed report, have a look inside the json file at a section called
- add a column in the HTML report to show the execution date, since you may have multiple entries now for the same feature
Thank's for your feedback on this new version :)
Hi, @hdorgeval I have tested the reporter v6.0.3 with chrome, edge, and firefox simultaneously and it is not showing the screenshots for each successful test. In my test code, I have added the code for capturing the screenshot at the end of every test. The screenshots are captured but not shown in the report for each browser.
For failing tests, screenshots are shown for the respective browser.
Refer screenshots.
Hi @vasantverma-LM , thanks a lot for your feedback.
I have published v6.0.4 that should solve your issue.
Hi @hdorgeval ,thanks for the update.
I have tested the latest version 6.0.4 and it is working fine when running tests locally but when testcafe tests are executed on BrowserStack , screenshots(both failure and success screenshots) are captured in testcafe but not shown in the report.
The above mentioned observation is for running multiple browsers simultaneously.
Attached is the screenshot of my TestRunner.js file
Hi @vasantverma-LM : thanks again for your feedback. To solve your issue I need the json files generated by the reporter. Do you think it is possible on your side to forward me at least one of these files?
Hi @hdorgeval I will try to reproduce the same on a sample testcafe project and then will share the json files.
Hi @hdorgeval , attached are the json files for tests executed on BrowserStack.
v6.0.4 Issues.zip.
Attached below is the screenshot of the report:
As shown in the screenshot, for iOS and android the OS column shows ? unknown .
During test execution, I got testcafe-reporter-cucumber-json: cannot get the Platform name from input string 'Chrome 83.0.4103.106 / Android 10 error message in the terminal.
I have a also attached the json file for iOS device.
Hi @vasantverma-LM, thanks for the data. Unfortunately I suspect this is the way TestCafé handles the userAgent on BrowserStack. I will publish a new version that will enable you to output debug data on the console.
I will ping you when this version is ready.
Thanks!
Hi @hdorgeval , thanks for the update.
Also plz provide a fix the screenshot issue not showing in the report when tests are executed on BrowserStack. The necessary files I have shared with you.
Hi @vasantverma-LM , I have published a v6.0.5, that, I hope, will fix your last issues.
Hi @hdorgeval I have tested the latest v6.0.5 but when executing testcafe tests on BrowserStack, screenshots are still not shown in the report
Hi @vasantverma-LM , you should have console logs like : cannot match browser...
Hi @vasantverma-LM , I have publish a v6.0.6. If screenshots are not included in the report, please have a look inside the JSON report file to a section runinfo at the end of the file:
"runInfo": {
"warnings": []
}The warnings section should give you the hint why TestCafé did not take the screenshot
Hi @hdorgeval ,I have tested the reporter v6.0.7 and when running the testcafe tests, screenshots were captured but not shown in the report.
Following are the warning message displayed in the json file,
"warnings": [
"The window maximization functionality is not supported by the Browserstack JS Testing API. Use the Browserstack Automate API.",
"Was unable to take a screenshot due to an error.\n\nError: Unable to locate the page area in the browser window screenshot at C:\Users\vasant.verma\Desktop\TestcafeFramework\MakeMyTrip\screenshots\2020-07-17_13-00-30\test-1\Chrome_83.0.4103.61_Windows_10\LoginPageTest-1.png, because the page area mark with ID 1058310196 is not found in the screenshot.\n at calculateClipInfo (C:\Users\vasant.verma\Desktop\TestcafeFramework\MakeMyTrip\node_modules\testcafe\src\screenshots\crop.js:114:19)\n at Object.cropScreenshot (C:\Users\vasant.verma\Desktop\TestcafeFramework\MakeMyTrip\node_modules\testcafe\src\screenshots\crop.js:131:18)\n at C:\Users\vasant.verma\Desktop\TestcafeFramework\MakeMyTrip\node_modules\testcafe\src\screenshots\capturer.js:137:40\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:97:5)\n at Capturer._capture (C:\Users\vasant.verma\Desktop\TestcafeFramework\MakeMyTrip\node_modules\testcafe\src\screenshots\capturer.js:118:9)\n at Capturer.captureAction (C:\Users\vasant.verma\Desktop\TestcafeFramework\MakeMyTrip\node_modules\testcafe\src\screenshots\capturer.js:170:16)\n at BrowserManipulationQueue._takeScreenshot (C:\Users\vasant.verma\Desktop\TestcafeFramework\MakeMyTrip\node_modules\testcafe\src\test-run\browser-manipulation-queue.js:54:20)\n at BrowserManipulationQueue.executePendingManipulation (C:\Users\vasant.verma\Desktop\TestcafeFramework\MakeMyTrip\node_modules\testcafe\src\test-run\browser-manipulation-queue.js:71:24)\n at TestRun.ServiceMessages. [as ready-for-browser-manipulation] (C:\Users\vasant.verma\Desktop\TestcafeFramework\MakeMyTrip\node_modules\testcafe\src\test-run\index.js:931:18)"
]
Hi @vasantverma-LM, screenshot are generated by TestCafé on disk but are not passed to the reporter, hence the warning and the fact that the reporter does not show them.
It is very complex to programmatically analyze the warnings given by TestCafe and find from that warnings which screenshot file is related to which feature/test and to which browser.
For me, it is a bug within TestCafé:
Testcafé takes the screenshot, but does not give it to the reporter because there is an error in post-processing that screenshot. At least TestCafé should give to the reporter the original screenshot.
I suggest you to open an issue in TestCafé repo.
Hi @hdorgeval , thanks for your response.
In the earlier version v5.1.0, screenshots were shown in the report when tests were executed on BrowserStack.Only it was not working for multiple browsers simultaneously.
I will try to report the issue on testcafe gitHub page but I am confused exactly what should I mention on the issue.
Hi @vasantverma-LM, I opened this issue on TestCafé repo.
Hi @hdorgeval ,Thanks
Hi @hdorgeval , I am using the latest version of testcafe v1.8.8 and testcafe-reporter-cucumber-json version: 6.1.0.
When running testcafe tests on BrowserStack, Error is not displayed for the failed testcases in the report.
Screenshot is attached for your reference.
Hi @vasantverma-LM, thank you for your feedback! I have reproduced the problem on my side. I will fix this asap.
Hi @vasantverma-LM , I have published a fix in v6.1.1.
I hope everything will be ok for you.