Test failures not correctly recognized

[novoj]

Hello @mikepenz, thank you for the plugin!

Unfortunatelly it seems that the JUnit 5 test failures are not correctly recognized by it. The configuration is as follows: https://github.com/FgForrest/evitaDB/blob/dev/.github/workflows/test-report.yml

But even if the tests fail: https://github.com/FgForrest/evitaDB/actions/runs/4808320797/jobs/8558154108
The report render the result as if everything is okay: https://github.com/FgForrest/evitaDB/actions/runs/4808320797/jobs/8558296700

Is there anything wrong in the configuration or there is a real issue in the plugin? Thank you for your support!

[mikepenz]

Based on the log I don't see a test failure, just a compilation failure.

Searching for Failures: 1, ...Failures: 9 did not result in any matches (all logs report Failures: 0)

[novoj]

Sorry, I referred wrong runs. This is correct:

https://github.com/FgForrest/evitaDB/actions/runs/4880097093/jobs/8707336172

It contains:

Error:  Failures: 
Error:    FilterIndexTest.deliberateTestFailure:64 Deliberate test failure.
[INFO] 
Error:  Tests run: 3143, Failures: 1, Errors: 0, Skipped: 13

Yet I see:

In following step of report generation: https://github.com/FgForrest/evitaDB/actions/runs/4880097093/jobs/8707457826

I'm uploading the output XML and downloading in the other workflow, but I double checked it and it should download the proper one. I also manually opened the artifact: https://github.com/FgForrest/evitaDB/suites/12666156144/artifacts/679089727 and I see the failure recorded there.

[mikepenz]

Looking through it, it seems the wrong report may be picked. 🤔

From the failed report I can see this order:

However when looking what the action saw is this:

I don't see a reason why it would not find pick it.

If you place the publish step right after the tests were run (without the downloading) does it work?

The failure build is here:
https://github.com/FgForrest/evitaDB/actions/runs/4880097093/jobs/8707336172

However it seems to have picked up:

With 4892719755 seems to be a different run id? (perhaps)

[novoj]

Ok, I'll try to move the the job to the same workflow to avoid artifact passing, and inform you about the result.

I'm still a newbie in GitHub actions, so there is high chance I missed something - most of my workflow code is copy & paste from different sources and guesswork.

Thank you very much for your time!

[novoj]

I have tried to add it to the main workflow, but the result processing took extremely long time, so I have to cancel it to stop burning my free time on GitHub: https://github.com/FgForrest/evitaDB/actions/runs/4908927186/jobs/8765006012

The workflow was simple: https://github.com/FgForrest/evitaDB/actions/runs/4908927186/workflow

What can take it so long to process a few XML files? Build with tests took around 7 mins, processing (until I cancelled it) took over 15 minutes:

[novoj]

I minified the plugin configuration to:

    - name: Publish Test Report
      uses: mikepenz/action-junit-report@v3
      if: success() || failure() # always run even if the previous step fails
      with:
        report_paths: '**/TEST-*.xml'
        fail_on_failure: true

And now the parsing took only 7 seconds and it found the failing test ok: https://github.com/FgForrest/evitaDB/actions/runs/4909112592/jobs/8765311158

Probably the annotations took long? I don't know.

But your suggestion is most probably correct and the plugin download something else than what's uploaded. I'll investigate more. Thanks.

[novoj]

I got it! The problem was in missing run_id configuration in download plugin. The correct configuration is:

      - name: Download a test results # download `evita-server.jar` artifact if the workflow we react to was successful
        uses: dawidd6/action-download-artifact@v2
        with:
          workflow: ${{ github.event.workflow_run.workflow_id }}
          run_id: ${{ github.event.workflow_run.id }}
          name: test-results          # artifact name

Thank you for your support and I'd like to buy you a coffee. I hope, you don't mind.

[mikepenz]

Thank you very much @novoj