Flank/flank

Flank should record fatal exception outcome and or details from a matrix execution

Opened this issue · 1 comments

Hello,

I've noticed that when a Fatal exception occurs on Firebase Test Lab (e.g,
Screenshot 2022-12-19 at 10 54 50 AM These are simply recorded in the matrix_ids.json file as {"outcome": "failure"}. For the purposes of daily reporting on test executions, I would love if Flank were to capture any other keys indicative of crashes when they happen. I know this is possible through the Cloud Tool Results API, but I would like if Flank could do that, as we simply parse artifacts. Even if it's a simple boolean from FailureDetail ({"crashed": true}) that would be helpful.

https://firebase.google.com/docs/test-lab/reference/toolresults/rest/v1beta3/Outcome#FailureDetail

https://firebase.google.com/docs/test-lab/reference/toolresults/rest/v1beta3/Outcome

Hi @AaronMT, after going through the code, I noticed that this information is being captured in the details field of the axes object. In the case of crashes, the details field will display Application crashed as its value.

crashed == true -> "Application crashed"