llvm/llvm-zorg

Incorrect parsing of the failed LLVM/Clang Unit test names for the failed log titles.

vvereschaka opened this issue · 3 comments

vvereschaka commented

When few tests get failed within the single unit test, the buildbot prepares each log with the same title for each failed test (see screenshots below). As example the check step got few failed tests for ADTTests unit test and all of them got the same LLVM-Unit::93 title. The buildbot needs to get a proper test name for the failure log title.

A list of failed unit test logs with the same titles:
llvm-failed-test-logs

A list of real name of the failed unit tests:
llvm-real-failed-test-list

The screenshots are for the following build: https://lab.llvm.org/buildbot/#/builders/86/builds/66095

To reproduce the problem on the builder in case the provided link got expired:

  • open builder UI: https://lab.llvm.org/buildbot/#/builders/86
  • click Force Build button at the top right corner
  • select revision field
  • enter revision: 4ae51570806ba5c5fcabe6d6dcbe52e3a5d5453b
  • specify a clean build
  • click Start Build button
slydiman commented

The full stdio log explains what happened:

...
PASS: LLVM-Unit :: ADT/./ADTTests.exe/55/93 (53 of 2188)
FAIL: LLVM-Unit :: ADT/./ADTTests.exe/4/93 (54 of 2188)
******************** TEST 'LLVM-Unit :: ADT/./ADTTests.exe/4/93' FAILED ********************
...
 Error msg:
[  DEATH   ] 
********************
PASS: LLVM-Unit :: ADT/./ADTTests.exe/38/93 (55 of 2188)
PASS: LLVM-Unit :: ADT/./ADTTests.exe/64/93 (56 of 2188)
...
PASS: LLVM-Unit :: Analysis/./AnalysisTests.exe/3/85 (112 of 2188)
FAIL: LLVM-Unit :: ADT/./ADTTests.exe/87/93 (113 of 2188)
******************** TEST 'LLVM-Unit :: ADT/./ADTTests.exe/87/93' FAILED ********************
...
 Error msg:
[  DEATH   ] 
********************
PASS: LLVM-Unit :: Analysis/./AnalysisTests.exe/29/85 (114 of 2188)
PASS: LLVM-Unit :: Analysis/./AnalysisTests.exe/30/85 (115 of 2188)
...
PASS: LLVM-Unit :: DebugInfo/DWARF/./DebugInfoDWARFTests.exe/86/87 (608 of 2188)
PASS: LLVM-Unit :: DebugInfo/DWARF/./DebugInfoDWARFTests.exe/84/87 (609 of 2188)
FAIL: LLVM-Unit :: ADT/./ADTTests.exe/90/93 (610 of 2188)
******************** TEST 'LLVM-Unit :: ADT/./ADTTests.exe/90/93' FAILED ********************
...
 Error msg:
[  DEATH   ] 
********************
PASS: LLVM-Unit :: DebugInfo/DWARF/./DebugInfoDWARFTests.exe/82/87 (611 of 2188)
PASS: LLVM-Unit :: DebugInfo/DWARF/./DebugInfoDWARFTests.exe/72/87 (612 of 2188)
...

The log name formatting is the following

name_part = name.rpartition('::')
self.step.addCompleteLog(code + ': ' + name_part[0].strip() + name_part[1] + os.path.basename(name_part[2]), activeVerboseLog)

So os.path.basename('ADT/./ADTTests.exe/90/93') returns 93.

It seems the LIT or LLVM-Unit prints incorrect test names to the full log.
Note the summary contains correct test names at the end of log:

Failed Tests (3):
  LLVM-Unit :: ADT/./ADTTests.exe/PagedVectorTest/EmptyTest
  LLVM-Unit :: ADT/./ADTTests.exe/PagedVectorTest/HalfPageFillingTest
  LLVM-Unit :: ADT/./ADTTests.exe/PagedVectorTest/ShrinkTest

Probably we must report this issue to the LIT or LLVM-Unit instead.

slydiman commented

FYI
https://reviews.llvm.org/D122251#3491642

selected_tests, discovered_tests = gtest.post_process_results(selected_tests, discovered_tests) updates test names before printing the summary.

  • 👀1
gkistanova commented

This issue has been fixed.