`-format markdown` isn't 100% markdown and `-nocolor` still outputs colour
alecthomas opened this issue · 9 comments
I'm attempting to use tparse in Buildkite to output a nice test summary, but the output is not 100% markdown. There appears to be normal(ish) test output prepended to markdown:
Which renders like this in Buildkite:
I'm using terminal-to-html to work around the colour issue.
Ouch, that's ugly. Let's fix that.
There are 2 things I've been thinking to address this:
Improve -format=markdown for all output
Currently, the markdown format is applied to tables only, but we should also ensure failures are markdown friendly. This becomes especially important for CI.
Note NO_COLOR=1 is supported today in case you want to disable that.
Add net new flag -ci=[buildkite,github,...] or -ci=true
If the above is done, I'm not sure this is required. But it's a thought I had ever since GitHub introduced GitHub Actions with Job Summaries. With the intention of outputting a "ci summary" tailored for the platform.
This might seem very similar to the above, but the idea is to have a summary that utilizes markdown features to output an opinionated summary
Example
✅ go test has finished, running 363 tests in 1.3 seconds and all tests passed. [...]
sort tests summary
Click to expand!
| STATUS | ELAPSED | TEST | PACKAGE |
|---|---|---|---|
| SKIP | 0.00 | TestSearchWrappersDontAlloc | sort |
| PASS | 0.58 | TestCountStableOps | sort |
| PASS | 0.18 | TestCountSortOps | sort |
| PASS | 0.09 | TestHeapsortBM | sort |
| PASS | 0.07 | TestSortLarge_Random | sort |
| PASS | 0.06 | TestStableBM | sort |
| PASS | 0.04 | TestStability | sort |
| PASS | 0.04 | TestSortBM | sort |
| PASS | 0.00 | TestSearch | sort |
| PASS | 0.00 | TestFloat64s | sort |
| PASS | 0.00 | TestStrings | sort |
| PASS | 0.00 | TestSlice | sort |
| PASS | 0.00 | TestInts | sort |
| PASS | 0.00 | TestReverseSortIntSlice | sort |
| PASS | 0.00 | TestNonDeterministicComparison | sort |
| PASS | 0.00 | TestSortStringSlice | sort |
| PASS | 0.00 | TestSortFloat64Slice | sort |
| PASS | 0.00 | TestSortIntSlice | sort |
| PASS | 0.00 | TestAdversary | sort |
| PASS | 0.00 | TestStableInts | sort |
| PASS | 0.00 | TestSearchExhaustive | sort |
| PASS | 0.00 | TestSearchWrappers | sort |
| PASS | 0.00 | TestSearchEfficiency | sort |
| PASS | 0.00 | Example | sort |
| PASS | 0.00 | Example_sortKeys | sort |
| PASS | 0.00 | Example_sortMultiKeys | sort |
| PASS | 0.00 | ExampleSearch | sort |
| PASS | 0.00 | ExampleSearch_descendingOrder | sort |
| PASS | 0.00 | ExampleSearchFloat64s | sort |
| PASS | 0.00 | ExampleSearchInts | sort |
| PASS | 0.00 | ExampleInts | sort |
| PASS | 0.00 | ExampleIntsAreSorted | sort |
| PASS | 0.00 | ExampleFloat64s | sort |
| PASS | 0.00 | ExampleFloat64sAreSorted | sort |
| PASS | 0.00 | ExampleReverse | sort |
| PASS | 0.00 | ExampleSlice | sort |
| PASS | 0.00 | ExampleSliceStable | sort |
| PASS | 0.00 | ExampleStrings | sort |
| PASS | 0.00 | Example_sortWrapper | sort |
fmt tests summary
Click to expand!
| SKIP | 0.00 | TestCountMallocs | fmt |
| PASS | 0.04 | TestScanInts | fmt |
| PASS | 0.01 | TestIsSpace | fmt |
| PASS | 0.00 | TestErrorf | fmt |
| PASS | 0.00 | TestComplexFormatting | fmt |
| PASS | 0.00 | TestReorder | fmt |
| PASS | 0.00 | TestFlagParser | fmt |
| PASS | 0.00 | TestStructPrinter | fmt |
| PASS | 0.00 | TestSlicePrinter | fmt |
| PASS | 0.00 | TestMapPrinter | fmt |
| PASS | 0.00 | TestEmptyMap | fmt |
| PASS | 0.00 | TestBlank | fmt |
| PASS | 0.00 | TestBlankln | fmt |
| PASS | 0.00 | TestFormatterPrintln | fmt |
| PASS | 0.00 | TestWidthAndPrecision | fmt |
| PASS | 0.00 | TestPanics | fmt |
| PASS | 0.00 | TestBadVerbRecursion | fmt |
| PASS | 0.00 | TestSprintf | fmt |
| PASS | 0.00 | TestNilDoesNotBecomeTyped | fmt |
| PASS | 0.00 | TestFormatterFlags | fmt |
| PASS | 0.00 | TestParsenum | fmt |
| PASS | 0.00 | TestScan | fmt |
| PASS | 0.00 | TestScan/StringReader | fmt |
| PASS | 0.00 | TestScan/ReaderOnly | fmt |
| PASS | 0.00 | TestScan/OneByteReader | fmt |
| PASS | 0.00 | TestScan/DataErrReader | fmt |
| PASS | 0.00 | TestScanln | fmt |
| PASS | 0.00 | TestScanln/StringReader | fmt |
| PASS | 0.00 | TestScanln/ReaderOnly | fmt |
| PASS | 0.00 | TestScanln/OneByteReader | fmt |
| PASS | 0.00 | TestScanln/DataErrReader | fmt |
| PASS | 0.00 | TestScanf | fmt |
| PASS | 0.00 | TestScanOverflow | fmt |
| PASS | 0.00 | TestNaN | fmt |
| PASS | 0.00 | TestInf | fmt |
| PASS | 0.00 | TestScanfMulti | fmt |
| PASS | 0.00 | TestScanfMulti/StringReader | fmt |
| PASS | 0.00 | TestScanfMulti/ReaderOnly | fmt |
| PASS | 0.00 | TestScanfMulti/OneByteReader | fmt |
| PASS | 0.00 | TestScanfMulti/DataErrReader | fmt |
| PASS | 0.00 | TestScanMultiple | fmt |
| PASS | 0.00 | TestScanEmpty | fmt |
| PASS | 0.00 | TestScanNotPointer | fmt |
| PASS | 0.00 | TestScanlnNoNewline | fmt |
| PASS | 0.00 | TestScanlnWithMiddleNewline | fmt |
| PASS | 0.00 | TestEOF | fmt |
| PASS | 0.00 | TestEOFAtEndOfInput | fmt |
| PASS | 0.00 | TestEOFAllTypes | fmt |
| PASS | 0.00 | TestUnreadRuneWithBufio | fmt |
| PASS | 0.00 | TestMultiLine | fmt |
| PASS | 0.00 | TestLineByLineFscanf | fmt |
| PASS | 0.00 | TestScanStateCount | fmt |
| PASS | 0.00 | TestFmtInterface | fmt |
| PASS | 0.00 | TestHexBytes | fmt |
| PASS | 0.00 | TestScanNewlinesAreSpaces | fmt |
| PASS | 0.00 | TestScanlnNewlinesTerminate | fmt |
| PASS | 0.00 | TestScanfNewlineMatchFormat | fmt |
| PASS | 0.00 | TestHexByte | fmt |
| PASS | 0.00 | TestStringer | fmt |
| PASS | 0.00 | ExampleErrorf | fmt |
| PASS | 0.00 | ExampleFscanf | fmt |
| PASS | 0.00 | ExampleFscanln | fmt |
| PASS | 0.00 | ExampleSscanf | fmt |
| PASS | 0.00 | ExamplePrint | fmt |
| PASS | 0.00 | ExamplePrintln | fmt |
| PASS | 0.00 | ExamplePrintf | fmt |
| PASS | 0.00 | ExampleSprint | fmt |
| PASS | 0.00 | ExampleSprintln | fmt |
| PASS | 0.00 | ExampleSprintf | fmt |
| PASS | 0.00 | ExampleFprint | fmt |
| PASS | 0.00 | ExampleFprintln | fmt |
| PASS | 0.00 | ExampleFprintf | fmt |
| PASS | 0.00 | Example_printers | fmt |
| PASS | 0.00 | Example_formats | fmt |
| PASS | 0.00 | ExampleGoStringer | fmt |
| PASS | 0.00 | ExampleStringer | fmt |
strings tests summary
Click to expand!
| PASS | 0.55 | TestCompareStrings | strings |
| PASS | 0.12 | TestCaseConsistency | strings |
| PASS | 0.00 | TestBuilder | strings |
| PASS | 0.00 | TestBuilderGrow | strings |
| PASS | 0.00 | TestBuilderWrite2 | strings |
| PASS | 0.00 | TestBuilderWrite2/Write | strings |
| PASS | 0.00 | TestBuilderWrite2/WriteRune | strings |
| PASS | 0.00 | TestBuilderWrite2/WriteRuneWide | strings |
| PASS | 0.00 | TestBuilderWrite2/WriteString | strings |
| PASS | 0.00 | TestBuilderWriteByte | strings |
| PASS | 0.00 | TestBuilderAllocs | strings |
| PASS | 0.00 | TestBuilderCopyPanic | strings |
| PASS | 0.00 | TestBuilderWriteInvalidRune | strings |
| PASS | 0.00 | TestClone | strings |
| PASS | 0.00 | TestCompare | strings |
| PASS | 0.00 | TestCompareIdenticalString | strings |
| PASS | 0.00 | TestBuilderReset | strings |
| PASS | 0.00 | TestReader | strings |
| PASS | 0.00 | TestReadAfterBigSeek | strings |
| PASS | 0.00 | TestReaderAt | strings |
| PASS | 0.00 | TestReaderAtConcurrent | strings |
| PASS | 0.00 | TestEmptyReaderConcurrent | strings |
| PASS | 0.00 | TestWriteTo | strings |
| PASS | 0.00 | TestReaderLenSize | strings |
| PASS | 0.00 | TestReaderReset | strings |
| PASS | 0.00 | TestReaderZero | strings |
| PASS | 0.00 | TestReplacer | strings |
| PASS | 0.00 | TestPickAlgorithm | strings |
| PASS | 0.00 | TestWriteStringError | strings |
| PASS | 0.00 | TestGenericTrieBuilding | strings |
| PASS | 0.00 | TestFinderNext | strings |
| PASS | 0.00 | TestFinderCreation | strings |
| PASS | 0.00 | TestIndex | strings |
| PASS | 0.00 | TestLastIndex | strings |
| PASS | 0.00 | TestIndexAny | strings |
| PASS | 0.00 | TestLastIndexAny | strings |
| PASS | 0.00 | TestIndexByte | strings |
| PASS | 0.00 | TestLastIndexByte | strings |
| PASS | 0.00 | TestIndexRandom | strings |
| PASS | 0.00 | TestIndexRune | strings |
| PASS | 0.00 | TestSplit | strings |
| PASS | 0.00 | TestSplitAfter | strings |
| PASS | 0.00 | TestFields | strings |
| PASS | 0.00 | TestFieldsFunc | strings |
| PASS | 0.00 | TestMap | strings |
| PASS | 0.00 | TestToUpper | strings |
| PASS | 0.00 | TestToLower | strings |
| PASS | 0.00 | TestToValidUTF8 | strings |
| PASS | 0.00 | TestSpecialCase | strings |
| PASS | 0.00 | TestTrimSpace | strings |
| PASS | 0.00 | TestTrim | strings |
| PASS | 0.00 | TestTrimFunc | strings |
| PASS | 0.00 | TestIndexFunc | strings |
| PASS | 0.00 | TestBuilderString | strings |
| PASS | 0.00 | TestRepeat | strings |
| PASS | 0.00 | TestRepeatCatchesOverflow | strings |
| PASS | 0.00 | TestRunes | strings |
| PASS | 0.00 | TestReadByte | strings |
| PASS | 0.00 | TestReadRune | strings |
| PASS | 0.00 | TestUnreadRuneError | strings |
| PASS | 0.00 | TestReplace | strings |
| PASS | 0.00 | TestTitle | strings |
| PASS | 0.00 | TestContains | strings |
| PASS | 0.00 | TestContainsAny | strings |
| PASS | 0.00 | TestContainsRune | strings |
| PASS | 0.00 | TestEqualFold | strings |
| PASS | 0.00 | TestCount | strings |
| PASS | 0.00 | TestCut | strings |
| PASS | 0.00 | ExampleBuilder | strings |
| PASS | 0.00 | ExampleCompare | strings |
| PASS | 0.00 | ExampleContains | strings |
| PASS | 0.00 | ExampleContainsAny | strings |
| PASS | 0.00 | ExampleContainsRune | strings |
| PASS | 0.00 | ExampleCount | strings |
| PASS | 0.00 | ExampleCut | strings |
| PASS | 0.00 | ExampleEqualFold | strings |
| PASS | 0.00 | ExampleFields | strings |
| PASS | 0.00 | ExampleFieldsFunc | strings |
| PASS | 0.00 | ExampleHasPrefix | strings |
| PASS | 0.00 | ExampleHasSuffix | strings |
| PASS | 0.00 | ExampleIndex | strings |
| PASS | 0.00 | ExampleIndexFunc | strings |
| PASS | 0.00 | ExampleIndexAny | strings |
| PASS | 0.00 | ExampleIndexByte | strings |
| PASS | 0.00 | ExampleIndexRune | strings |
| PASS | 0.00 | ExampleLastIndex | strings |
| PASS | 0.00 | ExampleLastIndexAny | strings |
| PASS | 0.00 | ExampleLastIndexByte | strings |
| PASS | 0.00 | ExampleLastIndexFunc | strings |
| PASS | 0.00 | ExampleJoin | strings |
| PASS | 0.00 | ExampleRepeat | strings |
| PASS | 0.00 | ExampleReplace | strings |
| PASS | 0.00 | ExampleReplaceAll | strings |
| PASS | 0.00 | ExampleSplit | strings |
| PASS | 0.00 | ExampleSplitN | strings |
| PASS | 0.00 | ExampleSplitAfter | strings |
| PASS | 0.00 | ExampleSplitAfterN | strings |
| PASS | 0.00 | ExampleTitle | strings |
| PASS | 0.00 | ExampleToTitle | strings |
| PASS | 0.00 | ExampleToTitleSpecial | strings |
| PASS | 0.00 | ExampleMap | strings |
| PASS | 0.00 | ExampleNewReplacer | strings |
| PASS | 0.00 | ExampleToUpper | strings |
| PASS | 0.00 | ExampleToUpperSpecial | strings |
| PASS | 0.00 | ExampleToLower | strings |
| PASS | 0.00 | ExampleToLowerSpecial | strings |
| PASS | 0.00 | ExampleTrim | strings |
| PASS | 0.00 | ExampleTrimSpace | strings |
| PASS | 0.00 | ExampleTrimPrefix | strings |
| PASS | 0.00 | ExampleTrimSuffix | strings |
| PASS | 0.00 | ExampleTrimFunc | strings |
| PASS | 0.00 | ExampleTrimLeft | strings |
| PASS | 0.00 | ExampleTrimLeftFunc | strings |
| PASS | 0.00 | ExampleTrimRight | strings |
| PASS | 0.00 | ExampleTrimRightFunc | strings |
Package summary
| STATUS | ELAPSED | PACKAGE | COVER | PASS | FAIL | SKIP |
|---|---|---|---|---|---|---|
| PASS | 1.52s | sort | 60.8% | 38 | 0 | 1 |
| PASS | 1.24s | strings | 98.1% | 115 | 0 | 0 |
| PASS | 0.66s | fmt | 95.2% | 75 | 0 | 1 |
My ideal would be that --markdown would convert all output to Markdown, including the colour.
I wanted to use -all in our monorepo, but the output is just overwhelming, but your idea of collapsible sections would be a great solution to that.
I wanted to use -all in our monorepo, but the output is just overwhelming, but your idea of collapsible sections would be a great solution to that.
Ye, for very large monorepos -all (-skip and -pass combined) is overwhelming. The idea behind this flag was to pair it with -slow N .. which reports slowest N (passed) tests per package.
Example:
go test time -cover -json -count=1 | tparse -slow 2 -all
Will limit -all output to the 2 slowest passed tests from the time package, out of 157. This way can quickly see that TestTimerStopStress and TestTicker are the worst offenders.
| STATUS | ELAPSED | TEST | PACKAGE |
|---|---|---|---|
| SKIP | 0.00 | TestCountMallocs | time |
| PASS | 3.00 | TestTimerStopStress | time |
| PASS | 1.10 | TestTicker | time |
| STATUS | ELAPSED | PACKAGE | COVER | PASS | FAIL | SKIP |
|---|---|---|---|---|---|---|
| PASS | 7.66s | time | 91.8% | 157 | 0 | 1 |
Ah nice, I wasn't quite sure what -slow did TBH.
Okay, I see some issues with the way I've used the https://github.com/charmbracelet/lipgloss package. I'll push through a few fixes in the next week or so (just need to find time) to specifically address the -format=markdown and color issues reported.
Here is to hoping GitHub adds some way to colorize markdown, github/markup#1440
| Before | After |
|---|---|
 |
 |
Alright, I think I got this into a workable state. For markdown output, needed to wrap the failed test output, otherwise, there is weird formatting with the way output contains indents that trigger markdown code fences.
And on local CI it continues to work like normal:
And then GitHub CI it also looks descent.
Going to clean this up and try and merge a branch tonight.
Ended up using ## for the package headers between failed tests. Here is what it looks like now...
There is still some work to do to polish markdown formatting, but I think this is in a better state than before. Pushed a v0.11.0 release.
I think the markdown looks a bit nicer now. It uses markdown sections to collapse the overwhelming tests automatically. Here I use -slow 20 to filter and display the slowest 20 tests only in that section once uncollapsed.
