ssbc/go-ssb

Fix flaky tests

Opened this issue · 6 comments

decentral1se commented

How to help

  • Pick a test from the list below 🗒️
  • Run the test over and over again until you see a failure (see below for HOWTO) 🏃‍♀️
  • If it fails, open up an issue to document your digging (e.g. #257) 🤓
  • Try to submit a fix 😄
  • If it doesn't fail after $a_lot_of_runs, let's re-enable it (e.g. #256) 🌞

Good Luck 🧑‍🚒 👷‍♀️

Testing the tests

cd graph
LIBRARIAN_WRITEALL=0 go test -v ./ -run TestBadger | tee -a test.log # wait $a_while
cat test.log | grep FAIL # check for failures
cat test.log | grep PASS$ | wc -l # count passes

TODO

KyleMaas commented

List of skipped tests currently (searching for SkipOnCI):

client/client_test.go-func TestStatusCalls(t *testing.T)

cmd/gossb-migrate-mf/migrate/migrate_test.go-func TestMigrate(t *testing.T) {

cmd/go-sbot/crashrecovery_test.go-func TestDontStartTwiceOnTheSameRepo(t *testing.T) {

graph/builder_test.go-func TestBadger(t *testing.T) {

graph/people_test.go-func TestPeople(t *testing.T) {
(This one's a little tricky, because it doesn't have a skip itself but it calls the run() function which has a skip)

message/legacy/verify_invalid_test.go-func TestInvalidFuzzed(t *testing.T) {

message/publish_test.go-func TestSignMessages(t *testing.T) {

Looks like a lot of this file:
multilogs/indexspeed_test.go:	if testutils.SkipOnCI(t) {

private/box2/spec_test.go-func TestSpec(t *testing.T) {

sbot/names_test.go-func TestNames(t *testing.T) {

sbot/persistence_test.go-func TestPersistence(t *testing.T) {

sbot/metafeed_test.go-func TestMetafeedIndexes(t *testing.T) {

sbot/unipub_test.go-func TestPublishUnicode(t *testing.T) {

sbot/groups_test.go-func TestPrivateGroupsManualDecrypt(t *testing.T) {

sbot/null_feed_test.go-func TestNullFeed(t *testing.T) 

sbot/null_feed_test.go-func TestNullFetched(t *testing.T) {

sbot/fsck_test.go-func TestFSCK(t *testing.T) {

tests/invite_legacy_test.go-func TestLegacyInviteJSCreate(t *testing.T) {

tests/invite_legacy_test.go-func TestLegacyInviteJSAccept(t *testing.T) {

tests/blocking_test.go-func TestBlocking(t *testing.T) {
KyleMaas commented

That's a much better format.

  • 🎉1
KyleMaas commented

Of note: if the tests are logging ANSI escape sequences, see this for how to filter them out:

https://superuser.com/questions/380772/removing-ansi-color-codes-from-text-stream

  • ❤️1
KyleMaas commented

Example command to run a test over and over again while filtering ANSI escapes:

watch 'LIBRARIAN_WRITEALL=0 go test -v ./ -run TestNullFeed | sed -e '"'"'s/\x1b\[[0-9;]*m//g'"'"' | tee -a ~/testlog.txt'
  • ❤️1
KyleMaas commented

Quick note: The TestInvalidFuzzed item is pointing to the draft pull request I closed and not the merged one.

  • 👍1
decentral1se commented

OK, that's every failure mapped out into an issue 😅

  • 👍2