myelnet/pop

TestMultipleGet is racy

Opened this issue · 1 comments

Sometimes fail on CI with output:

=== RUN   TestMultipleGet
{"level":"error","error":"No state for /1623943149477: datastore: key not found","time":"2021-06-17T15:19:10Z","message":"attempting to configure data store"}
2021-06-17T15:19:10.488Z	ERROR	fsm	fsm/fsm.go:92	Executing event planner failed: Invalid transition in queue, state `0`, event `4`:
    github.com/filecoin-project/go-statemachine/fsm.eventProcessor.Apply
        /home/runner/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/fsm/eventprocessor.go:137
2021-06-17T15:19:10.489Z	ERROR	fsm	fsm/fsm.go:92	Executing event planner failed: Invalid transition in queue, state `0`, event `4`:
    github.com/filecoin-project/go-statemachine/fsm.eventProcessor.Apply
        /home/runner/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/fsm/eventprocessor.go:137
{"level":"error","error":"No state for /1623943149478: datastore: key not found","time":"2021-06-17T15:19:10Z","message":"attempting to configure data store"}
2021-06-17T15:19:10.499Z	ERROR	fsm	fsm/fsm.go:92	Executing event planner failed: Invalid transition in queue, state `0`, event `4`:
    github.com/filecoin-project/go-statemachine/fsm.eventProcessor.Apply
        /home/runner/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/fsm/eventprocessor.go:137
2021-06-17T15:19:10.500Z	ERROR	fsm	fsm/fsm.go:92	Executing event planner failed: Invalid transition in queue, state `0`, event `4`:
    github.com/filecoin-project/go-statemachine/fsm.eventProcessor.Apply
        /home/runner/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/fsm/eventprocessor.go:137
2021-06-17T15:19:10.512Z	ERROR	gs-traversal	runtraversal/runtraversal.go:43	failed to load link=bafy2bzacecddt5b3fodntlgkrtin6nj6ybrflotjzcpqutw4zgxdpk73lfnnk, nBlocksRead=0, err=blockstore: block not found
2021-06-17T15:19:10.512Z	ERROR	gs-traversal	runtraversal/runtraversal.go:32	traversal completion check failed, nBlocksRead=0, err=skip
    node_test.go:623: 
        	Error Trace:	node_test.go:623
        	            				popn.go:280
        	            				popn.go:697
        	            				popn.go:752
        	            				node_test.go:626
        	Error:      	Not equal: 
        	            	expected: ""
        	            	actual  : "context deadline exceeded"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-
        	            	+context deadline exceeded
        	Test:       	TestMultipleGet

Adding additional logs:

=== RUN   TestMultipleGet
{"level":"info","time":"2021-07-09T09:03:39Z","message":"starting query"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"waiting for triage"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"selected an offer"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"started transfer"}
{"level":"error","error":"No state for /1625821418797: datastore: key not found","time":"2021-07-09T09:03:39Z","message":"attempting to configure data store"}
2021-07-09T09:03:39.805Z	ERROR	fsm	fsm/fsm.go:92	Executing event planner failed: Invalid transition in queue, state `0`, event `4`:
    github.com/filecoin-project/go-statemachine/fsm.eventProcessor.Apply
        /home/runner/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/fsm/eventprocessor.go:137
2021-07-09T09:03:39.805Z	ERROR	fsm	fsm/fsm.go:92	Executing event planner failed: Invalid transition in queue, state `0`, event `4`:
    github.com/filecoin-project/go-statemachine/fsm.eventProcessor.Apply
        /home/runner/go/pkg/mod/github.com/filecoin-project/go-statemachine@v0.0.0-20200925024713-05bd7c71fbfe/fsm/eventprocessor.go:137
{"level":"info","time":"2021-07-09T09:03:39Z","message":"finished transfer"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"starting query"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"waiting for triage"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"selected an offer"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"started transfer"}
{"level":"error","error":"No state for /1625821418798: datastore: key not found","time":"2021-07-09T09:03:39Z","message":"attempting to configure data store"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"finished transfer"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"starting query"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"waiting for triage"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"selected an offer"}
{"level":"info","time":"2021-07-09T09:03:39Z","message":"started transfer"}
2021-07-09T09:03:39.819Z	ERROR	gs-traversal	runtraversal/runtraversal.go:43	failed to load link=bafy2bzacedyrtucitacchlrjaikx7gymcuk4iw3q7sldemdbkeaszqhcihuz2, nBlocksRead=0, err=blockstore: block not found
2021-07-09T09:03:39.820Z	ERROR	gs-traversal	runtraversal/runtraversal.go:32	traversal completion check failed, nBlocksRead=0, err=skip
    node_test.go:701: 
        	Error Trace:	node_test.go:701
        	            				popn.go:327
        	            				popn.go:815
        	            				popn.go:887
        	            				node_test.go:704
        	Error:      	Not equal: 
        	            	expected: ""
        	            	actual  : "context deadline exceeded"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-
        	            	+context deadline exceeded
        	Test:       	TestMultipleGet
--- FAIL: TestMultipleGet (10.00s)