10.0.0 event forwarder crashes with 10.0.0 cluster

Question

10.0.0 event forwarder crashes with 10.0.0 cluster

programmerq opened this issue 2 years ago · 2 comments

When running event forwarder version 10.0.0 against a teleport 10.0.0 cluster, the following failure can occur:

INFO   Using batch size batch:20 event-handler/cli.go:192
INFO   Using namespace namespace:default event-handler/cli.go:193
INFO   Using type filter types:[] event-handler/cli.go:194
INFO   Skipping session events of type types:map[print:{}] event-handler/cli.go:195
INFO   Using start time value:&lt;nil&gt; event-handler/cli.go:196
INFO   Using timeout timeout:10s event-handler/cli.go:197
INFO   Using Fluentd url url:<https://localhost:8888/audit.log> event-handler/cli.go:198
INFO   Using Fluentd session url url:<https://localhost:8888/session> event-handler/cli.go:199
INFO   Using Fluentd ca ca:/keys/ca.crt event-handler/cli.go:200
INFO   Using Fluentd cert cert:/keys/client.crt event-handler/cli.go:201
INFO   Using Fluentd key key:/keys/client.key event-handler/cli.go:202
INFO   Using Teleport identity file file:/identity event-handler/cli.go:205
INFO   Using existing storage directory dir:/storage/teleport3025_3025 event-handler/state.go:114
INFO   Using initial cursor value cursor:{"date":"2022-07-14","iterator":{"CreatedAt":{"B":null,"BOOL":null,"BS":null,"L":null,"M":null,"N":"1657783138","NS":null,"NULL":null,"S":null,"SS":null},"CreatedAtDate":{"B":null,"BOOL":null,"BS":null,"L":null,"M":null,"N":null,"NS":null,"NULL":null,"S":"2022-07-14","SS":null},"EventIndex":{"B":null,"BOOL":null,"BS":null,"L":null,"M":null,"N":"0","NS":null,"NULL":null,"S":null,"SS":null},"SessionID":{"B":null,"BOOL":null,"BS":null,"L":null,"M":null,"N":null,"NS":null,"NULL":null,"S":"479d607a-5d85-4825-8b1e-79d1a1a6f608","SS":null}}} event-handler/app.go:191
INFO   Using initial ID value id:1a4faa83-8942-4514-a310-c01362fc0eb6 event-handler/app.go:192
INFO   Using start time from state value:2022-07-14 06:21:37 +0000 UTC event-handler/app.go:193
ERRO   Error ingesting Audit Log err:failed to parse event, session record is corrupted event-handler/events_job.go:82
ERRO   Watcher event loop failed error:[
ERROR REPORT:
Original Error: *trace.BadParameterError failed to parse event, session record is corrupted
Stack Trace:
	/go/pkg/mod/github.com/gravitational/teleport/api@v0.0.0-20220701151950-c3f9fd84a104/types/events/oneof.go:416 <http://github.com/gravitational/teleport/api/types/events.FromOneOf|github.com/gravitational/teleport/api/types/events.FromOneOf>
	/go/pkg/mod/github.com/gravitational/teleport/api@v0.0.0-20220701151950-c3f9fd84a104/client/client.go:1940 <http://github.com/gravitational/teleport/api/client.(*Client).SearchEvents|github.com/gravitational/teleport/api/client.(*Client).SearchEvents>
	/workspace/event-handler/teleport_events_watcher.go:175 main.(*TeleportEventsWatcher).getEvents
	/workspace/event-handler/teleport_events_watcher.go:122 main.(*TeleportEventsWatcher).fetch
	/workspace/event-handler/teleport_events_watcher.go:212 main.(*TeleportEventsWatcher).Events.func1
	/usr/local/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: failed to parse event, session record is corrupted] event-handler/events_job.go:67
ERRO   Terminating with fatal error [1]... error:[
ERROR REPORT:
Original Error: *trace.BadParameterError failed to parse event, session record is corrupted
Stack Trace:
	/go/pkg/mod/github.com/gravitational/teleport/api@v0.0.0-20220701151950-c3f9fd84a104/types/events/oneof.go:416 <http://github.com/gravitational/teleport/api/types/events.FromOneOf|github.com/gravitational/teleport/api/types/events.FromOneOf>
	/go/pkg/mod/github.com/gravitational/teleport/api@v0.0.0-20220701151950-c3f9fd84a104/client/client.go:1940 <http://github.com/gravitational/teleport/api/client.(*Client).SearchEvents|github.com/gravitational/teleport/api/client.(*Client).SearchEvents>
	/workspace/event-handler/teleport_events_watcher.go:175 main.(*TeleportEventsWatcher).getEvents
	/workspace/event-handler/teleport_events_watcher.go:122 main.(*TeleportEventsWatcher).fetch
	/workspace/event-handler/teleport_events_watcher.go:212 main.(*TeleportEventsWatcher).Events.func1
	/usr/local/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: failed to parse event, session record is corrupted] lib/bail.go:14

I suspect that this may be due to the 10.0.0 forwarder code being built against the teleport 9.3.9 api: https://github.com/gravitational/teleport-plugins/blob/v10.0.0/go.mod#L13

I would anticipate an error of this nature if the 10.0.0 code produces an event type that the 9.3.9 library does not understand.

gz#5934

Answer 1 · 2022-07-19T20:47:45.000Z

Since I forgot to include them yesterday, instructions to reproduce:

Deploy a 10.0.0 event handler to watch a 10.0.0 Teleport cluster.
Step through the new resource request feature per the guide in the docs here: https://goteleport.com/docs/enterprise/workflow/resource-requests/?scope=enterprise
Once an event for this new feature appears, the plugin will crash.

Example of a resource request event:

{"ei":0,"event":"access_request.search","uid":"1b6d7ea7-f9c6-419f-8ca5-5954fd39e594","code":"T5004I","cluster_name":"example","user":"alice","search_as_roles":["requester","access"],"resource_type":"node","namespace":"default"}

Other event types that are included in 10.0.x but not in 9.3.9 would also trigger the error.

This error also appears to be fixed by #600 which means that the v10.0.1 and v10.0.2 tags have the fix already, but haven't quite had their binary releases published.

This issue can be closed once those are published.

Answer 2 · 2022-07-20T07:16:57.000Z

v10.0.2 was just released