Slack Plugin Errors with Interrupted Workflow
programmerq opened this issue · 0 comments
programmerq commented
When using the Slack plugin we've encountered a couple of issues twice within a span of about three weeks. Even though both errors are reproducible, they do not seem to affect the operation of the plugin during testing. However, in a live environment, these errors cause the plugin to stop working while the pod remains in a healthy state.
- Error Message 1:
ERRO Failed to process deleted request error:[
ERROR REPORT:
Original Error: *status.Error rpc error: code = DeadlineExceeded desc = context deadline exceeded
Stack Trace:
/go/pkg/mod/github.com/gravitational/teleport/api@v0.0.0-20230705190545-14128605d913/client/client.go:1062 github.com/gravitational/teleport/api/client.(*Client).GetPluginData
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/lib/plugindata/cas.go:166 github.com/gravitational/teleport/integrations/lib/plugindata.(*CompareAndSwap[...]).getPluginData
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/lib/plugindata/cas.go:121 github.com/gravitational/teleport/integrations/lib/plugindata.(*CompareAndSwap[...]).Update
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/access/common/app.go:430 github.com/gravitational/teleport/integrations/access/common.(*BaseApp).updateMessages
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/access/common/app.go:305 github.com/gravitational/teleport/integrations/access/common.(*BaseApp).onDeletedRequest
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/access/common/app.go:163 github.com/gravitational/teleport/integrations/access/common.(*BaseApp).onWatcherEvent
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/lib/watcherjob/watcherjob.go:240 github.com/gravitational/teleport/integrations/lib/watcherjob.job.eventFuncHandler.func1
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/lib/process.go:209 github.com/gravitational/teleport/integrations/lib.jobFunc.DoJob
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/lib/process.go:97 github.com/gravitational/teleport/integrations/lib.NewProcess.func2.1
/usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit
User Message: rpc error: code = DeadlineExceeded desc = context deadline exceeded] request_id:f6d1b22c-87d9-4b7d-9346-1f9ba35e63d8 request_op:delete common/app.go:164
- Error Message 2:
ERRO Failed to process deleted request error:[
ERROR REPORT:
Original Error: trace.aggregate request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, context deadline exceeded
Stack Trace:
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/lib/plugindata/cas.go:136 github.com/gravitational/teleport/integrations/lib/plugindata.(*CompareAndSwap[...]).Update
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/access/common/app.go:430 github.com/gravitational/teleport/integrations/access/common.(*BaseApp).updateMessages
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/access/common/app.go:305 github.com/gravitational/teleport/integrations/access/common.(*BaseApp).onDeletedRequest
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/access/common/app.go:163 github.com/gravitational/teleport/integrations/access/common.(*BaseApp).onWatcherEvent
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/lib/watcherjob/watcherjob.go:240 github.com/gravitational/teleport/integrations/lib/watcherjob.job.eventFuncHandler.func1
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/lib/process.go:209 github.com/gravitational/teleport/integrations/lib.jobFunc.DoJob
/go/pkg/mod/github.com/gravitational/teleport@v0.0.0-20230705190545-14128605d913/integrations/lib/process.go:97 github.com/gravitational/teleport/integrations/lib.NewProcess.func2.1
/usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit
User Message: request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, request is already resolved, context deadline exceeded] request_id:d262fe1e-028b-4793-a7be-7bcc8230f9da request_op:delete common/app.go:164
Environment:
This has been observed when running the 13.2.0 and 13.2.2 Slack plugin with a Teleport Cloud tenant.
Expected Behavior:
Teleport Slack plugin operating without interruption and handling access requests as expected. When it does run into trouble like this, it should fail its health checks so Kube can romeve the broken pod and replace it.
Additional Context:
There are no other access request plugins in the environment, and there is exactly once instance of the slack plugin running.