Connections to Postgres Databases are terminated unexpectedly

[sudhindrasajjal]

Bug description

Connections to Postgres Databases are terminated unexpectedly when they are idle for some time. We noticed that when Posthog is idle for 30 minutes or so, and then tries communicating with Postgres, the connection is terminated and then restablished. It looks like an error not handled (?) in the postgres connector.

The logs are attached below.

2021-04-26 05:44:52 | ⌛⌛⌛ LPushing redis key delayed. Waiting over 30 sec to lpush key: celery
-- | --
  | 2021-04-26 05:44:52 | 10.244.130.1:49084 - "GET /healthz HTTP/1.1" 200
  | 2021-04-26 05:44:52 | at TCP.<anonymous> (net.js:673:12)
  | 2021-04-26 05:44:52 | at Socket.EventEmitter.emit (domain.js:467:12)
  | 2021-04-26 05:44:52 | at Socket.emit (events.js:315:20)
  | 2021-04-26 05:44:52 | at Socket.<anonymous> (/code/plugins/node_modules/pg/lib/connection.js:58:12)
  | 2021-04-26 05:44:52 | at Connection.EventEmitter.emit (domain.js:467:12)
  | 2021-04-26 05:44:52 | at Connection.emit (events.js:315:20)
  | 2021-04-26 05:44:52 | at Object.onceWrapper (events.js:421:28)
  | 2021-04-26 05:44:52 | at Connection.<anonymous> (/code/plugins/node_modules/pg/lib/client.js:132:73)
  | 2021-04-26 05:44:52 | [___2] 🔔 Error: Connection terminated unexpectedly
  | 2021-04-26 05:44:52 | at processTicksAndRejections (internal/process/task_queues.js:80:21)
  | 2021-04-26 05:44:52 | at endReadableNT (internal/streams/readable.js:1327:12)
  | 2021-04-26 05:44:52 | at Socket.EventEmitter.emit (domain.js:467:12)
  | 2021-04-26 05:44:52 | at Socket.emit (events.js:327:22)
  | 2021-04-26 05:44:52 | at Socket.<anonymous> (/code/plugins/node_modules/pg/lib/connection.js:102:12)
  | 2021-04-26 05:44:52 | at Connection.EventEmitter.emit (domain.js:467:12)
  | 2021-04-26 05:44:52 | at Connection.emit (events.js:315:20)
  | 2021-04-26 05:44:52 | at Object.onceWrapper (events.js:421:28)
  | 2021-04-26 05:44:52 | at Connection.<anonymous> (/code/plugins/node_modules/pg/lib/client.js:132:73)
  | 2021-04-26 05:44:52 | [___1] 🔔 Error: Connection terminated unexpectedly
  | 2021-04-26 05:44:52 | ⌛⌛⌛ LPushing redis key delayed. Waiting over 30 sec to lpush key: celery

These errors are triggered almost twice an hour. So if this is expected behavior maybe they can be silenced or handled in the code.

Expected behavior

For retries to be silent

How to reproduce

1. Let Posthog idle for around 30 mins
2. Run a database query from Posthog UI

Environment

• self-hosted PostHog, version/commit: release-1.24.0 via Helm chart on Kubernetes

Additional context

Thank you for your bug report – we love squashing them!

[Twixes]

Hey! Thanks a lot for the report. Looks like this occurs in our ingestion service, @posthog/plugin-server, so I'll move this to the service's repo.

[yakkomajuri]

Hey @sudhindrasajjal! Are you still experiencing this?

[Twixes]

Closing because of staleness, but feel free to reopen.