kysely-org/kysely-postgres-js

If the connection to PG fails during execution, it will cause the program to crash (infinite loop)

Closed this issue · 9 comments

VxRain commented

Env:

  • Win10
  • Node v20.14.0
  • kysely: 0.27.3
  • kysely-postgres-js: 2.0.0,
  • postgres: 3.4.4

Reproduction steps:

  1. Start the PostgreSQL service.
  2. Start the application.
  3. Kill the PostgreSQL process.
  4. The error repeats infinitely.
import { Kysely } from "kysely";
import { PostgresJSDialect } from "kysely-postgres-js";
import postgres from "postgres";

// listen for unhandledRejection and uncaughtException
process.on("unhandledRejection", (reason) => {
  console.error("unhandledRejection: ", reason);
});
process.on("uncaughtException", (err) => {
  console.error("uncaughtException: ", err);
});

const pg = postgres({
  host: "localhost",
  max: 10,
  port: 5432,
  database: "test",
  user: "test",
  password: "test",
});
const db = new Kysely({ dialect: new PostgresJSDialect({ postgres: pg }) });

async function main() {
  await pg`SELECT 1`; // check connection

  setInterval(async () => {
    const user = await db
      .selectFrom("User")
      .select("username")
      .where("id", "=", 1)
      .executeTakeFirstOrThrow();
    console.log(Date.now(), user);
  }, 1000);
}
main();

Error logs:

uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
uncaughtException:  TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///C:/Users/Administrator/Desktop/Workspace/kysely-postgres/node_modules/.pnpm/postgres@3.4.4/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Additionally, I found that if the connection to PostgreSQL fails when app start, the Kysely instance initializes normally, but executing a query throws an error: "Cannot read properties of undefined (reading 'replace')."

Related to #9, #10, #11, I think we should investigate further to resolve the related issues.

VxRain commented

I don't think this is an issue with Postgres.js. I also tested it using Postgres.js. After killing the PG process, Postgres.js correctly throws an error and attempts to reconnect. Once the reconnection is successful, the program outputs results correctly, which is the expected behavior.

joshxyzhimself commented 
file://server/node_modules/postgres/src/connection.js:389
      stack: { value: err.stack + query.origin.replace(/.*\n/, '\n'), enumerable: options.debug },
                                               ^

TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file://server/node_modules/postgres/src/connection.js:389:48)
    at errored (file://server/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file://server/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Node.js v20.13.1

this only happens when i try to run my nodejs server while postgresql db server is off or inaccessible (e.g. due to docker initialization)

in docker postgres if you have init sql scripts it will restart once. if your nodejs server tries to connect while it's initializing / restarting, you'll get this error above.

VxRain commented

@joshxyzhimself

Hi, I didn't encounter this issue when using only Postgres.js. Does this problem occur only when using the Docker image? Could you provide a minimal reproducible code or repository?

joshxyzhimself commented 
import { Kysely } from "kysely";
import { PostgresJSDialect } from "kysely-postgres-js";
import postgres from "postgres";

import type { KyselyDatabase } from "./index.js";

const POSTGRES_HOST = "localhost";
const POSTGRES_PORT = 5432;
const POSTGRES_DB = "postgres";
const POSTGRES_USER = "postgres";
const POSTGRES_PASSWORD = "postgres";

export const pgc = postgres({
  database: POSTGRES_DB,
  host: POSTGRES_HOST,
  port: POSTGRES_PORT,
  user: POSTGRES_USER,
  password: POSTGRES_PASSWORD,
  max: 10,
});

export const pgd = new PostgresJSDialect({ postgres: pgc });

export const pgdb = new Kysely<KyselyDatabase>({ dialect: pgd });

try {
  const users = await pgc`SELECT * FROM users;`;
  console.log({ users });
} catch (e) {
  if (e instanceof Error) {
    console.error("POSTGRES.JS ERROR CAUGHT");
    console.error(e);
  }
}

try {
  const users = await pgdb.selectFrom("users").selectAll().execute();
  console.log({ users });
} catch (e) {
  if (e instanceof Error) {
    console.error("KYSELY ERROR CAUGHT");
    console.error(e);
  }
}
user@device:~/Documents/server$ tsc
user@device:~/Documents/server$ node ./dist/postgres.mjs
POSTGRES.JS ERROR CAUGHT
Error: connect ECONNREFUSED 127.0.0.1:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1606:16)
    at cachedError (file:///home/user/Documents/server/node_modules/postgres/src/query.js:170:23)
    at new Query (file:///home/user/Documents/server/node_modules/postgres/src/query.js:36:24)
    at sql (file:///home/user/Documents/server/node_modules/postgres/src/index.js:112:11)
    at file:///home/user/Documents/server/dist/postgres.mjs:20:29 {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 5432
}
file:///home/user/Documents/server/node_modules/postgres/src/connection.js:389
      stack: { value: err.stack + query.origin.replace(/.*\n/, '\n'), enumerable: options.debug },
                                               ^

TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Node.js v20.13.1
user@device:~/Documents/server$ npx tsx ./src/postgres.mts 
POSTGRES.JS ERROR CAUGHT
Error: connect ECONNREFUSED 127.0.0.1:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1606:16)
    at cachedError (file:///home/user/Documents/server/node_modules/postgres/src/query.js:170:23)
    at new Query (file:///home/user/Documents/server/node_modules/postgres/src/query.js:36:24)
    at sql (file:///home/user/Documents/server/node_modules/postgres/src/index.js:112:11)
    at <anonymous> (/home/user/Documents/server/src/postgres.mts:27:23) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 5432
}
file:///home/user/Documents/server/node_modules/postgres/src/connection.js:389
      stack: { value: err.stack + query.origin.replace(/.*\n/, '\n'), enumerable: options.debug },
                                               ^

TypeError: Cannot read properties of undefined (reading 'replace')
    at queryError (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:389:48)
    at errored (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:384:17)
    at Socket.error (file:///home/user/Documents/server/node_modules/postgres/src/connection.js:376:5)
    at Socket.emit (node:events:519:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Node.js v20.13.1

dang it never reached console.error("KYSELY ERROR CAUGHT");

joshxyzhimself commented

Hi, I didn't encounter this issue when using only Postgres.js. Does this problem occur only when using the Docker image? Could you provide a minimal reproducible code or repository?

nope, it also occurs even in my localhost (no docker).

example provided above.

same results with tsc & tsx compiled typescript code.

you're right that it throws proper econrefused when using postgres.js, but when using the kysely interface it doesn't throw a catchable error at all.

joshxyzhimself commented

Found this issue upstream, they're using mssql but similar problem where app crashes when db unreachable.

kysely-org/kysely#836

VxRain commented

@joshxyzhimself Yes, we are encountering the same issue.

Currently, we can use

process.on("uncaughtException", (err) => {
  console.error("uncaughtException: ", err);
});

to catch the error. But in this case, it isn't very helpful because it keeps throwing errors in an infinite loop, blocking the main process and making the service completely unusable. I think this might need a thorough investigation @igalklebanov

igalklebanov commented

Hey 👋

This happens in postgres.js when using the reserve() method. See porsager/postgres#778 (comment). A bug.

For now, use this library @ v1.x.x (this is before reserve() was introduced in postgres.js).

Alternatively, extend the driver, and override the acquireConnection method with a retry mechanism around the .reserve() invocation. Retrying is a connection pool's responsibility, so this cannot be shipped in this library.

Alternatively, don't execute with Kysely, but compile and pass the compiled queries straight to postgres.js's unsafe method.

Alternatively, patch postgres.js with a fix.

  • ❤️2
igalklebanov commented

Found this issue upstream, they're using mssql but similar problem where app crashes when db unreachable.

kysely-org/kysely#836

Not the same.

kysely core uses tedious, not mssql, as the underlying driver for MssqlDialect.

Unlike postgres.js, tedious doesn't come with connection pool management, so we're using tarn.js (also used by mssql and knex) for pool management.

The issue was tedious throwing connection errors asynchronously if you don't subscribe to Connection's "error" events, causing uncaught exceptions. Undocumented behavior (not a bug) in tedious and a bug in kysely (not handling errors).

Here, postgres.js has a bug where the logic/state are wrong, causing the code to execute a wrong branch and throw a JavaScript error (query.origin is undefined, why you invoking replace bro?).

  • ❤️1