Upgrade Task failures for Postgres backed cluster on versions 1.37.3 Enterprise
jkirvan-gr opened this issue · 5 comments
Bug Description
Several errors are occurring on upgrade from 1.34.2 to 1.37.3. The upgrade was a completely fresh build in Docker. All containers, images, volumes, and cache were cleared before 1.34.2 was built yesterday.
I've attached the logs (trimmed of other support containers for brevity) and build files with all sensitive information redacted. The uuid-ossp errors occur on each restart. The Drop column task is not retried on successive restarts.
Errors-Updatefrom1.34.2to1.37.3.log
Errors-UpdateFrom1.34.2to1.37.3.zip
I am unable to confirm if this is related or another issue but, I have several workflows acting odd after the upgrade. All of the issues revolve around nodes executing successfully but, the output is empty or partial (e.g. - merge node only returns first input instead of both)
I do not mind logging into the database and executing these commands as the superuser but, I wasn't sure if other upgraded processes were not executed due to these failures. Also, wanted to report it so the steps to resolve can be documented if necessary.
n8n-worker2-1 | Starting migration RemoveNodesAccess1712044305787
n8n-worker2-1 | Finished migration RemoveNodesAccess1712044305787
postgres-1 | 2024-04-18 15:35:14.491 UTC [48] ERROR: column "nodesAccess" of relation "credentials_entity" does not exist
postgres-1 | 2024-04-18 15:35:14.491 UTC [48] STATEMENT: ALTER TABLE "credentials_entity" DROP COLUMN "nodesAccess"
n8n-1 | Migration "RemoveNodesAccess1712044305787" failed, error: column "nodesAccess" of relation "credentials_entity" does not exist
n8n-1 | Error: There was an error running database migrations
n8n-1 | QueryFailedError: column "nodesAccess" of relation "credentials_entity" does not exist
postgres-1 | 2024-04-18 15:35:14.510 UTC [50] ERROR: permission denied to create extension "uuid-ossp"
postgres-1 | 2024-04-18 15:35:14.510 UTC [50] HINT: Must be superuser to create this extension.
postgres-1 | 2024-04-18 15:35:14.510 UTC [50] STATEMENT: CREATE EXTENSION IF NOT EXISTS "uuid-ossp"
To Reproduce
- build n8n using dockerfile attached.
- Build a fresh 1.34.2 instance as configured in attached compose files
- Setup Environments and pull workflows
- Fixup credentials
- docker compose down
- rebuild n8n (from dockerfile attached)
- docker compose pull
- docker compose up
Expected behavior
Successful upgrade (no errors)
Operating System
Ubuntu 22.04.4 LTS (Docker 26.0.1)
n8n Version
1.37.3
Node.js Version
18.19.1
Database
PostgreSQL
Execution mode
queue
I'm wondering if this is a new bug, or a race condition with various containers trying to run migrations.
If you restart all containers, do you still see this error?
I think you are correct on the Drop column error. The nodeAccess column doesn't appear to be in the schema any longer.
We are still seeing the other error on repeated restarts.
Those errors look like the postgres user being used with n8n doesn't have all the permissions it needs. You could either:
- run
ALTER USER [[DB_USER]] WITH SUPERUSER;as a superuser to grant the db user all the permissions - not recommended. - run
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";as a superuser (in the n8n database) to create the UUID extension, and then ignore the permission warning (if it still continues to show up). - not recommended. - run
GRANT CREATE ON DATABASE [[DB_NAME]] to [[DB_USER]];as a superuser to grant correct permissions to the user on the n8n DB - recommended.
This is either new or it was missed in your documentation for postgres permissions. Please have your tech writers update the documentation.
Thanks for sorting this out. I'll close this since all issues are resolved.