n8n-io/n8n

Upgrading to v1.39.1 throws express-rate-limit Errors - CANNOT LOG IN TO UI

dkindlund opened this issue · 10 comments

dkindlund commented

Bug Description

After upgrading to v1.39.1, I see this error thrown in the n8n logs:

ValidationError: The 'X-Forwarded-For' header is set but the Express 'trust proxy' setting is false (default). This could indicate a misconfiguration which would prevent express-rate-limit from accurately identifying users. See https://express-rate-limit.github.io/ERR_ERL_UNEXPECTED_X_FORWARDED_FOR/ for more information.
    at Object.xForwardedForHeader (/usr/local/lib/node_modules/n8n/node_modules/express-rate-limit/dist/index.cjs:166:13)
    at Object.wrappedValidations.<computed> [as xForwardedForHeader] (/usr/local/lib/node_modules/n8n/node_modules/express-rate-limit/dist/index.cjs:338:22)
    at Object.keyGenerator (/usr/local/lib/node_modules/n8n/node_modules/express-rate-limit/dist/index.cjs:593:20)
    at /usr/local/lib/node_modules/n8n/node_modules/express-rate-limit/dist/index.cjs:644:32
    at /usr/local/lib/node_modules/n8n/node_modules/express-rate-limit/dist/index.cjs:625:5 {

When I try to login to the UI, I can't login because I see this error getting thrown:

image

So, I'm effectively LOCKED OUT of the UI, at this point.

I think this issue was caused by recently upgrading the express package in this commit, here:
dc42ac1

To Reproduce

  1. Deploy n8n v1.39.1 in a Google Cloud Run service
  2. Try to login
  3. See error in UI and in logs

Expected behavior

I don't expect n8n to break when upgrading. Plain and simple.

I have no idea how to enable the "trust proxy" setting and I really wish this was documented in some sort of upgrade notes. I checked here: https://docs.n8n.io/release-notes/ and there was NO mention of this issue.

Operating System

Google Cloud Run

n8n Version

1.39.1

Node.js Version

18.16

Database

PostgreSQL

Execution mode

main (default)

dkindlund commented

It looks like this is the code that's needed to be added to n8n:
https://github.com/GoogleCloudPlatform/nodejs-docs-samples/pull/3586/files#diff-03f690ebd3c288db3f9f3fc236259f272a1a8105d8587580d24d880000eda715R41

But I'm still trying to figure out where that equivalent code lives within the n8n codebase...

dkindlund commented

So it looks like this param WAS declared here:
https://github.com/n8n-io/n8n/blob/master/packages/cli/src/AbstractServer.ts#L71

And it's feature is anchored to the proxy_hops ENV variable... investigating further...

dkindlund commented

Okay, so there's an environment variable named N8N_PROXY_HOPS apparently... but it appears to be completely undocumented...

dkindlund commented

Okay, I set N8N_PROXY_HOPS=1 and it did NOT resolve the issue. This is incredibly frustrating...

dkindlund commented

Update: So apparently, changing N8N_PROXY_HOPS=1 did fix the first issue. But there was a second issue.

Essentially, for some reason, my browser was using an old (cached?) set of credentials/cookies to access n8n. Because those credentials expired upon upgrade, the client-side code did NOT respect that expiration -- and the browser attempted to retry using that expired cookie.

Because the browser kept trying to query n8n using expired credentials, express started throwing 429 errors (e.g., too many requests)... and that started happening even before I could properly log in to the UI to refresh my credentials/cookies.

So, the ultimate solution was to hard restart Google Chrome browser, manually flush all local cookies, wait a little bit of time (?), and THEN try again.

At that point, I was able to successfully log back into the N8N UI.

Thankfully, all of these issues did NOT seem to impact current running workflows at the time, but it was an incredibly frustrating experience upgrading -- mainly because none of these issues appeared to be documented ahead of time.

I want to upgrade n8n more often, but if this is how upgrades go, I don't think I can unless I set aside 2-3 hours to expect some sort of unexpected breakage.

dkindlund commented

Ultimately, I think the open item with this issue is to somehow provide documentation about:

  • What N8N_PROXY_HOPS is for
  • When to use it
  • When to not use it
  • And also call out in the release notes about this new issue -- if you're running n8n inside something like a Google Cloud Run service
Joffcom commented

I think it is safe to close this issue as there isn't a bug here but there is a learning on the proxy_hops, Oddly enough the N8N_PROXY_HOPS key was added back in 1.16.0 when we introduced rate limiting for the forgot password endpoint.

I suspect this could be triggered by a mix of the cache causing the lockout originally and the header issue being highlighted by another change.