Crash on startup with latest Docker (master)

Question

Crash on startup with latest Docker (master)

Elegant996 opened this issue a year ago · 33 comments

Elegant996 commented a year ago

Type: Bug Report

Try to follow the update procedure described in the README and try again before opening this issue.

Your Environment

Version used:
jesec/flood:master-distroless from Docker as of December 7th 2023.

Summary

Docker image fails to launch with the following: TypeError: Cannot read properties of undefined (reading 'push')

Expected Behavior

Flood should start.

Current Behavior

Flood does not start.

Possible Solution

Steps to Reproduce

Use jesec/flood:master-distroless from Docker
Access web UI
Observe container crashes

Context

Use Flood.

Answer 1 · 2023-12-08T09:36:55.000Z

can you try latest image and paste full stack trace here？I can't re-produce.

Answer 2 · 2023-12-08T10:15:29.000Z

Perhaps its because I'm not serving assets? I'm running a web server to serve the assets which has been working great until this last push. That error is literally all that appears before the container crashes:

Flood server 0.0.0-master.fc1e90d starting on https://10.244.5.200:3000



Starting without builtin authentication



Static assets not served



FATAL internal error. Please open an issue.

Uncaught exception: TypeError: Cannot read properties of undefined (reading 'push')

Please note that I have pulled the latest static assets from the latest docker image. I mount the assets directory from the image into the web server container. Same image ID.

My setup always pulls the latest image on startup.

Thanks!

Answer 3 · 2023-12-08T10:17:48.000Z

can you paste your full docker run command or compose config file?

Answer 4 · 2023-12-08T10:20:31.000Z

Not running with any extra Docker commands (using K8s) these are the env's though:

FLOOD_OPTION_ALLOWEDPATH: "/mnt"
FLOOD_OPTION_ASSETS: "false"
FLOOD_OPTION_AUTH: "none"
FLOOD_OPTION_BASEURI: "/"
FLOOD_OPTION_HOST: *uses container IP*
FLOOD_OPTION_RTSOCKET: "/run/rtorrent/rtorrent.sock"
FLOOD_OPTION_RUNDIR: "/flood/data"
FLOOD_OPTION_SSL: "true"
FLOOD_OPTION_SSLCERT: "/tls/tls.crt"
FLOOD_OPTION_SSLKEY: "/tls/tls.key"

Answer 5 · 2023-12-08T10:23:00.000Z

Ok, I'll take a look

Answer 6 · 2023-12-08T10:33:05.000Z

I still can not re-produce...

Answer 7 · 2023-12-08T10:37:37.000Z

can you try latest image which will print error's stack trace?

Answer 8 · 2023-12-09T02:32:57.000Z

Flood server 0.0.0-master.f42009f starting on https://10.244.5.250:3000

Starting without builtin authentication

Static assets not served

FATAL internal error. Please open an issue.
Uncaught exception: TypeError: Cannot read properties of undefined (reading 'push')
stack trace: TypeError: Cannot read properties of undefined (reading 'push')
    at ServerResponse._writeRaw (node:_http_outgoing:376:19)
    at ServerResponse._send (node:_http_outgoing:349:15)
    at ServerResponse.end (node:_http_outgoing:961:10)
    at /snapshot/flood/dist/index.js
    at g (/snapshot/flood/dist/index.js)
    at handleError (/snapshot/flood/dist/index.js)
    at X (/snapshot/flood/dist/index.js)
    at r.send (/snapshot/flood/dist/index.js)
    at ee (/snapshot/flood/dist/index.js)
    at s (/snapshot/flood/dist/index.js)
Stream closed EOF for media/rtorrent-0 (flood)

Sorry, stepped out for the evening. Double-checked asset locations for webserver, still correct and using this same image for those assets.

Answer 9 · 2023-12-09T04:24:39.000Z

I still can't find out reason…

does flood without any option works？

Answer 10 · 2023-12-09T05:51:00.000Z

please try latest image and paste stack trace again

Answer 11 · 2023-12-09T16:10:07.000Z

No log; just fails to start the container now. Instant crash.

Answer 12 · 2023-12-09T16:25:28.000Z

zero log at all?

Answer 13 · 2023-12-09T16:28:32.000Z

sorry, I still can't re-produce this, I'm using docker-compose.

Answer 14 · 2023-12-09T17:22:45.000Z

Interesting! I had a TCP health-check on port 3000 and the recent changes for testing somehow delayed the TCP health-check longer than my threshold permitted so K8s decided to kill the pod. I've disabled health-checks and got the log:

Flood server 0.0.0-master.be22e24 starting on https://10.244.5.187:3000

Starting without builtin authentication

Static assets not served

FATAL internal error. Please open an issue.
Uncaught exception: TypeError: Cannot read properties of undefined (reading 'push')
stack trace: TypeError: Cannot read properties of undefined (reading 'push')
    at ServerResponse._writeRaw (node:_http_outgoing:376:19)
    at ServerResponse._send (node:_http_outgoing:349:15)
    at ServerResponse.end (node:_http_outgoing:961:10)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
    at s (webpack://jesec/flood/webpack/startup:4:71)
Stream closed EOF for media/rtorrent-0 (flood)

Answer 15 · 2023-12-09T18:08:32.000Z

can you try jesec/flood:master-dbg image？stack trace from jesec/flood:master-distroless is unexpectedly useless…

Answer 16 · 2023-12-11T04:22:32.000Z

Now we're getting somewhere:

> flood@0.0.0-master.eaefd4c start:development:server
> NODE_ENV=development tsx watch server/bin/start.ts --host=0.0.0.0

npm ERR! code EACCES
npm ERR! syscall mkdir
npm ERR! path /.npm
npm ERR! errno -13
npm ERR! 
npm ERR! Your cache folder contains root-owned files, due to a bug in
npm ERR! previous versions of npm which has since been addressed.
npm ERR! 
npm ERR! To permanently fix this problem, please run:
npm ERR!   sudo chown -R 201615:201615 "/.npm"

npm ERR! Log files were not written due to an error writing to the directory: /.npm/_logs
npm ERR! You can rerun the command with `--loglevel=verbose` to see the logs in your terminal
Stream closed EOF for media/rtorrent-0 (flood)

Try running as non-root? 😄 201615 is the user and group since this uses sys for the NFS mounts and that ID has sufficient access.

Answer 17 · 2023-12-11T07:03:17.000Z

I'm guessing this is a different problem, dbg image run flood with npm and run process with different user.

How did you mount volume in your container?

Answer 18 · 2023-12-11T13:06:21.000Z

Mounting an emptyDir for the flood rundir and another one for the rtsocket. Regular directories for rtorrent are under /mnt which should not impact anything.

Answer 19 · 2023-12-11T13:27:48.000Z

Then I have no idea...

Answer 20 · 2023-12-11T14:08:09.000Z

You are trying this as well where assets are not served and a webserver is being run? This worked fine until the other day... It's definitely related to the latest commits...

Answer 21 · 2023-12-11T14:09:54.000Z

Yes, I use same options with docker-compose and still can't re-produce it.

I guess this is caused by #661, but you can't run start dbg image as expected so I can't debug it...

Answer 22 · 2023-12-11T15:23:57.000Z

Ah, well then let's try as root :)


> flood@0.0.0-master.eaefd4c start:development:server
> NODE_ENV=development tsx watch server/bin/start.ts --host=0.0.0.0

(node:36) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("file%3A///usr/src/app/node_modules/tsx/dist/loader.js", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
Stream closed EOF for media/rtorrent-0 (flood)

Answer 23 · 2023-12-11T15:25:00.000Z

Ah, well then let's try as root :)


> flood@0.0.0-master.eaefd4c start:development:server
> NODE_ENV=development tsx watch server/bin/start.ts --host=0.0.0.0

(node:36) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("file%3A///usr/src/app/node_modules/tsx/dist/loader.js", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
Stream closed EOF for media/rtorrent-0 (flood)

Looks like this container doesn't exit?

Answer 24 · 2023-12-11T16:59:49.000Z

Yeah, I didn't edit my config properly so another probe killed it early. The master-dbg appears to run without issue if I run it as root. I tried running distroless as root but the issue remained. Trying to see if there's another way to uncover what is happening...

Answer 25 · 2023-12-11T17:01:57.000Z

I'll try fix source map in distroless image

Answer 26 · 2023-12-11T17:15:44.000Z

please try latest distroless image again, stack trace should work now.

Answer 27 · 2023-12-11T17:55:08.000Z

Same errors but I realized something. If dbg works and the errors don't occur with npm, there's a good chance that the issue is actually related to the env's being used. After some deeper diving, these caused the issue:

FLOOD_OPTION_SSL: "true"
FLOOD_OPTION_SSLCERT: "/tls/tls.crt"
FLOOD_OPTION_SSLKEY: "/tls/tls.key"

Perhaps it's how those settings are being pushed?

Answer 28 · 2023-12-11T18:01:38.000Z

please paste stack trace…

Answer 29 · 2023-12-11T18:02:34.000Z

Same errors but I realized something. If dbg works and the errors don't occur with npm, there's a good chance that the issue is actually related to the env's being used. After some deeper diving, these caused the issue:
FLOOD_OPTION_SSL: "true"
FLOOD_OPTION_SSLCERT: "/tls/tls.crt"
FLOOD_OPTION_SSLKEY: "/tls/tls.key"
Perhaps it's how those settings are being pushed?

can you share your key and crt file？I have a ssl cert but didn't reproduce this issue.

Answer 30 · 2023-12-11T18:05:11.000Z

Yep, it's regenerated on container creation though. I'm not overly familiar with js but I was wondering if this line is somehow forcing SSL to be false?

Answer 31 · 2023-12-11T18:06:29.000Z

It mean ssl is disabled by defualt

Answer 32 · 2023-12-11T18:08:54.000Z

OK, I can reproduce now, it need a http request to trigger this issue.

Answer 33 · 2023-12-11T18:23:06.000Z

OK, this bug is caused by @fastify/express not supporting http2, I would just disable http2 and everything should works fine