New version of magicmirror does not work in docker

Question

New version of magicmirror does not work in docker

Closed this issue 2 years ago · 36 comments

Hello,

I am getting this issue when I am trying to launch the latest version of docker:

[22.07.2021 12:16.03.454] [LOG]   Starting MagicMirror: v2.16.0
[22.07.2021 12:16.03.462] [LOG]   Loading config ...
[22.07.2021 12:16.03.474] [LOG]   Loading module helpers ...
[22.07.2021 12:16.03.497] [ERROR] WARNING! Could not load config file. Starting with default configuration. Error found: Error: Cannot find module 'request'
Require stack:
- /opt/magic_mirror/modules/MMM-quote-of-the-day/node_helper.js
- /opt/magic_mirror/js/app.js
- /opt/magic_mirror/serveronly/index.js
[22.07.2021 12:16.03.499] [LOG]   Loading module helpers ...
[22.07.2021 12:16.03.504] [ERROR] Whoops! There was an uncaught exception...
[22.07.2021 12:16.03.514] [ERROR] Error: Cannot find module 'request'
Require stack:
- /opt/magic_mirror/modules/MMM-quote-of-the-day/node_helper.js
- /opt/magic_mirror/js/app.js
- /opt/magic_mirror/serveronly/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
    at Function.Module._resolveFilename (/opt/magic_mirror/node_modules/module-alias/index.js:49:29)
    at Function.Module._load (internal/modules/cjs/loader.js:746:27)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at Object.<anonymous> (/opt/magic_mirror/modules/MMM-quote-of-the-day/node_helper.js:2:17)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:14) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/opt/magic_mirror/modules/MMM-quote-of-the-day/node_helper.js',
    '/opt/magic_mirror/js/app.js',
    '/opt/magic_mirror/serveronly/index.js'
  ]
}
[22.07.2021 12:16.03.515] [ERROR] MagicMirror will not quit, but it might be a good idea to check why this happened. Maybe no internet connection?
[22.07.2021 12:16.03.516] [ERROR] If you think this really is an issue, please open an issue on GitHub: https://github.com/MichMich/MagicMirror/issues

Can you please advise me what I am doing wrong? v2.15 is working fine.

Answer 1 · 2021-07-22T11:22:31.000Z

Hi, The docker implementation is not part of the core repo, so i can't really give any support on this.

It looks like not all required modules are installed in the docket container.

Answer 2 · 2021-07-22T12:59:23.000Z

this is because the request module was removed in 2.16 because of deprecation. and some of the other library files that used it.

See this content from discord mm support channel

in the docs https://khassel.gitlab.io/magicmirror/configuration/ you find a way how to log in into the running container. The key is docker exec -it mm bash, with this command you are in the container and navigate to the module folder and in the module folder you can install the missing packages with npm install xy

Answer 3 · 2021-07-30T14:12:56.000Z

I agree that depreciated software should no longer be used, but there are some truly great Magicmirror modules that have not been maintained for years and still require the "request" module. These mm modules simply stopped working since the lastest release.

I've also tried to install the "request" module as you have suggested, but the problem is that the Docker container or Magicmirror itself restarts after ~15 seconds. In this short time it is not possible to run the npm install command successfully.

Any ideas?

Answer 4 · 2021-07-30T14:52:48.000Z

@theskyisthelimit This is more of an issue with the docker container. I don't think anyone can really help you here. Is it an option to post an issue in de docker container repo?

Answer 5 · 2021-07-30T16:40:00.000Z

@theskyisthelimit there is documentation in the docker site that says you have to docker exec into the container and do the npm install in the module folder there

Answer 6 · 2021-07-30T21:16:57.000Z

The problem is that you can't exec into the container because the container always restarts because of the request errors.

To solve this you have to do this with a working config. You can disable the modules in your config which are causing the errors or you can use the default config for this.

Answer 7 · 2021-07-31T06:20:03.000Z

Thanks @khassel for the suggestion. That's a great idea! I will try it later.

As far as I understad, this is only a termporary solution until a new version of magicmirror is pushed, correct?

I was using your docker image with the tag "latest" for years but now I will have to do this step manually every time a new version is pushed?

Could we have a dedicated tag for "latest release + request npm module"?

Answer 8 · 2021-07-31T07:30:45.000Z

Another (maybe simpler) workaround ist to change (temporarily) the command section in the docker-compose.yml from

    command: 
      - npm
      - run
      - server

to

    command: 
      - sleep
      - infinity

To be clear where to install request: Not in the main folder of mm but in every module folder where it is missing, so e.g. in the first post here it was missing in modules/MMM-quote-of-the-day so you have to cd into this folder an run npm install request there.

If you run it in the main folder it will not survive a container restart.

You have to do this only once (for every module where it is missing) until a new version of a module is released (where its may fixed because this is a problem of the module).

Answer 9 · 2021-07-31T08:38:16.000Z

Thank you @khassel

I was able to locate the MM Plugins (MMM-homeassistant-sensors, MMM-cryptocurrency, MMM-Sonos) and ran the npm install request command in each folder. Then I was able to start the container with the latest magic mirror release.

Everything still seems to be working after a restart of the docker container, unfortunately if I do a full recreation of the docker image (with portainer) it's back to it's old failing behaviour. What will happen when a new release of MM is released?

Answer 10 · 2021-07-31T10:37:08.000Z

If there are so many modules still depending on the request module, I have no problem adding it back again as a npm dependency. What do you think, @khassel?

Answer 11 · 2021-07-31T12:44:59.000Z

the 'problem' here is that the modules didn't have a package.json (mostly) or didn't has request listed in the package JSON at all.

so, installing it in every needing module will require scanning then all for node_helper and checking their maybe non-existing node_modules for theissing modules ( not just request, bit valid-url, UUID, and a couple others so far,)

on EVERY startup

cause the user could have installed another module

oh and waiting for author on used and no longer maintaned modules

Answer 12 · 2021-07-31T19:11:56.000Z

What do you think, @khassel?

I'm not for adding it back again.

As @sdetweil already mentioned this happens only if the author of this modules has "forgotten" to put e.g. request into the package.json of his module.

We should try to get rid of old components here and as long as workarounds exist, I think that's okay. I also see this as a reminder for the users of such modules to consider whether they really want to continue using such old and unmaintained modules. When I find the time I will add the problem with instructions on how to fix it in my FAQ.

Of course there have already been some support requests on the subject but the release has been out for 4 weeks and such requests should decrease.

If we would add it back again, it would only take effect with the next release (October 1st), by then we will be through with the topic anyway.

Answer 13 · 2021-07-31T19:13:04.000Z

You convinced me. 😂

Answer 14 · 2021-07-31T19:13:38.000Z

unfortunately if I do a full recreation of the docker image (with portainer) it's back to it's old failing behaviour.

I don't understand what you are doing here

What will happen when a new release of MM is released?

Nothing, same situation as with current release.

Answer 15 · 2021-07-31T19:16:13.000Z

Better solution than changing the docker is to send a PR to the problematic modules to fix the issue and use your own fork in the meantime.

Answer 16 · 2021-07-31T19:17:24.000Z

most of the old modules will never be fixed

Answer 17 · 2021-07-31T19:22:59.000Z

@MichMich Its not only a problem with the docker setup, when you install without devDependencies you have the same problem with a classic install.

Answer 18 · 2021-07-31T19:25:40.000Z

That's why docker is a "non officially supported" method of installation. 😅

@sdetweil, I know. But if someone really needs a specific module that stopped working, they can always fork the module and make the necessary modifications. That's the beauty of open source. 😁

Answer 19 · 2021-07-31T19:33:38.000Z

I hope not!

I do prod only installs to keep the disk space usage down

Answer 20 · 2021-07-31T22:10:26.000Z

When I find the time I will add the problem with instructions on how to fix it in my FAQ.

have updated my FAQ

I hope not!

I'm using npm ci --only=prod (or npm install --only=prod when i build from the develop branch) and there is no request installed afterwards.

Answer 21 · 2021-08-01T07:15:25.000Z

unfortunately if I do a full recreation of the docker image (with portainer) it's back to it's old failing behaviour.

I don't understand what you are doing here

@khassel The container is being discarded, the latest images is being pulled and it creates a new, fresh docker container. The only thing that stays are the persistent files within the attached volumes.

What will happen when a new release of MM is released?

Nothing, same situation as with current release.

So about 5 hours ago, the version v2.16.0 was pushed and my watchtower pulled the newest docker image and recreated the magicmirror container. Now magicmirror has the same problem because the request module is missing from those 3 mm modules. That's what I meant yesterday.

Is it a possible solution to fork the failing modules and only add that package.json with the request module? Does this do the trick?

Answer 22 · 2021-08-01T12:07:09.000Z

@theskyisthelimit yes, forking and adding a package.json and adding the dependency will 'fix' the problem.

Answer 23 · 2021-08-01T13:53:56.000Z

Thank you @sdetweil

So I tried and forked the MMM-Sonos Repository and added the file like this: https://github.com/theskyisthelimit/MMM-Sonos/blob/master/package.json

I deleted the old folder without the package.json, did a git clone from my forked repository and an npm install but it didn't work.

Did I miss anything?

Answer 24 · 2021-08-01T14:24:12.000Z

I would have done npm init, and then npm install request --save

Answer 25 · 2021-08-01T17:53:07.000Z

@theskyisthelimit so now we come closer why it was not persistent, checked it in my environment.

If such a module e.g. MMM-Sonos comes without a package.json file the workaround with "install request in the module folder" does not work. I found the installed request package after this installation in the node_modules folder of mm and not under the module folder.

In this case you have to execute npm init and then npm install request (both in the module folder).

Answer 26 · 2021-08-01T20:03:38.000Z

yes, without a package.json, npm install will look up the file system to find the next nearest node_modules folder to install and record it in.

Answer 27 · 2021-08-02T15:25:06.000Z

@MichMich @sdetweil Thank's guys it worked.

I forked all 3 repositories and created or added the request module to the package.json in each repository.
npm init and npm install request did the trick.

Answer 28 · 2021-08-02T15:32:10.000Z

Great to hear. No that you have a fork anyway, it might be worth sending a pull request to the original repos. That way other can benefit from your contribution.

Answer 29 · 2021-08-02T15:35:50.000Z

That's a good idea. The package.json looks like this atm

{ "name": "test", "version": "0.1.0", "dependencies": { "request": "*" } }

Can I leave it like this or do we need some additional informations?

Answer 30 · 2021-08-10T18:20:05.000Z

Maybe 3rd-party modules shouldn't be able to access dependencies of MagicMirror at all to avoid this kind of problems in the future?

Or maybe it would be enough to update the module development to actually encourage people to create a package.json

Answer 31 · 2021-08-10T18:40:32.000Z

@buxxi many of these modules are many years.old, no longer maintained. the developers ( probably nearly all not professional developers) didn't know and it worked just fine.

also, I'm not aware of any way to prevent npm from doing its designed job of managing the package list.
it looks up the directory tree to find libraries that satisfy the 'require()'

Answer 32 · 2021-08-10T19:23:57.000Z

Of course I know that's the case, just think it's kind of bad design to depend on an internal dependency which could probably be solved by a stricter api for how modules are implemented.

Breaking 3rd-party modules will always be a part of releasing new versions, if it's not a dependency it will be a tag in the html changing name, having another css-class or accessing a "private" field in MagicMirror. Being clear with what you should do and not do could at least make the "professional" developers not make this mistake in their modules (I started my own modules without package.json, so maybe one of my own could have this problem).

I know too little about the whole npm ecosystem, just thought maybe it was possible to sandbox each node_helper to only have access to its own node_modules. If not denying it, maybe a warn log would be enough if that's possible?

Answer 33 · 2021-08-10T19:59:21.000Z

Here the node documentation: https://nodejs.org/api/modules.html#modules_loading_from_node_modules_folders

Answer 34 · 2021-08-10T22:38:08.000Z

Yeah, doesn't seem to be anything there.

Being a professional Java-dev I thought maybe something like a custom class loader would be possible in other languages too (like application servers isolate each app).

Forget that suggestion :)

Broken unmaintained modules with enough demand will probably get a fork.

Answer 35 · 2021-12-22T07:41:24.000Z

CFenner/MMM-Sonos is fixed.

Answer 36 · 2022-04-17T07:07:55.000Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.