ami-iit/yarp-openmct

Enable a one line installation command through npm: skip the manual repo download

nunoguedelha opened this issue · 11 comments

nunoguedelha commented

In view of simplifying as much as possible the installation procedure, we would like to avoid the repository download step if the npm package manager workflow allows it.
After this optimization, the installation procedure would be like:

  1. In a terminal, go to the target folder which will contain the yarp-openmct repository, eg: cd ~/dev
  2. run the command: npm install https://github.com/dic-iit/yarp-openmct.git.

...and you would be done. The package manager would download the repository and install both the iCubTelemVizServer and openmctStaticServer folders content.

Implemented by #97 .

nunoguedelha commented

npm-install command

npm install <package> installs a package and any packages that it depends on (refer to the documentation in https://docs.npmjs.com/cli/v8/commands/npm-install), a package being:

  • a) a folder containing a program described by a package.json file
  • b) a gzipped tarball containing (a)
  • c) a url that resolves to (b)
  • d) a @ that is published on the registry (see registry) with (c)
  • e) a @ (see npm dist-tag) that points to (d)
  • f) a that has a "latest" tag satisfying (e)
  • g) a that resolves to (a)

So we can install the package by running

npm install <git remote url>[#<commit-ish>]

This command clones the repository with the protocols git, git+ssh, git+http, git+https, or git+file. Using git+http or git+https protocols is probably more widely used in the lab and only requires a token generated on Github. This protocol can also be used with the command:

npm install github:<githubname>/<githubrepo>[#<commit-ish>]

Install the package at https://github.com/githubname/githubrepo by attempting to clone it using git.

If # is provided, it will be used to clone exactly that commit. If the commit-ish has the format #semver:, can be any valid semver range or exact version, and npm will look for any tags or refs matching that range in the remote repository, much as it would for a registry dependency. If neither # or #semver: is specified, then master is used.

As with regular git dependencies, dependencies and devDependencies will be installed if the package has a prepare script before the package is done installing.

npm install mygithubuser/myproject
npm install github:mygithubuser/myproject

A package.json file is required for the package installation, it's the installation recipe. The repository yarp-openmct has two sub-modules each installed separately, as they have their own package.json and node_modules. We have two possible approaches for running a single installation from the repository root or using the repo URL target:

  1. Merge the two package.json into a single one located at the root of the repo.
  2. Keep the two original package.json files, but removing from them all but the dependencies, and pointing to them from a package.json under the repo root.

Option 2 works for a local installation after cloning the repo manually. But running npm install <git remote url> fails with the error:

npm ERR! premature close

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/nunoguedelha/.npm/_logs/2022-02-24T23_49_49_494Z-debug.log

But the log doesn't give much more insight on the problem.

As a first step to solve this issue, we go back to installing a simpler repository with a single package.json, yarp.js.

nunoguedelha commented

npm Install using Yarp.js URL

We first run, in an empty folder, the command

npm install https://github.com/robotology/yarp.js.git

The cmake-js compilation step fails.

We run the command again after removing the cmake-js compilation step

npm install https://github.com/robotology/yarp.js.git#2ea01c35f875743b30ebc23fcf4d57e2788fcd45

We get the console output:

npm WARN saveError ENOENT: no such file or directory, open '/Users/nunoguedelha/dev/testYarpOpenMct-install/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/Users/nunoguedelha/dev/testYarpOpenMct-install/package.json'
npm WARN testYarpOpenMct-install No description
npm WARN testYarpOpenMct-install No repository field.
npm WARN testYarpOpenMct-install No README data
npm WARN testYarpOpenMct-install No license field.

+ YarpJS@1.5.0
added 175 packages from 157 contributors and audited 175 packages in 11.818s

4 packages are looking for funding
  run `npm fund` for details

found 1 low severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details

Actually the command doesn't clone the repo content in the current folder, but rather as a node.js module under node_modules, along with the package dependencies:

node_modules
|----YarpJS
|     |----yarp.js
|     |----images
|     |----js
|     |----YarpJS
|     |----LICENSE
|     |----package.json
|     |----README.md
|     |----examples
|     |----CMakeLists.txt
|
|----chainsaw
|----destroy
|----@types
|----...
|
package-lock.json

Analyzing...

nunoguedelha commented

We can find in the Node.js documentation (https://nodejs.dev/learn/npm-dependencies-and-devdependencies):

When you install an npm package using npm install , you are installing it as a dependency.

@traversaro , I can't see a way out of this for now. npm install https://github.com/robotology/yarp.js.git is installing the package in the node_modules folder.

traversaro commented

Just to understand, how is this different from when YarpJS is installed as a dependency in step 7 and 8 of https://github.com/ami-iit/yarp-openmct ? If we go directly to install yarp-openmct with a one-line install bypassing the one-line installation of yarpjs (that anyhow we do not need) we would have the same problem?

nunoguedelha commented

We would have the same problem, at least partially. The fact that the package yarp-openmct would appear as a dependency under node_modules among all the other dozens of dependencies, wouldn't be a problem per se, but I suspect the location of the installed package under node_modules is causing the cmake-js step to fail in the case of yarp.js.

If after the yarp.js is installed we then go to the downloaded repo folder (./node_modules/YarpJS) we can run the cmake-js successfully.

I'm trying now with a new config option: the global-style. This causes npm to install the package into your local node_modules folder with the same layout it uses with the global node_modules folder. Only your direct dependencies will show in node_modules and everything they depend on will be flattened in their node_modules folders. This obviously will eliminate some deduping.

npm install github:robotology/yarp.js#2ea01c35f875743b30ebc23fcf4d57e2788fcd45 --global-style true

This results in the tree structure

|----node_modules
|     |----YarpJS
|     |     |----yarp.js
|     |     |----images
|     |     |----js
|     |     |----node_modules
|     |     |----YarpJS
|     |     |----LICENSE
|     |     |----package.json
|     |     |----README.md
|     |     |----examples
|     |     |----build
|     |     |----CMakeLists.txt
|----package-lock.json

and the cmake-js step succeeds.

I will try again the yarp-openmct install without the option global-style to check how the YarpJS dependency is handled, then try with the new option if till needed.

References

https://docs.npmjs.com/cli/v8/using-npm/config
https://docs.npmjs.com/cli/v8/configuring-npm/folders
https://docs.npmjs.com/cli/v8/commands/npm-install

nunoguedelha commented

I will try again the yarp-openmct install without the option global-style to check how the YarpJS dependency is handled,

This is related to the point you made before: YarpJS being already a dependency downloaded from Github from the beginning, it will probably have its own node_modules as in the original yarp-openmct install (with the manual cloning of the repo). That's why I want to check the install without the global-style option.

nunoguedelha commented

As the yarp-openmct was failing with option 2 of #95 (comment)...

  1. Merge the two package.json into a single one located at the root of the repo.
  2. Keep the two original package.json files, but removing from them all but the dependencies, and pointing to them from a package.json under the repo root.

...for starting I'm going for option 1.

nunoguedelha commented

@traversaro , I've edited #95 (comment):

the cmake-js step succeeds.

  • 👍1
nunoguedelha commented

Installing yarp-openmct without the global-style option

  1. Keep the two original package.json files, but removing from them all but the dependencies, and pointing to them from a package.json under the repo root.

I hadn't selected the right commit in my previous trial. With the proper commit we run

npm install https://github.com/dic-iit/yarp-openmct.git#f4bdff80fa8e1d2f3ddc57cf931308bb9275748c

which fails with

npm ERR! code ENOLOCAL
npm ERR! Could not install from "node_modules/yarp-openmct/iCubTelemVizServer" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/nunoguedelha/.npm/_logs/2022-02-25T14_57_31_850Z-debug.log
  1. Merge the two package.json into a single one located at the root of the repo.

Works better but still fails when compiling YarpJS.node with cmake-js:

...
[11/12] Building CXX object CMakeFiles/YarpJS.dir/YarpJS/src/YarpJS_Image.cpp.o
FAILED: CMakeFiles/YarpJS.dir/YarpJS/src/YarpJS_Image.cpp.o
/Users/nunoguedelha/miniforge3/envs/icubtelemviz/bin/x86_64-apple-darwin13.4.0-clang++ -DYarpJS_EXPORTS -I/Users/nunoguedelha/.cmake-js/node-x64/v14.17.0/include/node -I/Users/nunoguedelha/dev/testYarpOpenMct-install/node_modules/YarpJS/YarpJS/include -isystem /Users/nunoguedelha/miniforge3/envs/icubtelemviz/include/opencv4 -isystem /Users/nunoguedelha/miniforge3/envs/icubtelemviz/include -D_DARWIN_USE_64_BIT_INODE=1 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DBUILDING_NODE_EXTENSION -O3 -DNDEBUG -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC -MD -MT CMakeFiles/YarpJS.dir/YarpJS/src/YarpJS_Image.cpp.o -MF CMakeFiles/YarpJS.dir/YarpJS/src/YarpJS_Image.cpp.o.d -o CMakeFiles/YarpJS.dir/YarpJS/src/YarpJS_Image.cpp.o -c /Users/nunoguedelha/dev/testYarpOpenMct-install/node_modules/YarpJS/YarpJS/src/YarpJS_Image.cpp
In file included from /Users/nunoguedelha/dev/testYarpOpenMct-install/node_modules/YarpJS/YarpJS/src/YarpJS_Image.cpp:6:
/Users/nunoguedelha/dev/testYarpOpenMct-install/node_modules/YarpJS/YarpJS/include/YarpJS_Image.h:8:10: fatal error: 'nan.h' file not found
#include <nan.h>
         ^~~~~~~
1 error generated.
ninja: build stopped: subcommand failed.
ERR! OMG Process terminated: 1
...

because of header nan.h not found. Package nan is one of the yarp.js dependencies., usually found under node_modules, and now missing.

I'm trying now with the option global-style set to true...

nunoguedelha commented

Installing yarp-openmct with the global-style option = true

npm install https://github.com/dic-iit/yarp-openmct.git#fcc5c20b8f98bc0b8a9af2eafb5bc562528dcf5f --global-style true

This installation is successful. Next step is to try and run it. Still, we should understand why nan is not found when global-style = true, by looking at the compile include options.

nunoguedelha commented

Run yarp-openmct with the global-style option set to true

Error 1

Fails: the Open-MCT visualizer on the browser only displays an empty white window.

The openmctStaticServer/server.js itself is failing, as we observe the same failure if we run it standalone, without the telemetry server iCubTelemVizServer/iCubTelemVizServer.js. We take a closer look at the merge of the two package.json files from these to sub-modules.

On the Merge of the two package.json into a single one located at the root of the repo

package.json on iCubTelemVizServer:

yarp-openmct/iCubTelemVizServer/package.json

Lines 7 to 14 in 89d2e47

"dependencies": {
"express": "latest",
"express-ws": "2.0.0",
"jquery": "latest",
"socket.io": "1.5.1",
"socket.io-stream": "latest",
"YarpJS": "https://github.com/robotology/yarp.js"
},

package.json on openmctStaticServer:

yarp-openmct/openmctStaticServer/package.json

Lines 7 to 12 in 89d2e47

"dependencies": {
"express": "^4.17.1",
"express-ws": "^4.0.0",
"openmct": "https://github.com/nasa/openmct.git#1.7.8",
"ws": "^7.2.0"
},

package.json on the root:

yarp-openmct/package.json

Lines 7 to 16 in d53f562

"dependencies": {
"express": "latest",
"express-ws": "4.0.0",
"jquery": "latest",
"socket.io": "1.5.1",
"socket.io-stream": "latest",
"YarpJS": "https://github.com/robotology/yarp.js#2ea01c35f875743b30ebc23fcf4d57e2788fcd45",
"openmct": "https://github.com/nasa/openmct.git#1.7.8",
"ws": "^7.2.0"
},

Installed packages versions check

Module Required Resolved (before merge) Resolved (after merge)
iCubTelemVizServer/package.json:
┝╾ express latest 14.17.2 14.17.3
┝╾ express-ws 2.0.0 2.0.0 4.0.0
┝╾ ws ˆ1.0.0 5.2.0 5.2.3
┝╾ jquery latest 3.6.0 3.6.0
┝╾ socket.io 1.5.1 1.5.1 4.4.0
┝╾ socket.io-stream latest 0.9.1 0.9.1
┕╾ YarpJS https://github.com/robotology/yarp.js" master d2e4609
openmctStaticServer/package.json:
┝╾ express ^4.17.1 4.17.1 14.17.3
┝╾ express-ws ^4.0.0 (^5.2.0) 4.0.0 (ws: 5.2.2) 4.0.0 (ws: 5.2.3)
┝╾ openmct https://github.com/nasa/openmct.git#1.7.8 ab7e2c5 ab7e2c5
┕╾ ws ^7.2.0 7.4.6 7.5.7

There are a few upgrades, but there seems to be nothing wrong with the package versions. There isn't either any error returned from command npm ls (package dependency tree listing).

Inaccessible scripts folders

If we run the visualizer server https://github.com/ami-iit/yarp-openmct/blob/install/one-line-installer/openmctStaticServer/index.html, then the plugin installation starter page https://github.com/ami-iit/yarp-openmct/blob/install/one-line-installer/openmctStaticServer/server.js in debug mode, we get the console error message:

Failed to load resource: the server responded with a status of 404 (Not Found)
http://…essedDefault.jsonFailed to load resource: the server responded with a status of 404 (Not Found)
index.html

So the following script paths couldn't be found:

yarp-openmct/openmctStaticServer/index.html

Line 5 in d53f562

<script src="openmctdist/openmct.js"></script>

yarp-openmct/openmctStaticServer/index.html

Line 12 in d53f562

<script src="config/processedDefault.json"></script>

This relates directly to the static paths routed in

yarp-openmct/openmctStaticServer/static-server.js

Lines 7 to 8 in d53f562

router.use('/config', express.static(__dirname + '/../config'));
router.use('/openmctdist', express.static(__dirname + '/node_modules/openmct/dist'));
.

In debug mode we can see that __dirname resolves to <yarp-openmct-root-path>/openmctStaticServer, such that:

  • /config is routed to <yarp-openmct-root-path>/openmctStaticServer/../config => √
  • /openmctdist is routed to <yarp-openmct-root-path>/openmctStaticServer/node_modules/openmct/dist => 🚫 wrong path!

=> Fixed the route path:

router.use('/openmctdist', express.static(__dirname + '/../node_modules/openmct/dist'));

Error 2

image

Same problem but this time with the telemetry data server:

yarp-openmct/iCubTelemVizServer/iCubTelemVizServer.js

Lines 29 to 30 in d53f562

app.use(express.static(__dirname + '/node_modules'));
app.use(express.static(__dirname + '/node_modules/YarpJS/js'));

As the client script index.html requests the load of yarp.js script...

yarp-openmct/iCubTelemVizServer/index.html

Lines 84 to 85 in d53f562

<script src="/socket.io/socket.io.js"></script>
<script src="/yarp.js"></script>

..., while the static paths are wrong, the server tries to fetch yarp.js from <yarp-openmct-root>/iCubTelemVizServer/node_modules instead of <yarp-openmct-root>/node_modules.

Fixed the paths:

app.use(express.static(__dirname + '/../node_modules'));
app.use(express.static(__dirname + '/../node_modules/YarpJS/js'));