planarnetwork/dtd2mysql

npm install - looking for more instructions!

Closed this issue · 13 comments

eoghanmurray commented

Some notes (first time with npm so forgive my ignorance!):

  • When in the cloned folder, running npm install -g dtd2mysql as instructed actually downloads and installs the package from npmjs.com. This is obviously npm's fault for overloading the meaning of 'install', I presume it's intended to be executed from the parent folder of the project root folder (which feels weird!)

  • got an error npm WARN checkPermissions Missing write access to ~/projects/dtd2mysql/node_modules/minimist not sure why, but I fixed it with sudo npm i -g minimist

  • how is the dtd2mysql/dist/ folder generated? It looks to have installed correctly, but hangs when I run dtd2mysql -v ... running node /usr/lib/node_modules/dtd2mysql/bin/dtd2mysql.sh gives Error: Cannot find module '../dist/src/index.js', and indeed there is no dist folder in ~/projects/dtd2mysql/ am I missing a step?

    • Update: I saw the prepublishOnly under scripts in package.json and I tried running tsc -p ./ --outDir dist/ manually and found I was missing tsc. Running sudo apt install typescript as suggested by my shell was the wrong thing and I eventually got it with sudo npm install -g typescript. The dist folder still didn't get generated or give an error message after npm install -g dtd2mysql, but a manual run of tsc -p ./ --outDir dist/ outputs that directory now

  • Now running the global dtd2mysql still hangs with no error message, but I can run it manually with node /usr/lib/node_modules/dtd2mysql/bin/dtd2mysql.sh

Anyhow, thanks for your patience — I know it's onerous converting a project that just works on your own machine into something reproducible elsewhere; hopefully something that I missed from above can be added to the build instructions or package.json file.

linusnorton commented

Hi,

No problem at all - I'll be happy to make things more robust. I should probably make it clearer that you don't need to do the git clone.

You start with npm install -g dtd2mysql. You also shouldn't need typescript installed as the prepublishOnly script happens on my machine before publishing the package to the npm repository.

However, I can confirm that just running the global dtd2mysql command is not working. It looks like there is a relative path in there that it does not like. I will see if I can fix that

linusnorton commented

Okay, something I didn't expect. The GTFS command requires a bit of extra memory, however setting the --max_old_space_size=2048 in the shell command seems to upset node. I will remove it and find another solution for the GTFS command.

linusnorton commented

Apparently env doesn't pass on any arguments to node so it was getting very confused. Fixed with version 3.0.7

mk-fg commented

I think update of the dependency packages in general made script run way slower and require way more memory.

Re-running the thing on the same data I've used ~10 times before (to test #4, jic), but with deps updated after pull, have it run for 30+ min now, and going, while fairly sure it used to finish within 20 on this exact setup before.

mk-fg commented

I think update of the dependency packages in general made script run way slower and require way more memory.

Nope, sorry, double-checked, and it was definitely an issue on my end, please ignore.
Runs reasonably fine, 12min for the whole process.

Probably didn't have all (hdd-backed) db data cached in memory or something after dev machine reboot, and rarely rebooted it between previous runs, hence the wrong expectation.

eoghanmurray commented

I should probably make it clearer that you don't need to do the git clone

So I guess the first 'Install' section could be entitled 'Download and Run', and then there could be a more detailed section for developers who wish to alter the code, maybe under the 'Contributing' section?

linusnorton commented

I've updated the readme! Let me know how you get on. I have further documentation on the fares information if you need it.

mk-fg commented

I've updated the readme! Let me know how you get on.

My experience with trying to make it work just now:

% mkdir clean-install
% cd clean-install

% npm install dtd2mysql
npm WARN saveError ENOENT: no such file or directory, open 'clean-install/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open 'clean-install/package.json'
npm WARN node-installs No description
npm WARN node-installs No repository field.
npm WARN node-installs No README data
npm WARN node-installs No license field.

% ./node_modules/.bin/dtd2mysql --fares ../RJFAF499.ZIP
Extracting feeds/RJFAF499.ZIP to /tmp/dtd/fares/
Finished RJFAF499.RCM
Finished RJFAF499.RLC
Finished RJFAF499.TOC
Finished RJFAF499.TAP
Finished RJFAF499.TVL
Finished RJFAF499.SUP
Finished RJFAF499.TSP
Finished RJFAF499.TTY
Finished RJFAF499.RTE
Finished RJFAF499.DIS
Finished RJFAF499.FSC
Finished RJFAF499.LOC
Finished RJFAF499.RST
Finished RJFAF499.FNS
{ Error: Duplicate entry 'C-5h' for key 'restriction_header_key'
    at PromisePool.query (clean-install/node_modules/mysql2/promise.js:299:20)
    at MySQLTable.flush (clean-install/node_modules/dtd2mysql/dist/src/database/MySQLTable.js:29:33)
    at MySQLTable.close (clean-install/node_modules/dtd2mysql/dist/src/database/MySQLTable.js:37:14)
    at console.log.Promise.all.Object.values.map.t (clean-install/node_modules/dtd2mysql/dist/src/cli/ImportFeedCommand.js:87:134)
    at Array.map (native)
    at Interface.readStream.on (clean-install/node_modules/dtd2mysql/dist/src/cli/ImportFeedCommand.js:87:123)
    at emitNone (events.js:110:20)
    at Interface.emit (events.js:207:7)
    at Interface.close (readline.js:366:8)
    at ReadStream.onend (readline.js:146:10)
    at emitNone (events.js:110:20)
    at ReadStream.emit (events.js:207:7)
    at endReadableNT (_stream_readable.js:1045:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
  message: 'Duplicate entry \'C-5h\' for key \'restriction_header_key\'',
  code: 'ER_DUP_ENTRY',
  errno: 1062,
  sqlState: '#23000' }
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Pool is closed.
(node:20132) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Done
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): Error: Pool is closed.
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 3): Error: Pool is closed.
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 4): Error: Pool is closed.
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 5): Error: Pool is closed.
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 6): Error: Pool is closed.
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 7): Error: Pool is closed.
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 8): Error: Pool is closed.
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 9): Error: Pool is closed.
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 10): Error: Pool is closed.
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 11): Error: Pool is closed.
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 12): Error: Pool is closed.
(node:20132) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 13): Error: Pool is closed.
...

And 100s more of these ^^^
Assuming that mysql connection got closed and everything's failing from then on.

Maybe there should be check to stop on first fatal error instead of proceeding to bury it under these warnings? -- issue1

This error seem to be addressed in #8, which was recently merged here, but guess isn't on npm yet, trying to install from the repo directly:

% mkdir clean-install
% cd clean-install

% npm install git+https://github.com/open-track/dtd2mysql/
npm WARN saveError ENOENT: no such file or directory, open 'clean-install/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open 'clean-install/package.json'
npm WARN node-installs No description
npm WARN node-installs No repository field.
npm WARN node-installs No README data
npm WARN node-installs No license field.

% ./node_modules/.bin/dtd2mysql --fares ../RJFAF499.ZIP
module.js:487
    throw err;
    ^

Error: Cannot find module '../dist/src/index.js'
    at Function.Module._resolveFilename (module.js:485:15)
    at Function.Module._load (module.js:437:25)
    at Module.require (module.js:513:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (clean-install/node_modules/dtd2mysql/bin/dtd2mysql.sh:3:1)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)

Given that it's certainly possible to have dev version not yet on npm (example: gtfs branch), maybe there should be working instructions on how to make it work with "npm install"? -- issue2

Guess I'll end up with "git clone", only installing dependencies and using the thing via "ts-node ./src/index.ts", as that seem to work, but feel like there's gotta be an easier "npm install" oneliner, just not as obvious as straight-up npm install git+https://github.com/open-track/dtd2mysql/.

linusnorton commented

Yeah sorry there is a fix in master but I didn't publish yet. Got some man flu going on. Will publish new version later

mk-fg commented

No worries, had similar thing just last week myself :(
Though I feel like it'd be more useful to have info on how to install/run the thing from git in the README, rather than bringing npm up to speed in this one case (already moved past this myself, works fine from a checkout).

linusnorton commented

I've published the latest version with the fix for that file. Let me know if there are any further problems

eoghanmurray commented

Just on this, I still haven't gotten in working after clone without the tsc -p ./ --outDir dist/ step.
In the Contributing instructions, it would be great to see exactly what you do after making a change to run the updated version of ./bin/dtd2mysql.sh (the next step after npm test)?

linusnorton commented

typically just ts-node ./src/index.ts --command [options]