Farmer crashes after daemon 4.0.0/4.0.1 upgrade (which includes v7.0.0 of core)
Closed this issue · 5 comments
Package Versions
Replace the values below using the output from npm list storj. Use npm list -g storj if installed globally.
storjshare-daemon: 4.0.1
core: 7.0.0
Replace the values below using the output from node --version.
v6.11.2
Expected Behavior
Please describe the program's expected behavior. Include an example of your
usage code in the back ticks below if applicable.
The farmer doesn't crash.
Actual Behavior
Please describe the program's actual behavior. Please include any stack traces
or log output in the back ticks below.
The farmer crashes (randomly), but only one node seems to be affected. Other nodes are in different geolocations.
/home/felix/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/storj-lib/lib/network/farmer.js:447
const sigObj = secp256k1.sign(sighash, privkey);
^
RangeError: private key length is invalid
at RangeError (native)
at FarmerInterface.bridgeRequest (/home/felix/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/storj-lib/lib/network/farmer.js:447:28)
at ShardServer._sendExchangeReport (/home/felix/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/storj-lib/lib/network/shard-server.js:150:24)
at ReadableFileStream._handleTransferFinish (/home/felix/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/storj-lib/lib/network/shard-server.js:312:12)
at emitNone (events.js:91:20)
at ReadableFileStream.emit (events.js:185:7)
at endReadableNT (/home/felix/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/readable-stream/lib/_stream_readable.js:992:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickDomainCallback (internal/process/next_tick.js:128:9)
Steps to Reproduce
Please include the steps the reproduce the issue, numbered below. Include as
much detail as possible.
I have not yet found a way to reproduce the problem.
I was also having this problem after upgrading. I have 6 nodes on that box but only one was having this problem. It was so bad that the node stopped after 30 restart re-trys.
I thought I was able to resolve this problem by stopping all nodes:
storjshare-killall
Then re-trying the upgrade:
npm install --global storjshare-daemon
Then restarting all the nodes.
All nodes have been running with no issues for 10 mins. Will keep an eye on it.
Update: After 1hr 41mins the node restarted. It has restarted a total of 4 times in the last 10 minutes. Storage for the node had grown by 130 MB and everything else looked ok. Same error when it restarted.
/usr/lib/node_modules/storjshare-daemon/node_modules/storj-lib/lib/network/farmer.js:447
const sigObj = secp256k1.sign(sighash, privkey);
^
RangeError: private key length is invalid
at FarmerInterface.bridgeRequest (/usr/lib/node_modules/storjshare-daemon/node_modules/storj-lib/lib/network/farmer.js:447:28)
at ShardServer._sendExchangeReport (/usr/lib/node_modules/storjshare-daemon/node_modules/storj-lib/lib/network/shard-server.js:150:24)
at ReadableFileStream._handleTransferFinish (/usr/lib/node_modules/storjshare-daemon/node_modules/storj-lib/lib/network/shard-server.js:312:12)
at emitNone (events.js:91:20)
at ReadableFileStream.emit (events.js:188:7)
at endReadableNT (/usr/lib/node_modules/storjshare-daemon/node_modules/readable-stream/lib/_stream_readable.js:992:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickDomainCallback (internal/process/next_tick.js:128:9)
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"75.158.76.73\",\"port\":63201,\"nodeID\":\"f491e38ce1e2a6b6db78686f5700bfbdb0cc30a6\",\"lastSeen\$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"75.158.76.73\",\"port\":63201,\"nodeID\":\"f491e38ce1e2a6b6db78686f5700bfbdb0cc30a6\",\"lastSeen\$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"75.158.76.73\",\"port\":63201,\"nodeID\":\"f491e38ce1e2a6b6db78686f5700bfbdb0cc30a6\",\"lastSeen\$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"218.206.79.93\",\"port\":40002,\"nodeID\":\"f4f60a7f58126c437878da71f52ce9af4ece592b\",\"lastSeen$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"realshels.zapto.org\",\"port\":4120,\"nodeID\":\"f4df03233b3d7e50add087e17607cbc8d96535b6\",\"las$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"75.158.76.73\",\"port\":63201,\"nodeID\":\"f491e38ce1e2a6b6db78686f5700bfbdb0cc30a6\",\"lastSeen\$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"realshels.zapto.org\",\"port\":4120,\"nodeID\":\"f4df03233b3d7e50add087e17607cbc8d96535b6\",\"las$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"realshels.zapto.org\",\"port\":4120,\"nodeID\":\"f4df03233b3d7e50add087e17607cbc8d96535b6\",\"las$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.5.0\",\"protocol\":\"1.1.0\",\"hdKey\":\"xpub6AHweYHAxk1EhJSBctQD1nLWPog6Sy2eTpKQLExR1hfzTyyZQWvU4EYNXv1NJN7GpLYXnDLt4PzN874g6zSjAQdFCHZN7U7$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.7.0\",\"protocol\":\"1.1.0\",\"address\":\"5.9.150.177\",\"port\":4200,\"nodeID\":\"f4cd9d26afad18b790db7e49b03ebf5e427537c9\",\"lastSeen\":$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"91.218.249.43\",\"port\":4000,\"nodeID\":\"f4e78f2f07532350958b27f3d94c30d6bf1d9ba8\",\"lastSeen\$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"104.168.62.178\",\"port\":4000,\"nodeID\":\"f4fd47716ecfdb1eaedf54a4f41fb01f5baf3086\",\"lastSeen$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"35.195.166.48\",\"port\":48407,\"nodeID\":\"f6949f5fbf8758807416c1a2e2e93ea316641ce5\",\"lastSeen$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"35.195.166.48\",\"port\":48407,\"nodeID\":\"f6949f5fbf8758807416c1a2e2e93ea316641ce5\",\"lastSeen$
{"level":"error","message":"failed to relay publication to random neighbor {\"userAgent\":\"6.8.0\",\"protocol\":\"1.1.0\",\"address\":\"45.32.72.183\",\"port\":4000,\"nodeID\":\"f5957d2ed5cae32efed105bd371320c3f047e242\",\"lastSeen\"$
If you can get it, what's the length of your private keys? Have a feeling if it starts with two 0s then it chops off the first one
Found the issue, will have PR for it shortly.
i have taken a look at the private key in question, it does not start with any zeros:
f263cd7a83f455ab5fb54fe74e81f1c50f7f83fc45dedd23f00b860a37f5f3
i have changed one character in the middle, but i really do not care if somebody where to find the real key, i do not use it anymore.
i have not yet tried running that share yet again
Edit: I have taken a closer look and the key is in fact two chars too short. Possibly the share creation already skipped the zeros so there weren't any in the config file to begin with, then of course this fix will apply. Thanks
Yep, there are two leading zeros for that, the length is 62, when it should be 64. It's how the big number for the key is made into a string. The logic for why it's not padded is because we don't usually write 000010 and 000011, however with private keys the length is supposed to always be the same at 32 bytes.