storj-archived/kfs

Corrupted database

littleskunk opened this issue · 7 comments

littleskunk commented

Package Versions

Replace the values below using the output from storjshare --version.

deamon 2.4.3
core 6.2.0
protocol 1.1.0

Replace the values below using the output from node --version.

6.9.2

Actual Behavior

program does restarts 15-30 times a day

{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.013Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.013Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.013Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.013Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.014Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.014Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.014Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.014Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.014Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.014Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.014Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.014Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.014Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.015Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.015Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.015Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.015Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.015Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.015Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.015Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file specified.\r\n","timestamp":"2017-02-23T00:02:27.015Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\020.s/MANIFEST-000368: The system cannot find the file 

{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\024.s/MANIFEST-000359: The system cannot open the file.\r\n","timestamp":"2017-02-23T00:02:49.033Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\059.s: An unexpected network error occurred.\r\n","timestamp":"2017-02-23T00:02:49.410Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\059.s: An unexpected network error occurred.\r\n","timestamp":"2017-02-23T00:02:49.410Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\059.s: An unexpected network error occurred.\r\n","timestamp":"2017-02-23T00:02:49.411Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\031.s/000995.log: The system cannot open the file.\r\n","timestamp":"2017-02-23T00:02:49.420Z"}

{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.070Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.070Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.070Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.070Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.133Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.136Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.146Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.151Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.164Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.167Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.167Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.168Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.170Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.182Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.219Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.271Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.426Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.432Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.438Z"}
{"level":"warn","message":"missing or empty reply from contact","timestamp":"2017-02-23T00:59:58.467Z"}
{"level":"error","message":"Could not get usedSpace: IO error: U:\\TUNEL_CLI_B_800gb\\sharddata.kfs\\149.s: An unexpected network error occurred.\r\n","timestamp":"2017-02-23T00:59:58.600Z"}

Power loss or similar problems can damage the leveldb. This was reported a few times now from GUI and daemon users. I think we need the good old repair command again or even better avoid it somehow.

Workaround:
npm install -g kfs@3.0.0
kfs -d /path-to-sharddata.kfs compact

50% chance that this can repair the leveldb.

ne0ark commented

I am also still getting {"level":"error","message":"Could not get usedSpace: Corruption: CURRENT file does not end with newline","timestamp":"2017-04-12T04:40:47.331Z"}. Is it possible to re-add repair functionality mentioned here Level/levelup#236 .

drygoatair commented

For those landing on this page not knowing how to repair the database:

  1. Download and install node.js
  2. Open a command promt (Start -> Search -> 'cmd' + enter)
  3. Run commands from top post
drygoatair commented

@littleskunk If I can't repair my db, how should I proceed? Delete Storj data and create a new node?

user01010111 commented

I'm run in to this issue again. After upgrading to the latest version it's now throwing this error again:

{"level":"info","timestamp":"1503395192217","message":"Log file created."}

events.js:160
      throw er; // Unhandled 'error' event
      ^
OpenError: Corruption: no meta-nextfile entry in descriptor
    at /home/ethos/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/levelup/lib/levelup.js:117:34
    at /home/ethos/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/abstract-leveldown/abstract-leveldown.js:39:16

events.js:160
      throw er; // Unhandled 'error' event
      ^
OpenError: Corruption: no meta-nextfile entry in descriptor
    at /home/ethos/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/levelup/lib/levelup.js:117:34
    at /home/ethos/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/abstract-leveldown/abstract-leveldown.js:39:16
OpenError: Corruption: no meta-nextfile entry in descriptor
    at /home/ethos/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/levelup/lib/levelup.js:117:34
    at /home/ethos/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/abstract-leveldown/abstract-leveldown.js:39:16

events.js:160
      throw er; // Unhandled 'error' event
      ^

Running the workaround listed above has done nothing.

What do I need to do to fix this? I have about 200GB of seeded data I'd rather not abandon.

user01010111 commented

@littleskunk any ideas on the post above please? kfs -d <shard> compact does nothing to resolve :(

events.js:160
      throw er; // Unhandled 'error' event
      ^
OpenError: Corruption: no meta-nextfile entry in descriptor
    at /home/ethos/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/levelup/lib/levelup.js:117:34
    at /home/ethos/.nvm/versions/node/v6.11.2/lib/node_modules/storjshare-daemon/node_modules/abstract-leveldown/abstract-leveldown.js:39:16

It looks like storjshare repair-db used to look after this, but that command has been deprecated? I've found reference to needing to call leveldown.repair() but I'm confused as to how to do this?

user01010111 commented

.. Or not.

RichardLitt commented

👋 Hey! Thanks for this contribution. Apologies for the delay in responding!

We've decided to rearchitect Storj, so that we can scale better. You can read more about this decision here. This means that we are entirely focused on v3 at the moment, in the storj/storj repository. Our white paper for v3 is coming very, very soon - follow along on the blog and in our Rocketchat.

As this repository is part of the v2 network, we're no longer maintaining this repository. I am going to close this for now. If you have any questions, I encourage you to jump on Rocketchat and ask them there. Thanks!