Aquila-Network/aquila

Docker Volume Problem

sopaoglu opened this issue · 6 comments

sopaoglu commented

I am using following YAML to run AquilaDb

version: '3'
services:
  aquiladb:
    image: ammaorg/aquiladb
    ports:
      - "50051:50051"
    volumes:
      - /home/asd/db-data:/data
    restart: always
volumes:
  db-data:

But when I remove the container, I start again it but all the indexes are lost. Probably It happens to because all files under default_docsdb are removed and new ones are added.

sopaoglu commented

Probably, when model_hf file is loaded, a problem occurred because when I delete the model_hf file then I restart the container, everything looks okay.

freakeinstein commented

Could you please check the logs this way and share it here if you find any error?

  1. run docker ps and note down container id
  2. run docker exec -i -t <container id> /bin/bash
  3. run pm2 logs
sopaoglu commented

When I start first, the following logs are produced:


PM2        | 2019-12-30T12:14:02: PM2 log: PM2 home             : /root/.pm2
PM2        | 2019-12-30T12:14:02: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
PM2        | 2019-12-30T12:14:02: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
PM2        | 2019-12-30T12:14:02: PM2 log: BUS socket file      : /root/.pm2/pub.sock
PM2        | 2019-12-30T12:14:02: PM2 log: Application log path : /root/.pm2/logs
PM2        | 2019-12-30T12:14:02: PM2 log: Process dump file    : /root/.pm2/dump.pm2
PM2        | 2019-12-30T12:14:02: PM2 log: Concurrent actions   : 2
PM2        | 2019-12-30T12:14:02: PM2 log: SIGTERM timeout      : 1600
PM2        | 2019-12-30T12:14:02: PM2 log: ===============================================================================
PM2        | 2019-12-30T12:14:02: PM2 log: App [vecdb:0] starting in -fork mode-
PM2        | 2019-12-30T12:14:02: PM2 log: App [vecdb:0] online
PM2        | 2019-12-30T12:14:03: PM2 log: App [peer_manager:1] starting in -fork mode-
PM2        | 2019-12-30T12:14:03: PM2 log: App [peer_manager:1] online
PM2        | 2019-12-30T12:14:03: PM2 log: App [vecstore:2] starting in -fork mode-
PM2        | 2019-12-30T12:14:03: PM2 log: App [vecstore:2] online

/root/.pm2/logs/vecdb-error.log last 15 lines:
/root/.pm2/logs/peer-manager-error.log last 15 lines:
/root/.pm2/logs/vecstore-error.log last 15 lines:
/root/.pm2/logs/vecstore-out.log last 15 lines:
2|vecstore | FAISS index loading failed Error in faiss::{anonymous}::FileIOReader::FileIOReader(const char*) at index_io.cpp:136: Error: 'f' failed: could not open /data/model_hf for reading: No such file or directory
2|vecstore | Annoy index loading failed
2|vecstore | Starting server. Listening on port 50052.

/root/.pm2/logs/vecdb-out.log last 15 lines:
0|vecdb    | running VecID Worker
0|vecdb    | running VecID Worker

/root/.pm2/logs/peer-manager-out.log last 15 lines:
1|peer_man | peer events subscription done
1|peer_man | Example app listening on port 50053!
1|peer_man | -
0|vecdb  | running VecID Worker
1|peer_manager  | -
0|vecdb         | running VecID Worker
0|vecdb         | running VecID Worker

After I index some data and query, it is working:


1|peer_manager  | -
0|vecdb         | null { total_rows: 1000,
0|vecdb         |   offset: 0,
0|vecdb         |   rows:
0|vecdb         |    [ { id: '0088af8c0a336c5b0d2d0360d68a8ed2',
0|vecdb         |        key: '0088af8c0a336c5b0d2d0360d68a8ed2',
0|vecdb         |        value: [Object],
0|vecdb         |        doc: [Object] },
0|vecdb         |      { id: '008ca03de8f169b8d5a8a029286ba969',
0|vecdb         |        key: '008ca03de8f169b8d5a8a029286ba969',
0|vecdb         |        value: [Object],
0|vecdb         |        doc: [Object] },
0|vecdb         |      { id: '00ab52ffbe5ca2e5d4c352dc5927aeb1',
0|vecdb         |        key: '00ab52ffbe5ca2e5d4c352dc5927aeb1',
0|vecdb         |        value: [Object],
0|vecdb         |        doc: [Object] },

and then I remove the container, and start again

PM2        | 2019-12-30T13:34:46: PM2 log: PM2 home             : /root/.pm2
PM2        | 2019-12-30T13:34:46: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
PM2        | 2019-12-30T13:34:46: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
PM2        | 2019-12-30T13:34:46: PM2 log: BUS socket file      : /root/.pm2/pub.sock
PM2        | 2019-12-30T13:34:46: PM2 log: Application log path : /root/.pm2/logs
PM2        | 2019-12-30T13:34:46: PM2 log: Process dump file    : /root/.pm2/dump.pm2
PM2        | 2019-12-30T13:34:46: PM2 log: Concurrent actions   : 2
PM2        | 2019-12-30T13:34:46: PM2 log: SIGTERM timeout      : 1600
PM2        | 2019-12-30T13:34:46: PM2 log: ===============================================================================
PM2        | 2019-12-30T13:34:46: PM2 log: App [vecdb:0] starting in -fork mode-
PM2        | 2019-12-30T13:34:46: PM2 log: App [vecdb:0] online
PM2        | 2019-12-30T13:34:47: PM2 log: App [peer_manager:1] starting in -fork mode-
PM2        | 2019-12-30T13:34:47: PM2 log: App [peer_manager:1] online
PM2        | 2019-12-30T13:34:47: PM2 log: App [vecstore:2] starting in -fork mode-
PM2        | 2019-12-30T13:34:47: PM2 log: App [vecstore:2] online

/root/.pm2/logs/vecdb-error.log last 15 lines:
/root/.pm2/logs/vecstore-error.log last 15 lines:
/root/.pm2/logs/peer-manager-out.log last 15 lines:
1|peer_man | peer events subscription done
1|peer_man | Example app listening on port 50053!
1|peer_man | OpenError: IO error: lock /data/default_swarmdb/LOCK: Resource temporarily unavailable
1|peer_man |     at /AquilaDB/src/node_modules/levelup/lib/levelup.js:87:23
1|peer_man |     at /AquilaDB/src/node_modules/abstract-leveldown/abstract-leveldown.js:30:14
1|peer_man |     at /AquilaDB/src/node_modules/levelup/node_modules/deferred-leveldown/deferred-leveldown.js:20:21
1|peer_man |     at /AquilaDB/src/node_modules/abstract-leveldown/abstract-leveldown.js:30:14

/root/.pm2/logs/vecstore-out.log last 15 lines:
2|vecstore | FAISS index loading success
2|vecstore | Annoy index loading success
2|vecstore | Starting server. Listening on port 50052.

/root/.pm2/logs/vecdb-out.log last 15 lines:
0|vecdb    | running VecID Worker
0|vecdb    | running VecID Worker
0|vecdb    | running VecID Worker
0|vecdb    | running VecID Worker
0|vecdb    | Error while getting KNND.
0|vecdb    | running VecID Worker
0|vecdb    | running VecID Worker

/root/.pm2/logs/peer-manager-error.log last 15 lines:
1|peer_man | TypeError: Cannot read property 'rows' of undefined
1|peer_man |     at /AquilaDB/src/p2p/routing_table/index.js:157:34
1|peer_man | TypeError: Cannot read property 'rows' of undefined
1|peer_man |     at /AquilaDB/src/p2p/routing_table/index.js:157:34
1|peer_man | TypeError: Cannot read property 'rows' of undefined
1|peer_man |     at /AquilaDB/src/p2p/routing_table/index.js:157:34
1|peer_man | TypeError: Cannot read property 'rows' of undefined
1|peer_man |     at /AquilaDB/src/p2p/routing_table/index.js:157:34

0|vecdb  | running VecID Worker

However I bind a volume, when I query something, I cannot get a response. If I remove model_hf file, everything is running.

freakeinstein commented

@sopaoglu Whoops.. that’s a bug. Fixed. Thanks for your feedback.

Fix is available in develop branch and for docker image delete the old one and fetch bleeding labelled image

freakeinstein commented

see what's changed in source to fix: https://github.com/a-mma/AquilaDB/pull/48

freakeinstein commented

Changes made available to master branch and latest label for docker image.