1.56 Docker Startup

Question

1.56 Docker Startup

vbhayden opened this issue 3 years ago · 14 comments

Morning team,

Having a few issues with the Docker setup for the 1.56 update from yesterday, namely an indefined id property somewhere during startup, but there doesn't seem to be a trace back to whatever's causing it.

This was just running directly from the docker-compose.yml setup, so no funny business on my end that I know of. It does still start, but the server just responds with Cannot GET / for any resource requested.

Excited to try out 1.5 and hoping this is something small, as undefined stuff can sometimes be.

┌─────┬──────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id  │ name         │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├─────┼──────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0   │ CaSS Test    │ default     │ 1.5.6   │ cluster │ 43       │ 0s     │ 0    │ online    │ 0%       │ 37.0mb   │ root     │ disabled │
└─────┴──────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘

> cass@1.5.6 logs
> pm2 logs

[TAILING] Tailing last 15 lines for [all] processes (change the value with --lines option)
/root/.pm2/pm2.log last 15 lines:
PM2        | 2022-04-14T15:56:24: PM2 log: PM2 version          : 5.2.0
PM2        | 2022-04-14T15:56:24: PM2 log: Node.js version      : 16.14.2
PM2        | 2022-04-14T15:56:24: PM2 log: Current arch         : x64
PM2        | 2022-04-14T15:56:24: PM2 log: PM2 home             : /root/.pm2
PM2        | 2022-04-14T15:56:24: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
PM2        | 2022-04-14T15:56:24: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
PM2        | 2022-04-14T15:56:24: PM2 log: BUS socket file      : /root/.pm2/pub.sock
PM2        | 2022-04-14T15:56:24: PM2 log: Application log path : /root/.pm2/logs
PM2        | 2022-04-14T15:56:24: PM2 log: Worker Interval      : 30000
PM2        | 2022-04-14T15:56:24: PM2 log: Process dump file    : /root/.pm2/dump.pm2
PM2        | 2022-04-14T15:56:24: PM2 log: Concurrent actions   : 2
PM2        | 2022-04-14T15:56:24: PM2 log: SIGTERM timeout      : 1600
PM2        | 2022-04-14T15:56:24: PM2 log: ===============================================================================
PM2        | 2022-04-14T15:56:24: PM2 log: App [CaSS Test:0] starting in -cluster mode-
PM2        | 2022-04-14T15:56:24: PM2 log: App [CaSS Test:0] online

/root/.pm2/logs/CaSS-Test-out-0.log last 15 lines:
/root/.pm2/logs/CaSS-Test-error-0.log last 15 lines:
0|CaSS Test  | true
0|CaSS Test  | Binding endpoint: /api/util/reindex
0|CaSS Test  | Binding endpoint: /api/util/purge
0|CaSS Test  | Binding endpoint: /api/ping
0|CaSS Test  | Binding endpoint: /api/data/*
0|CaSS Test  | Binding endpoint: /api/sky/repo/search
0|CaSS Test  | Binding endpoint: /api/sky/repo/multiGet
0|CaSS Test  | Binding endpoint: /api/sky/repo/multiPut
0|CaSS Test  | Binding endpoint: /api/sky/admin
0|CaSS Test  | Binding endpoint: /api/sky/id/salts
0|CaSS Test  | Binding endpoint: /api/sky/id/create
0|CaSS Test  | Binding endpoint: /api/sky/id/commit
0|CaSS Test  | Binding endpoint: /api/sky/id/login
0|CaSS Test  | Binding endpoint: /api/asn/*
0|CaSS Test  | Binding endpoint: /api/ims/case/v1p0/CFDocuments
0|CaSS Test  | Binding endpoint: /api/ims/case/v1p0/CFAssociations
0|CaSS Test  | Binding endpoint: /api/ims/case/v1p0/CFAssociationGroupings
0|CaSS Test  | Binding endpoint: /api/ims/case/v1p0/CFConcepts
0|CaSS Test  | Binding endpoint: /api/ims/case/v1p0/CFItems
0|CaSS Test  | Binding endpoint: /api/ims/case/v1p0/CFItemAssociations
0|CaSS Test  | Binding endpoint: /api/ims/case/v1p0/CFItemTypes
0|CaSS Test  | Binding endpoint: /api/ims/case/v1p0/CFLicenses
0|CaSS Test  | Binding endpoint: /api/ims/case/v1p0/CFPackages/*
0|CaSS Test  | Binding endpoint: /api/ims/case/v1p0/CFRubrics
0|CaSS Test  | Binding endpoint: /api/ims/case/v1p0/CFSubjects
0|CaSS Test  | Binding endpoint: /api/ims/case/harvest
0|CaSS Test  | Binding endpoint: /api/ims/case/getDocs
0|CaSS Test  | Binding endpoint: /api/ceasn/*
0|CaSS Test  | Binding endpoint: /api/ctdlasn/*
0|CaSS Test  | Binding endpoint: /api/ctdlasn
0|CaSS Test  | Binding endpoint: /api/jsonld
0|CaSS Test  | Binding endpoint: /api/badge/pk
0|CaSS Test  | Binding endpoint: /api/badge/profile/*
0|CaSS Test  | Binding endpoint: /api/badge/cryptographicKey/*
0|CaSS Test  | Binding endpoint: /api/badge/class/*
0|CaSS Test  | Binding endpoint: /api/badge/assertion/*
0|CaSS Test  | Binding endpoint: /api/xapi/endpoint
0|CaSS Test  | Binding endpoint: /api/xapi/pk
0|CaSS Test  | Binding endpoint: /api/xapi/statement
0|CaSS Test  | Binding endpoint: /api/xapi/statements
0|CaSS Test  | Binding endpoint: /api/xapi/tick
0|CaSS Test  | Binding endpoint: /api/profile/latest
0|CaSS Test  | TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
0|CaSS Test  |     at new NodeError (node:internal/errors:371:5)
0|CaSS Test  |     at validateString (node:internal/validators:120:11)
0|CaSS Test  |     at Module.require (node:internal/modules/cjs/loader:998:3)
0|CaSS Test  |     at require (node:internal/modules/cjs/helpers:102:18)
0|CaSS Test  |     at workerThread (/app/node_modules/web-worker/cjs/node.js:214:9)
0|CaSS Test  |     at Object.<anonymous> (/app/node_modules/web-worker/cjs/node.js:79:56)
0|CaSS Test  |     at Module._compile (node:internal/modules/cjs/loader:1103:14)
0|CaSS Test  |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
0|CaSS Test  |     at Module.load (node:internal/modules/cjs/loader:981:32)
0|CaSS Test  |     at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
0|CaSS Test  |   code: 'ERR_INVALID_ARG_TYPE'
0|CaSS Test  | }
0|CaSS Test  | TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
0|CaSS Test  |     at new NodeError (node:internal/errors:371:5)
0|CaSS Test  |     at validateString (node:internal/validators:120:11)
0|CaSS Test  |     at Module.require (node:internal/modules/cjs/loader:998:3)
0|CaSS Test  |     at require (node:internal/modules/cjs/helpers:102:18)
0|CaSS Test  |     at workerThread (/app/node_modules/web-worker/cjs/node.js:214:9)
0|CaSS Test  |     at Object.<anonymous> (/app/node_modules/web-worker/cjs/node.js:79:56)
0|CaSS Test  |     at Module._compile (node:internal/modules/cjs/loader:1103:14)
0|CaSS Test  |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
0|CaSS Test  |     at Module.load (node:internal/modules/cjs/loader:981:32)
0|CaSS Test  |     at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
0|CaSS Test  |   code: 'ERR_INVALID_ARG_TYPE'
0|CaSS Test  | }
0|CaSS Test  | TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
0|CaSS Test  |     at new NodeError (node:internal/errors:371:5)
0|CaSS Test  |     at validateString (node:internal/validators:120:11)
0|CaSS Test  |     at Module.require (node:internal/modules/cjs/loader:998:3)
0|CaSS Test  |     at require (node:internal/modules/cjs/helpers:102:18)
0|CaSS Test  |     at workerThread (/app/node_modules/web-worker/cjs/node.js:214:9)
0|CaSS Test  |     at Object.<anonymous> (/app/node_modules/web-worker/cjs/node.js:79:56)
0|CaSS Test  |     at Module._compile (node:internal/modules/cjs/loader:1103:14)
0|CaSS Test  |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
0|CaSS Test  |     at Module.load (node:internal/modules/cjs/loader:981:32)
0|CaSS Test  |     at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
0|CaSS Test  |   code: 'ERR_INVALID_ARG_TYPE'
0|CaSS Test  | }
0|CaSS Test  | TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
0|CaSS Test  |     at new NodeError (node:internal/errors:371:5)
0|CaSS Test  |     at validateString (node:internal/validators:120:11)
0|CaSS Test  |     at Module.require (node:internal/modules/cjs/loader:998:3)
0|CaSS Test  |     at require (node:internal/modules/cjs/helpers:102:18)
0|CaSS Test  |     at workerThread (/app/node_modules/web-worker/cjs/node.js:214:9)
0|CaSS Test  |     at Object.<anonymous> (/app/node_modules/web-worker/cjs/node.js:79:56)
0|CaSS Test  |     at Module._compile (node:internal/modules/cjs/loader:1103:14)
0|CaSS Test  |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
0|CaSS Test  |     at Module.load (node:internal/modules/cjs/loader:981:32)
0|CaSS Test  |     at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
0|CaSS Test  |   code: 'ERR_INVALID_ARG_TYPE'
0|CaSS Test  | }
0|CaSS Test  | Current Elasticsearch Version: 7.16.2
0|CaSS Test  | Current Minimum Index Compatibility Version: 6.0.0-beta1
0|CaSS Test  | Waiting for cluster health...
0|CaSS Test  | red
0|CaSS Test  | Current Elasticsearch Version: 7.16.2
0|CaSS Test  | Current Minimum Index Compatibility Version: 6.0.0-beta1
0|CaSS Test  | CaSS listening at http://localhost:80
0|CaSS Test  | CaSS thinks it its endpoint is at http://cass/api/
0|CaSS Test  | Startup time 18401 ms

Answer 1 · 2022-04-14T17:35:38.000Z

Did you expose port 80 via the docker-compose or run parameters to your local host?

-p80:80 in the docker run statement

Answer 2 · 2022-04-14T17:38:00.000Z

Yeah it's all through the compose stuff, going into the container and curling localhost gets the same thing

ubuntu@ip-172-31-78-86:~/tla-cass$ docker exec -it cass bash

root@4132dc14a7cc:/app# curl localhost:80
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /</pre>
</body>
</html>

Answer 3 · 2022-04-14T17:40:24.000Z

What compose? Are you using a git repo or the docker.io builds?

Answer 4 · 2022-04-14T17:45:36.000Z

Have tried a few, but the one from the project root will also do it: https://github.com/cassproject/CASS/blob/master/docker-compose.yml

docker-compose up -d --build

Answer 5 · 2022-04-14T17:45:52.000Z

CASS/Dockerfile is a development dockerfile.
CASS/docker-compose.yml is the development docker-compose for that container.

CASS/docker/standalone/Dockerfile is the docker.io release dockerfile.
CASS/docker/standalone/docker-compose.yml (recently updated with latest version) is the docker.io compatible release docker-compose.yml

If you're assembling a system of systems, I highly recommend using the docker.io stuff and not the development stuff.
Otherwise, knowing what your intent is would probably help.

Answer 6 · 2022-04-14T17:47:09.000Z

Also, I think I know why you're getting that:

When you clone cass, you need to include submodules --recurse-submodules or the website won't show up.

Answer 7 · 2022-04-14T17:48:19.000Z

Oh that'd do it, yeah the image seems to work fine so I'll just use that.

Didn't notice the submodules bit, thanks Fritz.

Answer 8 · 2022-04-15T15:26:32.000Z

Looks like the problem has been kind of awkward to track down, and does seem to happen even with the image on a clean ubuntu 20 install.

To clarify, the use case / intent is just to get an instance of the 1.5.X CaSS running at all with Docker.

To reproduce:

Ubuntu 20
Install Docker + Docker Compose and add ubuntu or w/e to run without sudo
sudo sysctl -w vm.max_map_count=262144 for Elasticsearch
docker-compose up -d --build

...

0|CaSS  | Binding endpoint: /api/xapi/tick
0|CaSS  | Binding endpoint: /api/profile/latest
0|CaSS  | TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
0|CaSS  |     at new NodeError (node:internal/errors:371:5)
0|CaSS  |     at validateString (node:internal/validators:120:11)
0|CaSS  |     at Module.require (node:internal/modules/cjs/loader:998:3)
0|CaSS  |     at require (node:internal/modules/cjs/helpers:102:18)
0|CaSS  |     at workerThread (/usr/share/elasticsearch/CASS/node_modules/web-worker/cjs/node.js:214:9)
0|CaSS  |     at Object.<anonymous> (/usr/share/elasticsearch/CASS/node_modules/web-worker/cjs/node.js:79:56)
0|CaSS  |     at Module._compile (node:internal/modules/cjs/loader:1103:14)
0|CaSS  |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
0|CaSS  |     at Module.load (node:internal/modules/cjs/loader:981:32)
0|CaSS  |     at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
0|CaSS  |   code: 'ERR_INVALID_ARG_TYPE'
0|CaSS  | }
0|CaSS  | TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
0|CaSS  |     at new NodeError (node:internal/errors:371:5)
0|CaSS  |     at validateString (node:internal/validators:120:11)
0|CaSS  |     at Module.require (node:internal/modules/cjs/loader:998:3)
0|CaSS  |     at require (node:internal/modules/cjs/helpers:102:18)
0|CaSS  |     at workerThread (/usr/share/elasticsearch/CASS/node_modules/web-worker/cjs/node.js:214:9)
0|CaSS  |     at Object.<anonymous> (/usr/share/elasticsearch/CASS/node_modules/web-worker/cjs/node.js:79:56)
0|CaSS  |     at Module._compile (node:internal/modules/cjs/loader:1103:14)
0|CaSS  |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
0|CaSS  |     at Module.load (node:internal/modules/cjs/loader:981:32)
0|CaSS  |     at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
0|CaSS  |   code: 'ERR_INVALID_ARG_TYPE'
0|CaSS  | }
0|CaSS  | TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
0|CaSS  |     at new NodeError (node:internal/errors:371:5)
0|CaSS  |     at validateString (node:internal/validators:120:11)
0|CaSS  |     at Module.require (node:internal/modules/cjs/loader:998:3)
0|CaSS  |     at require (node:internal/modules/cjs/helpers:102:18)
0|CaSS  |     at workerThread (/usr/share/elasticsearch/CASS/node_modules/web-worker/cjs/node.js:214:9)
0|CaSS  |     at Object.<anonymous> (/usr/share/elasticsearch/CASS/node_modules/web-worker/cjs/node.js:79:56)
0|CaSS  |     at Module._compile (node:internal/modules/cjs/loader:1103:14)
0|CaSS  |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
0|CaSS  |     at Module.load (node:internal/modules/cjs/loader:981:32)
0|CaSS  |     at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
0|CaSS  |   code: 'ERR_INVALID_ARG_TYPE'
0|CaSS  | }
0|CaSS  | TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
0|CaSS  |     at new NodeError (node:internal/errors:371:5)
0|CaSS  |     at validateString (node:internal/validators:120:11)
0|CaSS  |     at Module.require (node:internal/modules/cjs/loader:998:3)
0|CaSS  |     at require (node:internal/modules/cjs/helpers:102:18)
0|CaSS  |     at workerThread (/usr/share/elasticsearch/CASS/node_modules/web-worker/cjs/node.js:214:9)
0|CaSS  |     at Object.<anonymous> (/usr/share/elasticsearch/CASS/node_modules/web-worker/cjs/node.js:79:56)
0|CaSS  |     at Module._compile (node:internal/modules/cjs/loader:1103:14)
0|CaSS  |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
0|CaSS  |     at Module.load (node:internal/modules/cjs/loader:981:32)
0|CaSS  |     at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
0|CaSS  |   code: 'ERR_INVALID_ARG_TYPE'
0|CaSS  | }

For the compose file, the only changes were pointing it to an image and adjusting the ES memory from 6g to 1g for local deployment testing.

version: '3'
services:
  cass:
    image: cassproject/cass:1.5.6
    container_name: cass
    networks:
      - esnet
    ports:
      - 80:80
    environment:
      - CASS_LOOPBACK=http://cass/api/
      - ELASTICSEARCH_ENDPOINT=http://elasticsearch:9200
      - PORT=80
      # - CASS_OIDC_ENABLED=true # turns on open id login
      # - CASS_OIDC_ISSUER_BASE_URL=http://keycloak:8080/auth/realms/master/ # point at your keycloak realm
      # - CASS_OIDC_CLIENT_ID=cass # name of your keycloak client
      # - CASS_OIDC_SECRET=M2nn59wDxkwUZ9SVEVAJADYnUeKP78vU # secret from your keycloak client
      # - CASS_OIDC_BASE_URL=http://localhost/ # where to redirect after login
      # - CASS_BANNER_MESSAGE=CUI # message to display in a banner
      # - CASS_BANNER_TEXT_COLOR=red # valid css color value for text color of banner message
      # - CASS_BANNER_BACKGROUND_COLOR=yellow # valid css color value for background color of banner
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - etc:/app/etc
    depends_on:
      - elasticsearch
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
    container_name: elasticsearch
    networks:
      - esnet
    environment:
      - cluster.name=docker-cluster
      - node.name=node-1
      - cluster.max_shards_per_node=3000
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ports:
      - 9200:9200
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data

networks:
  esnet: 

volumes:
  etc:
    driver: local
  esdata1:
    driver: local
  esdata2:
    driver: local

Answer 9 · 2022-04-15T17:29:58.000Z

Hey, those errors are normal and have to do with spinning up some webworkers. They don't affect operation. You should see something like

0|CaSS  | CaSS listening at https://localhost:80
0|CaSS  | CaSS thinks it its endpoint is at https://localhost/api/
0|CaSS  | Startup time 10992 ms
0|CaSS  | Total Heap Size 1.93GB

once it is running. That tells you it is now listening to connections.

Answer 10 · 2022-04-15T18:13:47.000Z

Yeah that's what it looked like, but following the current compose setup results in it just sort of hanging for several minutes (7+) without much indication that it'll resolve etc.

I've gotten it to run a few times after changing volume settings / permissions, so I was curious if there are additional environment assumptions that you guys use when deploying this way?

Answer 11 · 2022-04-15T18:24:58.000Z

Oh found it, so the DockerHub image uses the standalone config (which also has its own Elasticsearch instance?) and assigns:

RUN /bin/sh -c echo '-Xms4g' >> config/jvm.options # buildkit

during the build, which quietly causes a memory issue for smaller machines / vm's.

Adding a similar line to the cass environment section in compose seems to lets things start up:

environment:
  - "ES_JAVA_OPTS=-Xms1g - Xmx1g"

Is there an image that doesn't include its own elasticsearch overhead, or would that just be the node-1.5.6 one?

Answer 12 · 2022-04-15T18:35:19.000Z

That'd be the node-1.5.6 one, you'd have to configure your own elasticsearch index and set the ELASTICSEARCH_ENDPOINT environment variable to hit it.

Answer 13 · 2022-04-15T18:42:40.000Z

Ok, easy enough - thanks fritz!

Answer 14 · 2022-04-15T18:43:50.000Z

Note that the elasticsearch version should be in the 7.x.x version number, 8.x.x isn't supported yet.