Initial db migration failure causes trotto app to return 500 internal server error
kentoku24 opened this issue · 5 comments
The symptom and steps to reproduce:
- Launch trotto instance.
- Launch postgres instance.
- Trotto instance keeps returning 500 internal server error.
Root cause:
When trotto app and postgres instance are written in one docker-compose file,
both are launched almost simultaneously.
Solution:
a: add health check to trotto instance, so it will eventually restarted, and DB migration after restart should success.
b: add retry capability to the initial database migration process.
c: inform users to make sure to launch database prior to trotto instance.
Thank you for submitting this, @kentoku24! I think a lazy retry could work well. Did you try any of these solutions in your own branch?
Im seeing the same issue. @kentoku24 , were you able to get it working?
After trying one of the suggested fix im still getting the same error. Im not sure whats going on. Below is my repro case.
this is my app.yml:
sessions_secret: QinOf8Z5W/oiTS5uJ/=
postgres:
url: "postgresql://example:example@db/golinks"
admins:
Docker-compose:
version: "3.5"
services:
go-links:
image: trotto:1.0
depends_on:
db:
condition: service_healthy
ports:
- "9095:8000"
networks:
- traefik
- internal
volumes:
- /etc/localtime:/etc/localtime:ro
- ${DATA_DIR}/go-links:/data
restart: always
db:
image: postgres
env_file:
- .env
environment:
POSTGRES_DB: golinks
POSTGRES_USER: example
POSTGRES_PASSWORD: example
networks:
- internal
healthcheck:
test: [ "CMD", "pg_isready", "-q", "-d", "golinks", "-U", "example" ]
timeout: 45s
interval: 10s
retries: 10
volumes:
- /etc/localtime:/etc/localtime:ro
- ${DATA_DIR}/go-links/db:/var/lib/postgresql/data/
restart: always
networks:
traefik:
external: true
internal:
internal: true
Logs During Start UP
go-links_db :
today at 4:08:12 PM
today at 4:08:12 PM PostgreSQL Database directory appears to contain a database; Skipping initialization
today at 4:08:12 PM
today at 4:08:12 PM 2021-05-14 16:08:12.914 PDT [1] LOG: starting PostgreSQL 13.2 (Debian 13.2-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
today at 4:08:12 PM 2021-05-14 16:08:12.914 PDT [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
today at 4:08:12 PM 2021-05-14 16:08:12.914 PDT [1] LOG: listening on IPv6 address "::", port 5432
today at 4:08:12 PM 2021-05-14 16:08:12.917 PDT [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
today at 4:08:12 PM 2021-05-14 16:08:12.924 PDT [27] LOG: database system was shut down at 2021-05-14 16:08:03 PDT
today at 4:08:12 PM 2021-05-14 16:08:12.929 PDT [1] LOG: database system is ready to accept connections
go-links_web:
today at 4:08:23 PM ../scripts/upgrade_db.sh: 1: cd: can't cd to server/src
today at 4:08:24 PM INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
today at 4:08:24 PM INFO [alembic.runtime.migration] Will assume transactional DDL.
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [15] [INFO] Starting gunicorn 20.0.4
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [15] [INFO] Listening at: http://0.0.0.0:8000 (15)
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [15] [INFO] Using worker: sync
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [18] [INFO] Booting worker with pid: 18
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [19] [INFO] Booting worker with pid: 19
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [20] [INFO] Booting worker with pid: 20
today at 4:08:24 PM [2021-05-14 16:08:24 -0700] [21] [INFO] Booting worker with pid: 21
When i go to the site:
500 Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
Web logs; go-links_web:
today at 4:08:46 PM [2021-05-14 16:08:46,876] ERROR in app: Exception on / [GET]
today at 4:08:46 PM Traceback (most recent call last):
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
today at 4:08:46 PM response = self.full_dispatch_request()
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
today at 4:08:46 PM rv = self.handle_user_exception(e)
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
today at 4:08:46 PM reraise(exc_type, exc_value, tb)
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
today at 4:08:46 PM raise value
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
today at 4:08:46 PM rv = self.dispatch_request()
today at 4:08:46 PM File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
today at 4:08:46 PM return self.view_functionsrule.endpoint
today at 4:08:46 PM File "/usr/src/app/server/src/main.py", line 161, in home
today at 4:08:46 PM 'default_namespace': config.get_default_namespace(current_user.organization),
today at 4:08:46 PM File "/usr/src/app/server/src/shared_helpers/config.py", line 100, in get_default_namespace
today at 4:08:46 PM if session.get('org_default_ns_exp') and session.get('org_default_ns_exp') > datetime.datetime.utcnow():
today at 4:08:46 PM TypeError: can't compare offset-naive and offset-aware datetimes
`
Is there an update on the above issue?
I tried setting up the trotto in my own ec2 instance and i am getting 500 error too
Hi @swapnachagam! We will work on this next week and provide examples around configuring this with docker-compose.
Keep you posted!