production error: 404 page not found
Closed this issue · 20 comments
Navigating to sub.domain.tld/chatgpt returns 404 page not found.
connection is secure and letsencrypt appears to be working
You have to use "prod"-level config to connect remotely. Have you ever tried to connect using IP address, which is the same as HOST_IP in .env? or, can you access MySQL database using subdomain:3306, with admin account id/password you defined in .env?
Perhaps the error is caused by something that happened when the fastapi app booted. If you have any error logs from your docker container, please share them with me.
I think it may be a traefik issue regarding connecting via local ips. I am pasting docker logs
`Running in: main
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Downloading tokenizer.model: 100%|██████████| 500k/500k [00:00<00:00, 109MB/s]
Downloading (…)cial_tokens_map.json: 100%|██████████| 435/435 [00:00<00:00, 935kB/s]
Downloading (…)okenizer_config.json: 100%|██████████| 727/727 [00:00<00:00, 2.94MB/s]
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data] Unzipping tokenizers/punkt.zip.
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data] /root/nltk_data...
[nltk_data] Unzipping taggers/averaged_perceptron_tagger.zip.
[2023-05-16 15:07:37,549] SQLAlchemy:CRITICAL - Current DB connection of ProdConfig: db/stammer@stammer_admin
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 145, in init
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 452, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1268, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
rec = pool._do_get()
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
with util.safe_reraise():
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in exit
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
return self._create_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
return _ConnectionRecord(self)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 678, in init
self.__connect()
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
with util.safe_reraise():
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in exit
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 637, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 616, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 352, in init
self.connect()
File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 635, in connect
self._get_server_information()
File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 1056, in _get_server_information
packet = self._read_packet()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 729, in _read_packet
packet.raise_for_error()
File "/usr/local/lib/python3.11/site-packages/pymysql/protocol.py", line 221, in raise_for_error
err.raise_mysql_exception(self._data)
File "/usr/local/lib/python3.11/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.OperationalError: (1130, "172.16.0.13' is not allowed to connect to this MySQL server")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/uvicorn", line 8, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 404, in main
run(
File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 569, in run
server.run()
File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 60, in run
return asyncio.run(self.serve(sockets=sockets))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 67, in serve
config.load()
File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 477, in load
self.loaded_app = import_from_string(self.app)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string
module = importlib.import_module(module_str)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1206, in _gcd_import
File "", line 1178, in _find_and_load
File "", line 1149, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/app/main.py", line 42, in
app = create_app(config=config)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/app/common/app_settings.py", line 27, in create_app
db.start(config=config)
File "/app/app/database/connection.py", line 179, in start
if not database_exists(root_url):
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy_utils/functions/database.py", line 498, in database_exists
return bool(_get_scalar_result(engine, sa.text(text)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy_utils/functions/database.py", line 442, in _get_scalar_result
with engine.connect() as conn:
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3264, in connect
return self._connection_cls(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 147, in init
Connection._handle_dbapi_exception_noconnection(
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2426, in _handle_dbapi_exception_noconnection
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 145, in init
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 452, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1268, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
rec = pool._do_get()
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 168, in _do_get
with util.safe_reraise():
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in exit
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 166, in _do_get
return self._create_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
return _ConnectionRecord(self)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 678, in init
self.__connect()
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
with util.safe_reraise():
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in exit
raise exc_value.with_traceback(exc_tb)
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 637, in connect
return dialect.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 616, in connect
return self.loaded_dbapi.connect(*cargs, **cparams)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 352, in init
self.connect()
File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 635, in connect
self._get_server_information()
File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 1056, in _get_server_information
packet = self._read_packet()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pymysql/connections.py", line 729, in _read_packet
packet.raise_for_error()
File "/usr/local/lib/python3.11/site-packages/pymysql/protocol.py", line 221, in raise_for_error
err.raise_mysql_exception(self._data)
File "/usr/local/lib/python3.11/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
raise errorclass(errno, errval)
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1130, "172.16.0.13' is not allowed to connect to this MySQL server")
(Background on this error at: https://sqlalche.me/e/20/e3q8)`
That error happened in code 'if not database_exists(root_url):' in connection.py. And pymysql 1130 error means you have no previledge to access the database. But root account should have all previledges to access all database.
Perhaps this happened because fastapi tried to connect mysql, while mysql is still setting up root account for the first time. I think restarting all docker container will solve this problem. You can test connection to mysql using other db management tool such as tableplus. It's good practice to check db working well using these tools. Check both root account and your db account can access your database (in this case, stammer)
I added adminer
adminer: image: adminer restart: always labels: - "traefik.enable=true" - "traefik.http.routers.adminer.rule=Host(adminer.gkb.domain.com)" - "traefik.http.routers.adminer.entrypoints=websecure" - "traefik.http.services.adminer.loadbalancer.server.scheme=http" - "traefik.http.services.adminer.loadbalancer.server.port=8080" - "traefik.http.routers.adminer.tls=true" - "traefik.http.routers.adminer.tls.certresolver=myresolver" - "traefik.http.routers.adminer.tls.domains[0].main=gkb.domain.com" - "traefik.http.routers.adminer.tls.domains[0].sans=adminer.gkb.domain.com" depends_on: - proxy networks: reverse-proxy-public: ipv4_address: 172.16.0.14 # static IP
And get connection error
Host '172.16.0.14' is not allowed to connect to this MySQL server
As stated above "cosogi/chatgpt-webapp-server-api:latest" exits with
1130, "172.16.0.13' is not allowed to connect to this MySQL server
So definitely mysql refusing to accept connection. I have restarted docker several times always same issue
I forgot to add this environment
MYSQL_ROOT_HOST: "172.*.*.*"
By default, the root account host is localhost, that allows only connection from localhost.
So can you try again adding MYSQL_ROOT_HOST: "172.*.*.*" in db.environment section in yaml?
maybe you need to docker-compose down -v and docker-compose up --build
Connection now allowed for other user but root is denied and chatgpt-webapp-server-api exits with error .
raise errorclass(errno, errval) sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, "Access denied for user 'root'@'172.16.0.13' (using password: YES)")
That's weird. Can you try it again with MYSQL_ROOT_HOST: "%"? Was it possible to connect as a non-root user?
Also, if you modify the yaml file, you need to run down -v because all settings related to the mysql account must be deleted.
New error
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'db' ([Errno 111] Connection refused)")
want me to setup a digital ocean server for production testing?
Can't connect to Mysql server on 'db'` could be because you're trying to run the api server outside of docker, or it could be because the mysql server won't open. If it's the latter, it's possible that something is wrong with your mysql settings and the container is not booting. I'll try installing it on my own subdomain and let you know.
I can connect to the server with adminer container. It is only the root user that is having problem. root can't connect from Adminer either. maybe update app to use non-root account?
I just completed an installation on a test subdomain, and everything worked on one shot. The mysql root account was also connected remotely just fine. It could be that the .env file is misconfigured, or that the old data is not cleared when configuring the docker container. Can you try git cloning to a different folder with a new commit and creating a new environment?
here's my .env config
API_ENV="prod"
DEFAULT_LLM_MODEL="gpt_3_5_turbo"
DOCKER_MODE=True
MYSQL_DATABASE="traffic"
MYSQL_TEST_DATABASE="testing_db"
MYSQL_ROOT_PASSWORD="somepassword="
MYSQL_USER="traffic_admin"
MYSQL_PASSWORD="somepassword"
REDIS_DATABASE=0
REDIS_PASSWORD="someredispassword"
HOST_IP="123.123.123.123"
HOST_MAIN="test.my.domain"
HOST_SUB="test.my.domain"
// skip the rest
The reason I need the root account in my app is so that I can use it to grant privileges to non-root accounts and create tables in the database. Without the root account, non-root accounts will not be able to do this because they do not have privileges.
If you only want to use the non-root account, you can delete all the parts of the start function in the SQLAlchemy class that use the root account. You can do this by deleting the lines containing root_engine and root_url.
Okay I will rebuild that server right away. What Linux distribution?
Ubuntu-22.04
leave commented? # - ./my.cnf:/etc/mysql/conf.d/my.cnf
i am not using it just comment it
On a new server, all dockers are running fine (I think this line port:8000, may have been the problem). However domain.tld/chat. is not reachable and letsencrypt folder is empty
Can you connect '/docs' then? traefik will route external 80/443 to internal 8000 port. Use another browser or use different email in .env
I think that's TLS certificate issue
I changed email and browser - same connection refused error. I will leave it for tomorrow. Side note, we should rate limit requests to letsencrypt. Off to bed now, it's 1:am here
Latest version release installed without issues