sqlite3.OperationalError: unable to open database file after initial docker-compose up --build

[oleiade]

Hi 👋🏻

I've been trying to play around with test api locally. To do so I've cloned the repo and ran docker-compose up --build -d. I expected the docker-compose ps command to show the test_api service up, and to tell me which port was exposed. However, although the build phase succeeded, the launch phase failed:

…/test-api.k6.io on  master via 🐍 v3.8.5 took 1m29s
➜ docker-compose ps
NAME                COMMAND                  SERVICE             STATUS              PORTS
test_api            "sh -c 'python proje…"   test_api            exited (1)

Checking out the logs indicates that for some reason Django was unable to open the SQLite3 DB file:

…/test-api.k6.io on  master via 🐍 v3.8.5
➜ docker logs test_api
/srv/test-api.k6.io/static_resources
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 207, in get_new_connection
    conn = Database.connect(**conn_params)
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/test-api.k6.io/project/manage.py", line 20, in <module>
    main()
  File "/srv/test-api.k6.io/project/manage.py", line 16, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/makemigrations.py", line 101, in handle
    loader.check_consistent_history(connection)
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/loader.py", line 290, in check_consistent_history
    applied = recorder.applied_migrations()
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/recorder.py", line 77, in applied_migrations
    if self.has_table():
  File "/usr/local/lib/python3.9/site-packages/django/db/migrations/recorder.py", line 55, in has_table
    with self.connection.cursor() as cursor:
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 235, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 207, in get_new_connection
    conn = Database.connect(**conn_params)
django.db.utils.OperationalError: unable to open database file

My setup:

• Windows 11 with WSL2
• Docker Desktop configured to use WSL2 as a backend for containers

Cheers 🐻

[oleiade]

The README says that the loaddata script should be called by the docker-compose build phase. Assuming it might not have been the case, I tried to run it myself on my local machine (ubuntu, not inside docker) and got the following result:

sh devops/loaddata.sh
: not founddata.sh: 2:
: not founddata.sh: 4:
/usr/bin/env: ‘python\r’: No such file or directory

There might be something wrong between the chair and the keyboard...

[oleiade]

I managed to setup my local development environment. There were two dependencies missing for the pip install -r requirements.txt command to work on Ubuntu though:

• sudo apt install libmysqlclient-dev
• pip install wheel

project/manage.py runserver returns the same error as running the loaddata script does, which now leads me this issue is specific to my setup and could be the cause of the docker-compose up related failure. This Stack Overflow Post helped, but I'm not satisfied with modifying the manage.py file. The part of the issue is most likely related some Windows weirdness.

[oleiade]

Continuing my investigation I think I found the root cause of the initial sqlite3.OperationalError: unable to open database file error. The project's api config wants to write the db file in a WORK_DIR location:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(WORK_DIR, 'db.sqlite3'),
    },
}

WIth a bit of debugging, I found out that in my case the path that WORK_DIR resolves to was never created, therefore all the DB-related python manage.py functions failed. If I replace WORK_DIR with BASE_DIR in the previous example, it fixes things. It's been a long long time since I used django, and I don't know if I should expect Django to create transient folders.

In the end my workaround to this issue is to create workdir by hand: mkdir workdir. I imagine this step could also be part of the docker-compose command bootstraping the test_api container; but I'm not sure it's supposed to. Also windows specific issue I had to dos2unix loaddata.sh.