bookish-enigma: A Python repository from rossi1

Backend Requirements

Docker.
Docker Compose.
Poetry for Python package and environment management.

Frontend Requirements

Node.js (with npm).

Backend local development

Start the stack with Docker Compose:

docker-compose up -d

Now you can open your browser and interact with these URLs:

Frontend, built with Docker, with routes handled based on the path: http://localhost

Backend, JSON based web API based on OpenAPI: http://localhost/api/

Automatic interactive documentation with Swagger UI (from the OpenAPI backend): http://localhost:8000/docs

Alternative automatic documentation with ReDoc (from the OpenAPI backend): http://localhost:8000/redoc

PGAdmin, PostgreSQL web administration: http://localhost:5050

Note: The first time you start your stack, it might take a minute for it to be ready. While the backend waits for the database to be ready and configures everything. You can check the logs to monitor it.

To check the logs, run:

docker-compose logs

To check the logs of a specific service, add the name of the service, e.g.:

docker-compose logs backend

If your Docker is not running in localhost (the URLs above wouldn't work) check the sections below on Development with Docker Toolbox and Development with a custom IP.

Backend local development, additional details

General workflow

By default, the dependencies are managed with Poetry, go there and install it.

From ./backend/app/ you can install all the dependencies with:

$ poetry install

Then you can start a shell session with the new environment with:

$ poetry shell

Next, open your editor at ./backend/app/ (instead of the project root: ./), so that you see an ./app/ directory with your code inside. That way, your editor will be able to find all the imports, etc. Make sure your editor uses the environment you just created with Poetry.

Modify or add SQLAlchemy models in ./backend/app/app/models/, Pydantic schemas in ./backend/app/app/schemas/, API endpoints in ./backend/app/app/api/, CRUD (Create, Read, Update, Delete) utils in ./backend/app/app/crud/.

Migrations

As during local development your app directory is mounted as a volume inside the container, you can also run the migrations with alembic commands inside the container and the migration code will be in your app directory (instead of being only inside the container). So you can add it to your git repository.

Make sure you create a "revision" of your models and that you "upgrade" your database with that revision every time you change them. As this is what will update the tables in your database. Otherwise, your application will have errors.

Start an interactive session in the backend container:

$ docker-compose exec backend bash

If you created a new model in ./backend/app/app/models/, make sure to import it in ./backend/app/app/db/base.py, that Python module (base.py) that imports all the models will be used by Alembic.
After changing a model (for example, adding a column), inside the container, create a revision, e.g.:

$ alembic revision --autogenerate -m "Add column tag to Product model"

Commit to the git repository the files generated in the alembic directory.
After creating the revision, run the migration in the database (this is what will actually change the database):

$ alembic upgrade head

If you don't want to use migrations at all, uncomment the line in the file at ./backend/app/app/db/init_db.py with:

Base.metadata.create_all(bind=engine)

and comment the line in the file prestart.sh that contains:

$ alembic upgrade head

Persisting Docker named volumes

You need to make sure that each service (Docker container) that uses a volume is always deployed to the same Docker "node" in the cluster, that way it will preserve the data. Otherwise, it could be deployed to a different node each time, and each time the volume would be created in that new node before starting the service. As a result, it would look like your service was starting from scratch every time, losing all the previous data.

That's specially important for a service running a database. But the same problem would apply if you were saving files in your main backend service (for example, if those files were uploaded by your users, or if they were created by your system).

To solve that, you can put constraints in the services that use one or more data volumes (like databases) to make them be deployed to a Docker node with a specific label. And of course, you need to have that label assigned to one (only one) of your nodes.

Adding services with volumes

For each service that uses a volume (databases, services with uploaded files, etc) you should have a label constraint in your docker-compose.yml file.

To make sure that your labels are unique per volume per stack (for example, that they are not the same for prod and stag) you should prefix them with the name of your stack and then use the same name of the volume.

Then you need to have those constraints in your docker-compose.yml file for the services that need to be fixed with each volume.

To be able to use different environments, like prod and stag, you should pass the name of the stack as an environment variable. Like:

STACK_NAME=stag-test-com sh ./scripts/deploy.sh

To use and expand that environment variable inside the docker-compose.yml files you can add the constraints to the services like:

version: '3'
services:
  db:
    volumes:
      - 'app-db-data:/var/lib/postgresql/data/pgdata'
    deploy:
      placement:
        constraints:
          - node.labels.${STACK_NAME?Variable not set}.app-db-data == true

note the ${STACK_NAME?Variable not set}. In the script ./scripts/deploy.sh, the docker-compose.yml would be converted, and saved to a file docker-stack.yml containing:

version: '3'
services:
  db:
    volumes:
      - 'app-db-data:/var/lib/postgresql/data/pgdata'
    deploy:
      placement:
        constraints:
          - node.labels.test-com.app-db-data == true

Note: The ${STACK_NAME?Variable not set} means "use the environment variable STACK_NAME, but if it is not set, show an error Variable not set".

If you add more volumes to your stack, you need to make sure you add the corresponding constraints to the services that use that named volume.

Then you have to create those labels in some nodes in your Docker Swarm mode cluster. You can use docker-auto-labels to do it automatically.

`docker-auto-labels`

You can use docker-auto-labels to automatically read the placement constraint labels in your Docker stack (Docker Compose file) and assign them to a random Docker node in your Swarm mode cluster if those labels don't exist yet.

To do that, you can install docker-auto-labels:

pip install docker-auto-labels

And then run it passing your docker-stack.yml file as a parameter:

docker-auto-labels docker-stack.yml

You can run that command every time you deploy, right before deploying, as it doesn't modify anything if the required labels already exist.

(Optionally) adding labels manually

If you don't want to use docker-auto-labels or for any reason you want to manually assign the constraint labels to specific nodes in your Docker Swarm mode cluster, you can do the following:

First, connect via SSH to your Docker Swarm mode cluster.
Then check the available nodes with:

$ docker node ls


// you would see an output like:

ID                            HOSTNAME               STATUS              AVAILABILITY        MANAGER STATUS
nfa3d4df2df34as2fd34230rm *   dog.example.com        Ready               Active              Reachable
2c2sd2342asdfasd42342304e     cat.example.com        Ready               Active              Leader
c4sdf2342asdfasd4234234ii     snake.example.com      Ready               Active              Reachable

then chose a node from the list. For example, dog.example.com.

Add the label to that node. Use as label the name of the stack you are deploying followed by a dot (.) followed by the named volume, and as value, just true, e.g.:

docker node update --label-add test-com.app-db-data=true dog.example.com

Then you need to do the same for each stack version you have. For example, for staging you could do:

docker node update --label-add stag-test-com.app-db-data=true cat.example.com

Docker Compose files and env vars

There is a main docker-compose.yml file with all the configurations that apply to the whole stack, it is used automatically by docker-compose.

These Docker Compose files use the .env file containing configurations to be injected as environment variables in the containers.

They also use some additional configurations taken from environment variables set in the scripts before calling the docker-compose command.

It is all designed to support several "stages", like development, building, testing, and deployment. Also, allowing the deployment to different environments like staging and production (and you can add more environments very easily).

They are designed to have the minimum repetition of code and configurations, so that if you need to change something, you have to change it in the minimum amount of places. That's why files use environment variables that get auto-expanded. That way, if for example, you want to use a different domain, you can call the docker-compose command with a different DOMAIN environment variable instead of having to change the domain in several places inside the Docker Compose files.

Also, if you want to have another deployment environment, say preprod, you just have to change environment variables, but you can keep using the same Docker Compose files.

The .env file

The .env file is the one that contains all your configurations, generated keys and passwords, etc.

Depending on your workflow, you could want to exclude it from Git, for example if your project is public. In that case, you would have to make sure to set up a way for your CI tools to obtain it while building or deploying your project.

One way to do it could be to add each environment variable to your CI/CD system, and updating the docker-compose.yml file to read that specific env var instead of reading the .env file.

URLs

These are the URLs that will be used and generated by the project.

Development URLs

Development URLs, for local development.

Frontend: http://localhost

Backend: http://localhost:8000/api/

Automatic Interactive Docs (Swagger UI): https://localhost:8000/docs

Automatic Alternative Docs (ReDoc): https://localhost:8000/redoc

PGAdmin: http://localhost:5050

rossi1/bookish-enigma