Messaging System with RabbitMQ/Celery and Python Application behind Nginx

This deploys a Python application behind Nginx that interacts with RabbitMQ/Celery for email sending and logging functionality. The below architecture diagram visualizes the process: app-architecture

Set up RabbitMQ Locally

  • Install RabbitMQ:
brew install rabbitmq ##for macOS users
  • Start RabbitMQ:
# starts a local RabbitMQ node
brew services start rabbitmq
Screenshot 2024-07-12 at 20 29 55

The default login credential for both the username and password is "guest".

Set up Celery

Celery is an asynchronous task queue/job queue system for Python, designed to handle real-time operations. It can be used to schedule tasks that run in the background, allowing for concurrency and parallel execution of code.

Celery is a Python package so it will be installed with pip. This will be done in a virtual environment.

  • Create and activate a Python virtual environment
python3 -m venv myenv
source myenv/bin/activate
  • Install Celery and other requirements from the requirements.txt file
pip install -r requirements.txt

Setup and Start the Python application

The provided Python application creates a simple web application that does two things:

  1. Sends Emails: It allows you to trigger the sending of an email by visiting a specific URL in your web browser. The email sending itself happens in the background through Celery. Celery is configured to use RabbitMQ as its message broker (where it keeps track of tasks). It creates an email message, connects to a Gmail SMTP server, logs in, and sends the email.

  2. Logs Time: It also has a feature to log the current time to a file when you visit another specific URL.

Email setup

You can use an existing email address or create a new one. Configure 2FA for whichever. In the email settings, search "App Password" and create an App password. Be sure to copy the password somewhere.

Screenshot 2024-07-13 at 08 01 24 Screenshot 2024-07-13 at 08 02 16

A .env.template file has been provided in this repo. Change the name to .env and replace the email and app password env variables.

Set Up Log Directory

The time logs should be logged at /var/log/messaging_system.log. This path has already been set in the app to be created if it doesn't exist but you can configure it manually using the below commands:

sudo touch /var/log/messaging_system.log
sudo chmod a+rw /var/log/messaging_system.log

The code has already been written to do this, but you can run these commands if you want a manual approach.

Install Nginx

Install Nginx on your local and replace the default nginx.conf file with the one provided in this repo. This allows Nginx to route requests from localhost:5000 where the flask app will be running.

Start the Application

  • In your terminal, run:
python app.py  --port 5000
  • In another terminal, run:
celery -A app.celery worker --loglevel=info
  • Access the app at localhost
Screenshot 2024-07-13 at 13 45 51
  • In another tab, load http://localhost:5000/?talktome=true to generate time logs
Screenshot 2024-07-12 at 21 55 28
  • In another tab, load http://localhost:5000/?sendmail=kuberneteslinux@gmail.com
Screenshot 2024-07-12 at 21 56 22
  • Check RabbitMQ to be sure it was queued successfully
Screenshot 2024-07-13 at 13 56 38

After some seconds, it should change to ready Screenshot 2024-07-13 at 13 56 47

If you check the terminal where Celery is running, you should see the below:

Screenshot 2024-07-12 at 21 58 24

This indicates that the email-sending task was received, processed by a specific Celery worker (ForkPoolWorker-8), and completed successfully.

  • Check your email address for the mail sent
Screenshot 2024-07-12 at 22 04 06

Setup Ngrok

  • Install ngrok via Homebrew with the following command:
brew install ngrok/ngrok/ngrok
  • Connect your account Next, connect your ngrok agent to your ngrok account. If you haven't already, sign up for an ngrok account. Copy your ngrok authtoken from your ngrok dashboard. Run the following command in your terminal to install the authtoken and connect the ngrok agent to your account.
ngrok config add-authtoken <TOKEN>
  • Create a static domain

You can create a free one on your ngrok dashboard under the setup and installation tab.

Screenshot 2024-07-12 at 21 52 40

Copy and paste command displayed

ngrok http --domain=<unique-domain.ngrok-free.app> 80
Screenshot 2024-07-12 at 21 47 32
  • Copy and open the URL in your browser
Screenshot 2024-07-12 at 20 54 12
  • Click on the Visit Site button to access your website
Screenshot 2024-07-13 at 13 45 28
  • Access the time logs using the url
Screenshot 2024-07-13 at 13 50 16
  • Send an email
Screenshot 2024-07-13 at 13 50 40
  • You can also access your Ngrok dashboard at localhost:4040
Screenshot 2024-07-13 at 13 52 13