Tutorial: Self-Hosted Node.js App Deployment on AWS EC2 or On-Premise and other Cloud Servers provider with CI/CD

Section

I. Create and EC2 on AWS

1. Type "EC2" and Select it.

3. Click on Launch instance

4. Type a name for your server

5. Choose Ubutnu and Choose Amazon Machine Image (AMI)

6. Choose Instance type

7. Searching t3.medium for server with 2vCPU and 4GB Ram.

8. Click on t3.medium. If you want more RAM, you can choose t3.large.

9. Key pair (login). Click create new key pair. This will use for login to server and using in CI/CD

10. Type Key pair name. Example "self-hosted"

11. Click on Create key pair in this popup

12. Check Allow HTTPS traffic from the internet (For SSL later)

13. Check Allow HTTP traffic from the internet (For port 80)

14. Choose storage size for SSD in "EBS Volumes". Example 32GB

15. Change Volume type (SSD Type).

16. Click on General purpose SSD (gp3)…

17. Click on Advanced details. To setup terminate protection, Cloudwatch Detailed Monitoring and UserData (For preinstall some of software we need)

18. Enable Termination protection

19. Enable Detailed CloudWatch monitoring

20. Scrolldown to User data - optional

Add this content below:

#!/bin/bash
sudo apt update -y

# Install Docker using Snap
sudo snap install docker

# Grant permission for run Docker without root user
sudo chmod 666 /var/run/docker.sock

# Install Nginx proxy manager
sudo docker run -d \
  --name app \
  --restart unless-stopped \
  -p 80:80 \
  -p 443:443 \
  -p 81:81 \
  -v ./data:/data \
  -v ./letsencrypt:/etc/letsencrypt \
  jc21/nginx-proxy-manager:latest

23. Click on Launch instance

Attach Static IP (May be in other cloud platform in Vietnam, Static IP will be available by default)

1. Type "Elastic IPs" in Search box

2. Click on Elastic IPs

3. Click on Allocate Elastic IP address

4. Click on Allocate

5. Click on Associate this Elastic IP address

6. Make sure Resource type is Instance. Then choose instance in Instance Select

7. Choose your suitable instance

8. Click on Associate

9. On Allocated IPv4 address. Click copy IPv4 address to use for setup domain name, CI/CD,...

GitHub: Step-by-Step Instructions for Setting Up a Workflow

Setup Github Action Workflow first

This code is a GitHub Actions workflow that automates the deployment of a project to a server in a staging environment. Let's go through the code step by step to understand its functionality:

The code starts by defining the environment variables. The APP_ENV variable is set to the value stored in the STAGING secret.
The workflow is triggered when a push event occurs on the main branch.
The build-and-deploy job is defined, which runs on an ubuntu-latest machine.
The steps of the job are as follows:

a. Pull code: This step uses the actions/checkout action to fetch the latest code from the repository.

b. Extract env file multi line: This step creates an environment file (.env) based on the APP_ENV value. It removes any existing .env file, creates a temporary file (.env_temp), writes the APP_ENV value to it, converts spaces to newlines, and appends the contents to the final .env file. Finally, it removes the temporary file.

c. Docker: This step builds a Docker image for the project using the docker build command. It then saves the image as a tar file (your-project-name.tar).

d. Copy Docker image to ec2 use SSH Key: This step uses the appleboy/scp-action action to copy the Docker image tar file and the docker-compose.yml file to the target server. It connects to the server using SSH, specified by the STAGE_SERVER_HOST, STAGE_SERVER_KEY_SSH, and other credentials stored as secrets.

e. Executing remote ssh commands using SSH Key: This step uses the appleboy/ssh-action action to execute remote SSH commands on the target server. It connects to the server using SSH and runs the following commands:
- sudo docker system prune -a -f: Cleans up any unused Docker resources on the server.
- cd ~/your-project-name: Changes the directory to the project directory on the server.
- sudo snap install docker: Installs Docker on the server (if not already installed).
- sudo docker load --inputnestjs-22.tar: Loads the Docker image from the tar file.
- sudo docker-compose up -d --force-recreate: Starts the project using Docker Compose, recreating containers if necessary.
- rm -rfnestjs-22.tar: Removes the Docker image tar file from the server.

This workflow automates the process of building a Docker image, copying it to a server, and deploying the project using Docker Compose. It ensures consistency and simplifies the deployment process in a staging environment.

1. Click on Actions

2. Click on set up a workflow yourself

3. Type "deploy.staging.yml"

4. Copy workflow content from file deploy.staging.yml in folder .github/workflows and paste to input

5. Click on Commit changes...

6. Click on Commit message

7. Select Create a new branch for this commit and start a pull request…

8. Type "prepare-workflow"

9. Click on Propose changes

10. Click on base: main

11. Click on main…

12. Click on Create pull request

13. Click on Create pull request

Managing Secrets and Variables in GitHub Actions

Currently, we will set up for the following keys:

secrets.STAGING
secrets.STAGE_SERVER_HOST
secrets.STAGE_SERVER_KEY_SSH

Note that all of secret key in Github Action workflow must set up on Settings for each repo before use it in Github Action.

1. Click on Settings

2. Click on Secrets and variables

3. Click on Actions

4. Click on New repository secret

5. Type "STAGING"

6. Type "TEST=123"

7. Click on Add secret

8. Click on New repository secret

9. Paste "STAGE_SERVER_HOST" into input

10. Type "abc-ap-south-east-1.ec2.com"

11. Click on Add secret

12. Click on New repository secret

13. Paste "STAGE_SERVER_KEY_SSH" into input

14. Type "Your server key ssh copy from file pem. Must use vscode for copy key properly, don't use notepad or text editor"

15. Click on Add secret

Prepare Dockerfile and docker-compose:

Both file are available in this repository.Make sure your image name is the same as tar file name, which created by docker build command in Build Stage in Github Action. You can see some picture below for more detail

After complete all, we can merge pull request set up github action and ready for deploy to Server EC2.

How to open port for testing web, and prepare for using Portainer later, setup Security Group.

In page detail of EC2, Select Tab "Security". Choose a Security Group.

Click on Inbound rules

Click on Edit inbound rules

Click Add Rule

Choose Custom TCP

Choose suitable Port, example 3000

Change CIDR Blocks to 0.0.0.0

Save Rules

Setup SSL with Caddy Web Server

Setup Portainer CE for manager, logging Docker Container:

https://docs.portainer.io/start/install-ce/server/docker/linux#deployment

Command to setup

docker volume create portainer_data
docker run -d -p 8000:8000 -p 9000:9000 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

Using

http://localhost:9000 or http://<ip_address>:9000

Login UI

Main UI

Container List

View Detail Environment

Checking status, logs, stat

Setup Domain on Namecheap

Copy Your Elastic IPs

Login Namecheap and choose your domain to manage

Click Advanced DNS

Add new Record

Choose A Record

Type wildcard domain name: Example "test" mean "test.todooy.com"

Type your Elastic IP which you copied and click green ticked to save

Setup SSL

Connect to your Server:

Install Caddy web server

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl &&\
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg &&\
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list &&\
sudo apt update -y &&\
sudo apt install caddy -y

Give permission HTTPS for caddy

sudo setcap CAP_NET_BIND_SERVICE=+eip $(which caddy)

Restart Caddy server for take effect

caddy stop
caddy start

Add Caddyfile config

Using the Service If using a Caddyfile, you can edit your configuration with nano, vi, or your preferred editor:

sudo nano /etc/caddy/Caddyfile

kuma.thienlam3.line.pm {
		reverse_proxy localhost:3001
}

Press CTR+O, Press Enter to save, and press CTRL + X to return back to terminal

You can place your static site files in either /var/www/html or /srv. Make sure the caddy user has permission to read the files.

To verify that the service is running:

sudo systemctl status caddy

The status command will also show the location of the currently running service file.

When running with our official service file, Caddy's output will be redirected to journalctl. To read your full logs and to avoid lines being truncated:

journalctl -u caddy --no-pager | less +G If using a config file, you can gracefully reload Caddy after making any changes:

sudo systemctl reload caddy

Add Cronjob to automatically check ssl expire and auto renew

0 0,12 * * * sudo certbot renew --nginx --post-hook "systemctl reload nginx"

Add Cronjob to automatically check ssl expire and auto renew with google chat notification

0 0,12 * * * curl -X POST -H 'Content-Type: application/json' "https://chat.googleapis.com/v1/spaces/AAAAKiLkcjo/messages?key=aaaa&token=123" -d '{"text": "'"$(sudo certbot renew --nginx)"'"}'

Save this file

Reload Caddy web server to apply domain name without downtime

caddy reload

Waiting about 2 to 15 minutes and then cerification obtained appear, and we are done.

Create S3 with cloud formation

(Old method for Domain and SSL)

I will still keep tutorial about Nginx, HTTPS and SSL

Create a file in folder /etc/nginx/site-enabled/your_domain_name And paste this content

server {
        listen 80;
        server_name test.thienlam3.line.pm;
        return 301 https://test.thienlam3.line.pm$request_uri;
}

#Point test.thienlam3.line.pm to port 3002
server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;

        server_name test.thienlam3.line.pm http://test.thienlam3.line.pm;
        location / {
              proxy_pass http://localhost:3002;
              client_max_body_size 5M;
              proxy_set_header Host $host;
              proxy_set_header X-Real-IP $remote_addr;
              proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
              proxy_set_header X-Forwarded-Proto $scheme;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection "upgrade";
              proxy_read_timeout 86400;
        }
}

Save this file, then using certbot to auto generated ssl for above config

sudo certbot --nginx

After choose suitable domain, then reload Nginx to take effect

sudo service nginx reload

Check nginx status again

sudo service nginx status

lamminhthien/self-hosted-on-server-on-promise-and-cloud