Production ready server setup.
Creating a Linux instance
To create an instance, I will be using Digital Ocean (DO) as its UI is simple and clean to use. In DO world they are called droplets. Though you can use AWS EC2, Linoe, Vultr, etc... whatever you like.
If you dont have an account on DO then use the link (https://m.do.co/c/3208f08b3324) to get 100$ credit to get started.
Youtube tutorial
Before you continue, if you wanna learn in depth that what each of the stuff does then you can watch my youtube playlist for the same setup. There might be something more than this. YouTube - Rock solid Server setup tutorial
Generate a SSH key (if not already done)
Open a terminal and run the following command
ssh-keygen
You will be prompted to save and name the key.
Generating public/private rsa key pair. Enter file in which to save the key (/Users/USER/.ssh/id_rsa)
Next you will be asked to create and confirm a passphrase for the key (highly recommended):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
This will generate two files, by default called id_rsa and id_rsa.pub. Next, add this public key.
Copy the public key in clipboard to be later used when creating a DO droplet.
cat id_rsa.pub | pbcopy
Create a CENTOS 8.x droplet on digital ocean
I chose the following paramters for my droplet, only the things you need to select are written here, rest everything is default.
Selection | Value |
---|---|
Image | CentOS 8.x |
Plan | Basic (5$) |
Add block storage | None |
Datacenter region | Bangalore (choose which is nearest to you) |
Authentication | SSH |
For Adding SSH keys to DO, click NEW SSH key, and follow simple instructions.
Finally click on create Droplet and sit back and relax for a moment.
SSH into your droplet with root user
Get the IP of the droplet from your DO droplets page, and then ssh into it.
ssh root@XXX.XXX.XXX.XXX
# XXX.XXX.XXX.XXX represents your IP
The authenticity of host '139.59.47.191 (139.59.47.191)' can't be established.
ECDSA key fingerprint is SHA256:rup2QTATg6cOBDcMiP0abSmsOO+4eAMKA4q/Z7O2xVc.
Are you sure you want to continue connecting (yes/no)?
# type yes and press enter
Now you will or might get error like this
Warning: Permanently added '139.59.47.191' (ECDSA) to the list of known hosts.
root@139.59.47.191: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
Now add the generated ssh key in
ssh-add ~/.ssh/do_youtube
# Identity added: /Users/killer/.ssh/do_youtube (jhondoe@iMac.local)
Finally now you can SSH into your remote machine
ssh root@XXX.XXX.XXX.XXX
Huh, long work, but your droplet is up and running and time to play with it...and make it production ready.
Update the OS using dnf package manager (comes pre-install with centos 8.x)
dnf update -y
It will really take sometime, so sit back relax and have a cup of tea
Install packages using dnf
dnf install curl vim git wget '@Development tools' nmap net-tools
Once again, it will take sometime, so sit back relax and have a second cup of tea
Adding EPEL to CentOS 8
dnf install epel-release
dnf upgrade
Start and install automatic updates
dnf install dnf-automatic -y
systemctl enable --now dnf-automatic.timer
systemctl list-timers *dnf-*
Install Snapd from
(https://snapcraft.io/docs/installing-snapd)
sudo dnf install snapd -y
sudo systemctl enable --now snapd.socket
sudo ln -s /var/lib/snapd/snap /snap
Setup sudo user with root priveleges
- create a User with your choice of USERNAME, we will make this user have sudo access
adduser USERNAME
- add this user to the wheel group using
usermod -aG wheel USERNAME
(wheel group is always present in centos and has sudo access)
usermod -aG wheel USERNAME
-
Allow wheel group to have password-less sudo
- edit file
/etc/sudoers
(usingvisudo
)
visudo /etc/sudoers
-
%wheel ALL=(ALL) NOPASSWD: ALL
-
validate using
/usr/sbin/visudo -cf /etc/sudoers
OR
- edit file
-
add a file called USERNAME with the content below to only have this user to use password-less sudo
visudo /etc/sudoers.d/USERNAME
USERNAME ALL=(ALL) NOPASSWD: ALL
# validate using
/usr/sbin/visudo -cf /etc/sudoers
Set authorized_key for remote user USERNAME using rsync
rsync --archive --chown=USERNAME:USERNAME ~/.ssh /home/USERNAME
OR
# (700 only current user can Read Write Execute)
# (600 only current user can Read Write)
su USERNAME
mkdir .ssh
chmod 700 .ssh
touch .ssh/authorized_keys
chmod 600 .ssh/authorized_keys
# copy **PUBLIC** SSH KEY in this file created above
Finally exit out of your remote machine and SSH into your remote machine as this sudouser USERNAME
ssh USERNAME@XXX.XXX.XXX.XXX
Setup sshd_config
cd /etc/ssh
cat /etc/ssh/ssh_config
#create a file here like ssh_config.conf
/etc/ssh/ssh_config.d/*.conf
# with following content
PasswordAuthentication no
Permitrootlogin no
# Save and quit and validate...
# optionally, but not recommended to directly edit the sshd_config, though you can
# edit `sshd_config`
# PermitRootLogin no (OR without-password)
# PasswordAuthentication no
# AllowUsers killer (etc...)
# Or you can also use 'DenyUsers' directive to deny particular users, (am not doing it)
# Finally
Validate sshd_config
sshd -T -f /etc/ssh/sshd_config
# OR (from any path)
sshd -t
Reload SSHD service
systemctl reload sshd
Optionally, for this new user, create ssh public/private to be used on github for private repos.
# run this command when loggin into ssh with this new user only
ssh-keygen -t ed25519
(Optional) Delete other user centos
userdel centos
# You can also delete that user's home directory and mail spool by using the -r flag with the command
# recommended
userdel -r centos
(Optional) Install fish terminal
I recommend installing fish shell, since it prettifies the terminal session and provides with autocompletion and much more.
sudo dnf install fish -y
# List all present shells
cat /etc/shells
# change shell to fish instead of bash
sudo usermod --shell $(which fish) USERNAME
# exit and ssh again, to use fish shell
Install Nodejs
(https://nodejs.org/en/download/)
There are two options to install nodejs, one is using dnf and other using snaps
- To see a list of available streams, where stream corresponds to the major version of Node.js. For example, to install Node.js 14:
dnf module list nodejs
# Choose the version using:
sudo dnf module enable nodejs:14 -y
sudo dnf module install nodejs:14 -y
node --version
npm --version
######OR
- using snap craft (https://github.com/nodejs/snap)
sudo snap install node --classic --channel=14
Setup PM2 (process manager for node applications)
Install and run the startup script
# ssh as current USER not as root
sudo npm install -g pm2
Either use pm2 startup
and follow the instructions
OR use this script below
Start and enable pm2 as a service
sudo env PATH=$PATH:/usr/local/bin pm2 startup systemd -u USERNAME --hp /home/USERNAME
Check status of pm2
systemctl status pm2-USERNAME
IMPORTANT NOTE: If the Active status is not Active: active (running) and is something like failed or activating , then execute the following commands in order (just for info, this is related to SELINUX), and everything should be fine, else if still it creates a problem you can make an issue in this repo only after trying out the fix.
# Change to root user
sudo su
# The following start command will timeout
systemctl start pm2-USERNAME
# Then do this
ausearch -c 'systemd' --raw | audit2allow -M my-systemd
semodule -i my-systemd.pp
# NOTE: this is NOT a mistake, it has to be done twice
systemctl start pm2-USERNAME
# Again, the above start command will timeout
ausearch -c 'systemd' --raw | audit2allow -M my-systemd
semodule -i my-systemd.pp
systemctl start pm2-USERNAME
# Finally the above start command will start the pm2 service, and verify it with
systemctl status pm2-USERNAME
# sudo reboot
Start the Nodejs application
-
Login as the user created above with root privilages
-
Install yarn global packages
sudo npm install -g yarn
- Starting the Nodejs application
# Clone the repo any dir, example: inside this folder `~/application`
cd ~/application
git clone REPO_URL application
yarn install
# To start pm2 in cluster mode, ie, to start 2 instances of application use (-i 2) or else simple pm2 start app.js
PORT=4000 pm2 start app.js -n my_app
pm2 save
- list application using
pm2 monit
NGINX as Reverse proxy
- Install Nginx (make sure epel-release is already installed, though already done above)
dnf install nginx
- Enable SELinux for Nginx httpd_t (IMPORTANT)
semanage permissive -a httpd_t
# OR
sudo setsebool -P httpd_can_network_connect on
# Other SELinux commands
getenforce
# read selinux context (type) of files and processes
# -P = persist setting
sudo setsebool -P httpd_can_network_connect on
# To read home dirs, specially for app public directory served through nginx
sudo setsebool -P httpd_enable_homedirs on
chcon -Rt httpd_sys_content_t /home/USERNAME/app/public
- Disable Nginx Server Header
# nginx.conf
http {
# Disable emitting nginx version on error pages and in the "Server" response header field
server_tokens off;
}
-
create
example.conf
file inside/etc/nginx/conf.d/
-
Sample server configutaion file are present in templates folder.
SSL certificates using certbot
(https://certbot.eff.org/lets-encrypt/snap-nginx.html)
- Install snapd, if not already done, install Snapd from (https://snapcraft.io/docs/installing-snapd)
sudo dnf install snapd -y
sudo systemctl enable --now snapd.socket
sudo ln -s /var/lib/snapd/snap /snap
- Install certbot from snapd
sudo snap install core
sudo snap refresh core
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot
- Generate the certificate using the below command
sudo certbot --nginx -d example.com,www.example.com -n -m youremail@example.com --agree-tos
# --nginx Automatic, GETS and INSTALLS
# -n non-interactive
# -m for emails about certs
Test automatic renewal
sudo certbot renew --dry-run
Check renew status timer
systemctl list-timers *certbot*
Github Webhook support for automated deployments
https://github.com/adnanh/webhook
First create a webhook from repository, is is located under settings of the repo. settings → webhook → add webhook
- add a payload url of your choice (https://example.com/hooks/redeploy-app)
- any secret
content-type
to:application/json
- just the push event
- active
payload url: is the url which will be called whenever a push occurs on your repo (example: https://example.com/hooks/redeploy-app) secret: is the secret used to verify that the url is triggered by github
Only then follow these steps below.
- Install webhook using binary (https://github.com/adnanh/webhook/releases)
cd ~
wget https://github.com/adnanh/webhook/releases/download/2.8.0/webhook-linux-amd64.tar.gz
# 62ab801c7337a8b83de8d6ae8d7ace81 webhook-linux-amd64.tar.gz
# Check and verify md5 checksum (use md5 on MacOS and md5sum on linux)
md5sum webhook-linux-amd64.tar.gz
tar -xvf webhook-linux-amd64.tar.gz
sudo mv webhook-linux-amd64/webhook /usr/local/bin
rm -rf webhook-linux-amd64.tar.gz webhook-linux-amd64
webhook --version
# Run below on to test if its able to execute and then close (Ctrl+C)
webhook -verbose
- Create
redeploy.sh
script inside~/webhooks
folder (you can choose ur folder name but from now i am considering its called webhooks)
cd ~
mkdir webhooks
cd webhooks
touch redeploy.sh hooks.json
# change redeploy.sh permissions to be executable
chmod +x redeploy.sh
-
Redeploy script
redeploy.sh
is present in templates folder (you can modify according to your needs) -
hooks.json
is present in thetemplates
folder of this repo (you can modify according to your needs but read documentation)
-
Detailed Reference @ https://github.com/adnanh/webhook/blob/master/docs/Hook-Definition.md
- id: This is the endpoint used in url, you can change if you want or keep it as is Note: In our case when creating a webhook on Github, the webhook url would look something like this: https://servername.com/webhooks/github-push
- execute-command: Full Path to the redeploy script (not with ~, example: /home/killer/webhooks/redeploy.sh)
- command-working-directory: Full Path to the working directory, where this redeploy script needs to be executed, in case of a nodejs application, it should be the place where package.json is placed.
Refer hooks.json in templates folder
- If you don't understand just leave it as is, and ONLY change SOME_SUPER_SECRET_FROM_GITHUB_WHILE_CREATING_WEBHOOK, and only specify the branch name to deploy
- Only redeploy on a push to the master branch. Change this value if your branch has a different name
- Try your hook is testing mode
cd ~/webhooks
webhook -hooks hooks.json -hotreload -verbose -ip "127.0.0.1" -http-methods post
# Leave the webhook running in verbose mode and try to commit changes and it should work
- If everything above works and you have automated deployment, woo-hoo, time to do some more stuff, i.e. create a systemd service for the same
- create a file called
webhook.service
inside/etc/systemd/system
- change permission of file webhook.service to 0644
chmod 644 webhook.service
- copy the contents from the
webhook.service
file from the templates folder- NOTE:
- killer with the name of user who starts this service, can be USERNAME of logged in ssh user
- /home/killer/webhooks/hooks.json this is my path, provide a complete path to your hooks.json created above
- NOTE:
- Start and enable the service using:
sudo systemctl start webhook.service
(starts the service)sudo systemctl enable webhook.service
(Enable, so that service is auto-started on reboot)sudo systemctl status webhook.service
(Check status, and make sure its running and enabled)
- If the service is NOT active, it is due to selinux, use the command below and troubleshoot, and it will provide you with details as to what to execute and IT WILL WORK.
journalctl -xe
- In case you want to reload/restart/stop/disbale service:
sudo systemctl reload webhook.service
sudo systemctl restart webhook.service
sudo systemctl stop webhook.service
sudo systemctl disable webhook.service
Fail2Ban
# Assuiming you already install epel-repo (which is already done above)
sudo dnf install fail2ban -y
sudo systemctl start fail2ban
sudo systemctl enable fail2ban
# To see failed ssh attempts
tail -f /var/log/secure
grep 'sshd.*Failed password for' /var/log/secure
Configure Fail2ban settings, create or edit the /etc/fail2ban/jail.d/jail-sshd.conf
file using a text editor such as vi/vim or nano/emacs, with the contents from file jail-sshd.conf
inside templated folder, save and exit file, and reload fail2ban
sudo systemctl reload fail2ban
Helper commands for fail2ban (sshd)
sudo fail2ban-client status sshd
Unban an IP:
sudo fail2ban-client set sshd unbanip x.x.x.x
Ban an IP:
sudo fail2ban-client set sshd banip x.x.x.x
Firewalld commands
sudo dnf install firewalld
sudo systemctl start firewalld
sudo systemctl enable firewalld
sudo firewall-cmd --add-service http --zone public --permanent
sudo firewall-cmd --add-service https --zone public --permanent
sudo firewall-cmd --reload
# List all ports and services by the following commands
firewall-cmd --list-ports
firewall-cmd --list-services
# Note about: dhcpv6-client which allows incoming DHCP v6 responses to pass - this is the dhcpv6-client rule. If you're not running DHCP v6 on your network or you are using static IP addressing, then you can disable it.
# Close all other services and ports, which are not required
# The command refrence is below
sudo systemctl reload firewalld
sudo systemctl restart firewalld
sudo firewall-cmd --state
sudo firewall-cmd --list-services
sudo firewall-cmd --get-zones
sudo firewall-cmd --zone public --list-all
sudo firewall-cmd --zone=public --permanent --add-service=http
sudo firewall-cmd --remove-service http --zone public
firewall-cmd --get-default-zone
sudo firewall-cmd --zone=public --add-port=3000/tcp
sudo firewall-cmd --zone=public --remove-port=3000/tcp
Timezones and NTP
Configure Timezones
Our first step is to set our server's timezone. This is a very simple procedure that can be accomplished using the timedatectl command:
- First, take a look at the available timezones by typing:
sudo timedatectl list-timezones
This will give you a list of the timezones available for your server. When you find the region/timezone setting that is correct for your server, set it by typing:
sudo timedatectl set-timezone region/timezone
- For instance, to set it to United States eastern time, you can type:
sudo timedatectl set-timezone America/New_York
Your system will be updated to use the selected timezone. You can confirm this by typing:
sudo timedatectl
Configure NTP Synchronization
Now that you have your timezone set, we should configure NTP. This will allow your computer to stay in sync with other servers, leading to more predictability in operations that rely on having the correct time.
For NTP synchronization, we will use a service called ntp, which we can install from CentOS's default repositories:
sudo dnf install ntp
Next, you need to start the service for this session. We will also enable the service so that it is automatically started each time the server boots:
sudo systemctl start ntpd
sudo systemctl enable ntpd
Your server will now automatically correct its system clock to align with the global servers.
List services
systemctl list-unit-files
systemctl --type=service
app.js
For Nodejs Express Applications always disable "x-powered-by" response header inside app.disable('x-powered-by');
Change default SSH Port
We are going to change the default SSH port (i.e. 22) to something else like 2607
# change to root
sudo su
# IMPORTANT: First add port to firewall and then reload firewall
firewall-cmd --add-port=2607/tcp --zone=public --permanent
firewall-cmd --reload
#Edit this file
vim /etc/ssh/sshd_config
# Find commented "Port 22" line and change to
PORT 2607
# For SELinux systems use the command below (change port accordingly)
semanage port -a -t ssh_port_t -p tcp 2607
systemctl reload sshd
Debugging and Errors
- For CANNOT SET LOCALE ERROR, login as root user for setting locale
utf-8
vim /etc/environment
- add these lines - LANG=en_US.utf-8 - LC_ALL=en_US.utf-8
Author
Contribute
You can fork this repo and send me a PR.
License
This project is licensed under the MIT License.