A redesigned and scalable Rest API using:
- Express.js
- PostgreSQL
- NGINX
- PM2
- dotenv
Deployed on AWS EC2 instances, and stress-tested with:
- K6
- Loader.io
To run the node server locally:
- Clone this repo
npm install
to install project dependencies- Install PostgreSQL and configure per instructions below
- Update the
.env.example
file
- run
npm install
to install dependencies
The following instructions were used for installing and configuring Postgres v14
on both Ubuntu 20.04 and macOS systems.
The needs of this REST API required the database to return a nested object to the front-end. PostgreSQL is an open source object-relational database, and is known for its reliability and performance. The software is capable of performing aggregate functions that were necessary for the API's /meta
endpoint.
apt
, also known asapt-get
, is a package handler for Ubuntu- The following commands will update the package lists that are configured inside
/etc/apt/sources.list
sudo apt update
sudo apt upgrade
Confirm you want to continue by entering y
when prompted.
- Or, if you prefer to breeze past the confirmation dialogs, use the
-y
option:
sudo apt update -y && sudo apt upgrade -y
sudo apt install postgresql
Confirm you want to continue by entering y
when prompted.
sudo systemctl status postgresql
Likewise, to start the service:
sudo systemctl start postgresql
And to stop the service:
sudo systemctl stop postgresql
- Follow the instructions on the Homebrew site.
- Helpful instructions are available here 🙏
To start the service with Homebrew and automatically enable launch at login:
brew services start postgresql
Check the status of your installed Homebrew services:
brew services list
To stop the service:
sudo systemctl stop postgresql
- For a full list of available Homebrew commands:
brew services -h
sudo -u postgres psql
You should see in your terminal :
postgres=#
CREATE DATABASE your_database;
Note the trailing semicolons in SQL code!
The semicolon is often, though not always, used in SQL code as a statement terminator.
Wrap your chosen password in single quotes:
CREATE USER your_username WITH PASSWORD 'password';
If this was successful, you should see CREATE ROLE
in the terminal.
GRANT ALL TO your_username ON your_database;
More Official PostgreSQL resources
Getting Started With PostgreSQL DOCS
Basic SQL commands in PostgreSQL to get you started:
-
\q
- Exit psql -
\l
or\l+
- List databases -
\dt
ordt+
- List tables -
\c
+<your_database>
to change to a different database -
Press
q
to close a command menu if the terminal displays a:
or(END)
- From the terminal of your PostgreSQL server, run:
psql postgres -f ./pg_reviews_etl.sql
-
Transferring from .CSV files to a temporary table first, as type
TEXT
, is significantly faster, but does require more RAM. Once the file(s) are loaded into PostgreSQL, they are parsed with data type constraints and foreign keys and indexes are added. The temporary tables are dropped to free up memory space before proceeding to the next file. -
I used SCP to send .CSV files to my virtual machine instance. I deployed on
AWS EC2
t2.micro This API is designed to be scale-agnostic and will work with any cloud service running Ubuntu Server 20.04.
6. Set up the .env file to use with the Express application server instances. The variables should correspond with the PostgreSQL config settings
... SSH into the remote machine, install node, etc...
npm start
Runs the Express app server using nodemon
8. Install PM2 to daemonize the server process. I'd suggest using a startup script, and configuring to reload if it crashes
10. To increase throughput and decrease load on any one particular app server, install NGINX as a web server in front of your app server instances.
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=myzone:10m
loader_threshold=300 loader_files=200;
upstream reviews_api {
server 11.111.111.111:3000/;
server 2.22.222.222:3000/;
server 3.33.333.333:3000/;
}
server {
listen 80;
listen [::]:80;
proxy_cache myzone;
proxy_cache_valid 5m;
location / {
proxy_pass http://reviews_api;
}
}
/reviews/
GET /reviews/&product_id=12345
GET /reviews/&product_id=12345&sort=date&count=10
Parameter | Type | Required | Description |
---|---|---|---|
product_id |
int |
YES | Will return an error if invalid |
sort |
string |
NO | date , relevance , helpful . Default 'relevance' sorts by date first, then by average helpfulness |
count |
int |
NO | Results per page. Default 5 |
page |
int |
NO | Which page of results to return. Default 1 |
Returns metadata for a current project
GET /reviews/meta:product_id
Parameter | Type | Required | Description |
---|---|---|---|
product_id |
int |
YES | Will return an error if invalid |
{
"product_id": "2",
"ratings": {
2: 1,
3: 1,
4: 2,
// ...
},
"recommended": {
0: 5
// ...
},
"characteristics": {
"Size": {
"id": 14,
"value": "4.0000"
},
"Width": {
"id": 15,
"value": "3.5000"
},
"Comfort": {
"id": 16,
"value": "4.0000"
},
// ...
}
POST /reviews
[
"https://link.to/photo1.jpg",
"https://link.to/photo2.jpg",
// ...
]
Parameter | Type | Required | Description |
---|---|---|---|
product_id | integer |
YES | Required ID |
rating | int |
NO | Integer (1-5) indicating the review rating |
summary | text |
NO | Summary text of the review |
body | text |
NO | Continued or full text of the review |
recommend | bool |
NO | Value indicating if the reviewer recommends the product |
name | text |
YES | Username for question asker |
text |
YES | Email address for question asker | |
photos | [text] |
NO | Array of text urls that link to images to be shown |
characteristics | object |
NO | Object of keys representing characteristic_id and values. { "14": 5, "15": 5 //...} |
PUT /reviews/:review_id/helpful
Parameter | Type | Required | Description |
---|---|---|---|
product_id |
int |
YES | Will return an error if invalid |
PUT /reviews/:review_id/report
Updates a review to show it was reported. Note, this action does not delete the review, but the review will not be returned in the above GET request.
Parameter | Type | Required | Description |
---|---|---|---|
product_id |
int |
YES | Will return an error if invalid |