While the project can still be used for reference, the reality is that it has aged and no longer reflects best practices. To make this clear, the project is now archived.
Live App Demo: https://node-api-starter-angular-app.experiments.explabs.io
Live API: https://node-api-starter.experiments.explabs.io/v1/hello
A boilerplate for Node.js APIs designed for app consumption, written in Typescript.
This project has two purposes:
- Provide a boilerplate for modern Node.js API development with the following requirements
- fully configured development environment
- created to be used primarily as a backend for SPAs or PWAs
- have all the most common requirements already implemented (authentication, database integration, CI integration and more)
- Serve as a reference for various implementations from CD with CircleCI to putting it all together with an example app.
You can find the Angular App source code here https://github.com/feredean/node-api-starter-angular-app
- Requirements
- Getting started
- Environment variables
- Deployment
- Project structure
- Build scripts
- Import path quirks
- Debugging
- Testing
- Dependencies
- Resources
- Related projects
- License
There are two ways to go about handling requirements. You can either follow the quick start path where you dump all the dependencies in your system or you can go down the fancy setup that will give you more flexibility in the future by using nvm and docker.
- Install Node Version Manager
- Configure nvm Shell Integration (highly recommend setting up zsh together with oh my zsh). Once you set it up it will automatically change the node version if the project has a
.nvmrcfile. - Install docker
- Run MongoDB in a docker container
docker run -d --name mongo-dev -p 27017:27017 mongo:4.0
Set VSCode's Typescript import module specifier for the workspace to relative for more information have a look here
# Get the latest snapshot
git clone --depth=1 https://github.com/feredean/node-api-starter.git <project_name>
# Change directory
cd <project_name>
# Install dependencies
npm install
# Build the project
npm run build
# Copy the .env.example contents into the .env
cat .env.example > .env
# Run (development mode) the API on port 9100
npm run watchTo build the project in VS Code press cmd + shift + b. You can also run tasks using the command pallet (cmd + shift + p) and select Tasks: Run Task > npm: start to run npm start for you.
Finally, navigate to http://localhost:9100/v1/hello and you now have access to your API
For how environment variables are imported and exported have a look in src/config/secrets. Here you can also change the requiredSecrets or the way mongoURI is constructed if for example you wish to use username/password when connecting to mongo in the development environment.
| Name | Description |
|---|---|
| The session secret is used to sign the JWT tokens | |
| SESSION_SECRET | A quick way to generate a secret: node -e "console.log(require('crypto').randomBytes(256).toString('base64'));" |
The mongo host and port are not necessarily taken from the .env file they can be provided by the deployment environment such as k8s |
|
| MONGO_HOST | mongo host |
| MONGO_PORT | mongo port |
| MONGO_DATABASE | name of the database |
| MONGO_USERNAME | mongo user - not used for development, required for production |
| MONGO_PASSWORD | mongo user's password - not used for development, required for production |
| Facebook credentials used for sign in with Facebook - currently not implemented | |
| FACEBOOK_ID | Facebook ID |
| FACEBOOK_SECRET | Facebook Secret |
Sendgrid credentials used by the nodemailer package in forgot/reset password functionality |
|
| SENDGRID_USER | Sendgrid account user name |
| SENDGRID_PASSWORD | Sendgrid account password |
AWS user used for uploading files to s3 with AmazonS3FullAccess Policy |
|
| AWS_ACCESS_KEY_ID | AWS Access key ID |
| AWS_ACCESS_KEY_SECRET | AWS Access key secret |
| This will be used to create a REGEX that will block origins that don't match | |
| CORS_REGEX | use localhost:\d{4}$ for development and domain\.tld$ for production |
The example in this project is built around the existence of a kubernetes cluster. You can easily change to your infrastructure of choice by changing the deploy step in .circleci/config.yml to pull the docker image wherever you need it.
# pull the image from docker hub and deploy it to the k8s cluster
deploy:
docker:
- image: feredean/circleci-kops:0.1.0
environment:
IMAGE_NAME: feredean/node-api-starter
KOPS_STATE_STORE: s3://k8s-explabs-io-state-store
steps:
- run:
name: Deploy to k8s cluster
command: |
# Ensure AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set in the project's env vars
kops export kubecfg --name k8s.explabs.io
kubectl set image deploy/node-api-starter node-api-starter=$IMAGE_NAME:$CIRCLE_SHA1Depending on your cloud provider of choice you can fairly quickly set up a managed, production-ready environment for deploying containerized applications.
This project is deployed on a cluster set up with kops on aws spot instances. If there is interest I plan on going more in depth on this subject and provide a walk-through.
If you're like me and don't want the headache and uncertainty of managing your own production database take a look at mongodb's atlas. If you feel up to the task there are some kubernetes projects like KubeDB that can be of use.
First you need to have an .env.prod file that has all the secrets that will be used in production. A node-starter secret needs to be created, it is used by the API deployment.
kubectl create secret generic node-starter --from-env-file=.env.prodNotice that in .kubernetes/deployment.yaml the environment is loaded from the node-starter secret
envFrom:
- secretRef:
name: node-starterFinally you need to create the kubernetes deployment, service and optionally the horizontal pod autoscaler that can later be paired with the cluster autoscaler. To do this simply run the following:
kubectl create -f .kubernetes/deployment.yamlIf somehow a deadly bug has managed to make its way past the test suite and got deployed to production where it's wreaking havoc you need to run following command:
kubectl rollout undo deployment <your deployment name>This will instantly roll back the deployment to the previous one.
To integrate with CircleCI:
-
Go to CircleCI and create an account
-
Link your project
-
Add the needed environment variables to run the test
# Used to connect to the kubernetes cluster AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY # Used for publishing the image DOCKERHUB_PASS DOCKERHUB_USERNAME
-
Make master branch a protected branch require
ci/circleci: testcheck before merging from feature branches. Once a PR is merged into master CircleCI will automatically build, test and deploy the new version of the API.
Congratulations! You how have an API set up and ready to embrace the CD workflow!
| Name | Description |
|---|---|
| .circleci | Contains CircleCI settings for continuous deployment |
| .kubernetes | Contains kubernetes configuration for running the app on a cluster (auto-scaling included) |
| .vscode | Contains VS Code specific settings |
| dist | Contains the distributable (or output) from your TypeScript build. This is the code you ship |
| node_modules | Contains all your npm dependencies |
| src | Contains your source code that will be compiled to the dist dir |
| src/api | Contains all the API versions each with it's own controllers for the configured routes |
| src/config | Contains all the configuration needed to setup the API (express, routes and passport) |
| src/models | Models define Mongoose schemas that will be used in storing and retrieving data from MongoDB |
| src/types | Holds .d.ts files not found on DefinitelyTyped |
| src/utils | Contains API wide snippets (Logger, Error Formatter) |
| src/server.ts | Entry point to your express app |
| test | Contains your tests. Separate from source because there is a different build process |
| test/tsconfig.json | Config settings for compiling the tests |
| .env | All the env variables needed to run the app. Gitignored, will be loaded by dotenv |
| .env.example | All the env variables needed to run the app. An example list of the keys that must exist in .env files |
| .env.prod | All the env variables needed to run the app in production. Gitignored, will be used in the deployment |
| .eslintignore | Config settings for paths to exclude from linting |
| .eslintrc | Config settings for ESLint code style checking |
| .nvmrc | A file containing the node version used in the project automatically loaded by nvm |
| Dockerfile | Used to build the docker image in the dockerize job in .circleci/config.yml |
| jest.config.js | Used to configure Jest running tests written in TypeScript |
| package.json | File that contains npm dependencies as well as build scripts |
| tsconfig.json | Config settings for compiling server code written in TypeScript |
npm scripts can be found in package.json in the scripts section. They can call each other which means it's very easy to compose complex builds out of simple individual build scripts.
| Npm Script | Description |
|---|---|
start |
Runs tsc -w (continuously watches .ts files and re-compiles when a change is made) and nodemon dist/server.js (runs node with nodemon so the process restarts when a change is made) concurrently. Use this for development |
test |
Runs tests using Jest test runner verbosely and generate a coverage report |
test:watch |
Runs tests in watch mode |
test:debugger |
Waits for a debugger to get attached and then runs tests |
test:debugger:watch |
Waits for a debugger to get attached and runs tests in watch mode |
build |
Full build. Runs build-ts and lint build tasks |
build-ts |
Compiles all source .ts files to .js files in the dist folder |
lint |
Runs ESLint on project files |
check-deps |
Audits and upgrades (inside package.json run npm install to apply) dependencies to their latest stable version |
To change the way VSCode does auto import simply search for typescript import module in settings and change it to relative for the workspace.
You need to do this because
module names are considered resource identifiers, and are mapped to the output as they appear in the source
As a result the import paths will be copied over to the compiled js require paths. The compiled code will not work since the tsconfig options are not applied to the output. The Typescript compiler does not want to become a build tool. Normally in frontend projects this is taken care of by build tools such as webpack. There are packages that offer solutions, more on this here.
If you really want to use absolute paths you can find a working example of this project using a different approach at this commit. I decided to drop it going forward since imports are usually added via autocompletion. The visual improvements from
import { UserDocument, User } from "../../../models/User";
import {
SESSION_SECRET,
SENDGRID_USER,
SENDGRID_PASSWORD
} from "../../../config/secrets";
import {
JWT_EXPIRATION,
UNSUBSCRIBE_LANDING,
RECOVERY_LANDING,
SENDER_EMAIL
} from "../../../config/settings";
import { formatError } from "../../../util/error";
import {
passwordResetTemplate,
passwordChangedConfirmationTemplate
} from "../../../resources/emails";
import { SUCCESSFUL_RESPONSE } from "../../../util/success";to
import { User, UserDocument } from "models/User";
import {
SESSION_SECRET,
SENDGRID_USER,
SENDGRID_PASSWORD
} from "config/secrets";
import { JWT_EXPIRATION, UNSUBSCRIBE_LANDING } from "config/settings";
import { formatError } from "util/error";
import * as emailTemplates from "resources/emails";
import { SUCCESSFUL_RESPONSE } from "util/success";do not justify the complexity that comes with adding absolute path support.
Debugging TypeScript requires source maps to be enabled in tsconfig.json:
"compilerOptions" {
"sourceMap": true
}In .vscode folder you can find the launch.json file. Here you can find the configuration that tells VS Code how to attach the debugger to the node process.
{
"type": "node",
"request": "attach",
"name": "Attach Debugger to Process ID",
"processId": "${command:PickProcess}",
"protocol": "inspector"
}Once this configuration is added make sure that either the app is running (npm run watch) or tests are running in debug mode (npm run watch-test-debugger). Now add breakpoints, hit F5, select the process you want to attach the debugger to and you're ready to go!
This project uses Jest. When writing tests that interact with mongoose keep this in mind.
When writing integration tests that use a shared resource (a database for example) you need to keep in mind that jest will test separate files in parallel which will lead to tests interfering with each other. For example lets say you want to test that GET /v1/account/ will return a user you inserted just before you made the call. In another file you need to create a user in order to test something else. If you use the same database it is possible that GET /v1/account/ will sometimes return one user (the one inserted in the test) and other times return multiple users (that got inserted by other tests).
In order to avoid this you have some options:
- Keep all the tests that use a shared resource in the same file
- Get really creative with your setup
- Use the option
--runInBandto force all the tests to run serially in the current process - Set up the tests in such a way that each file uses a separate database
After running into issue with all the other options I decided to move all the tests into one file.
In order to properly load modules in the test suites a new test/tsconfig.json file is needed.
In jest.config.js you can find setupFilesAfterEnv: ["./test/setup.ts"] where the test environment variables are set. In the setup file you can also find the initMongo and disconnectMongo helper functions. They are used to connect/disconnect to the test database and empty the database before starting a test. The Typescript compilation to JS will happen in memory using the test/tsconfig.json file.
To run the tests simply use npm test. If you want to use jest watch mode use npm run watch-test.
This year Palantir has announced the deprecation of TSLint.
In order to avoid bifurcating the linting tool space for TypeScript, we therefore plan to deprecate TSLint and focus our efforts instead on improving ESLint’s TypeScript support.
This project is using ESLint with typescript-eslint/recommended settings.
| Package | Description |
|---|---|
| aws-sdk | Amazon Web Services SDK - used to connect to s3 |
| bcrypt | A library to help you hash passwords. |
| body-parser | Express 4 middleware. |
| compression | Express 4 middleware. |
| cors | Express middleware that can be used to enable CORS with various options |
| dotenv | Loads environment variables from .env file. |
| express | Node.js web framework. |
| fbgraph | Facebook Graph API library. |
| jsonwebtoken | An implementation of JSON Web Tokens. |
| lodash | General utility library. |
| mongoose | MongoDB ODM. |
| morgan | HTTP request logger middleware |
| multer | Middleware for handling multipart/form-data |
| nodemailer | Node.js library for sending emails. |
| passport | Simple and elegant authentication library for node.js |
| passport-facebook | Sign-in with Facebook plugin. |
| passport-local | Sign-in with Username and Password plugin. |
| uuid | Simple, fast generation of RFC4122 UUIDS. |
| validator | A library of string validators and sanitizers. |
| winston | Logging library |
| Package | Description |
|---|---|
| @types | Dependencies in this folder are .d.ts files used to provide types |
| concurrently | Utility that manages multiple concurrent tasks. Used with npm scripts |
| eslint | Linter for JavaScript and TypeScript files |
| jest | Testing library for JavaScript |
| nodemon | Utility that automatically restarts node process on code changes |
| npm-check-updates | Upgrades package.json dependencies to the latest versions, ignoring specified version |
| supertest | HTTP assertion library |
| ts-jest | A preprocessor with sourcemap support to help use TypeScript with Jest |
| typescript | JavaScript compiler/type checker that boosts JavaScript productivity |
If you're the type of person that likes to live life on the bleeding edge feel free to use npm run check-deps
This section is a list of resources for building an API that can be useful in certain situations
- If you are unsure what format your API's JSON responses should have take a look at this specification and see if it could work for your project.
- Kong is a cloud-native, fast, scalable, and distributed Microservice Abstraction Layer (also known as an API Gateway, API Middleware or in some cases Service Mesh). It boasts a lot of cool features and of course works with kubernetes
- RESTful API Modeling Language (RAML) makes it easy to manage the whole API lifecycle from design to sharing. It's concise - you only write what you need to define - and reusable. It is machine readable API design that is actually human friendly.
- Brought to you by Heroku, 12factor is a methodology for building software-as-a-service applications
I highly recommend taking a look at both Sahat's Hackathon Starter and Microsoft's TypeScript Node Starter. Both have been great help and a source of inspiration for setting up this project.
The MIT License (MIT)
Copyright (c) 2019 Tiberiu Feredean
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.