FormSG: A TypeScript repository from SuriyaRuk

Contributing
- IMPORTANT NOTE TO ALL CONTRIBUTORS
Features
Local Development (Docker)
Testing
- Testing Prerequisites
- Running tests
Architecture
MongoDB Scripts
Support
Database Alternatives
Acknowledgements

Contributing

We welcome all contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas to code open sourced by the Government Technology Agency of Singapore. Contributors will also be asked to sign a Contributor License Agreement (CLA) to ensure that everybody is free to use their contributions.

IMPORTANT NOTE TO ALL CONTRIBUTORS

Before contributing, please read CONTRIBUTING.md. In particular, we strongly encourage contributors to please first discuss the change you wish to make via GitHub issue, email, or any other method with the repository owners beforehand. Otherwise, we may not be able to review or accept your PR.

Features

FormSG is a form builder application built, open sourced and maintained by the Open Government Products team of the Singapore Government Technology Agency to digitise paper processes.

Notable features include:

19 different form field types, including attachments, tables, email and mobile
Verified email and mobile phone fields via integrations with Twilio and AWS SES
Automatic emailing of submissions for forms built with Email Mode
Encryption for data collected on forms built with Storage Mode
(Singapore government agencies only) Citizen authentication with SingPass
(Singapore government agencies only) Citizen authentication with sgID
(Singapore government agencies only) Corporate authentication with CorpPass
(Singapore government agencies only) Automatic prefill of verified data with MyInfo
Webhooks functionality via the official FormSG JavaScript SDK and contributor-supported FormSG Ruby SDK
Variable amount and Itemised payments on forms with stripe integration

Local Development (Docker)

Prerequisites

Install docker and docker-compose.

First Setup

To install the relevant npm packages (frontend, backend and virus-scanner), run the following in the root direcory:

npm install && npm --prefix serverless/virus-scanner install

To prevent breaking changes to webpack4 introduced in node 17 and above, enable the --openssl-legacy-provider flag:

export NODE_OPTIONS=--openssl-legacy-provider

If you are on Mac OS X, you may want to allow Docker to use more RAM (minimum of 4GB) by clicking on the Docker icon on the toolbar, clicking on the "Preferences" menu item, then clicking on the "Resources" link on the left.

Running Locally

Run the following shell commands to build the Docker image from scratch. This will usually take 10 or so minutes. These commands runs the backend services specified under docker-compose.yml and the React frontend on the native host.

npm run build:frontend
npm run dev

After the Docker image has finished building, the React application can be accessed at localhost:3000. The backend API server can be accessed at localhost:5001.

If there are no dependency changes in package.json or changes in the src/app/server.ts file, you can run

docker-compose up

which does not rebuild the Docker image from scratch. This command usually only takes ~15 seconds to finish starting up the image.

Adding dependencies

Run npm install as per usual.

For backend, run

docker-compose up --build --renew-anon-volumes

which will rebuild the backend Docker image and not reuse the existing node_modules volume.

As frontend project is currently not using Docker, no other steps are required.

Accessing email locally

We use MailDev to access emails in the development environment. The MailDev UI can be accessed at localhost:1080 when the Docker container runs.

Environment variables

Docker-compose looks at various places for environment variables to inject into the containers. The following is the order of priority:

Compose file
Shell environment variables
Environment file
Dockerfile

FormSG requires some environment variables to function. More information about the required environment variables are in DEPLOYMENT_SETUP.md.

We provide a .template-env file with the secrets blanked out. You can copy and paste the variables described into a self-created .env file, replacing the required values with your own.

Trouble-shooting

You can consult TROUBLESHOOTING.md for common issues that developers face and how to resolve them.

Testing

The docker environment has not been configured to run tests. Thus, you will need to follow the following local build guide to get tests running locally.

Testing Prerequisites

The team uses macOS for development.

Make you sure have the following node version & package manager on your machine:

"node": ">=18.12.1"
"npm": ">=8.19.2"
"mongo": ">=4.0.0"

Run

nvm install 18
npm install
pip install "localstack[full]"

to install node modules and Localstack locally to be able to run tests. Note that localstack[full] is only compatible with Python 3.7 and above.

Running tests

Unit tests

npm run test

will build the backend and run our backend unit tests. The tests are located at __tests__/unit/backend.

If the backend is already built, you can run

npm run test-ci

Frontend tests are located at frontend/__tests__. They can be run with

npm run test:frontend

End-to-end tests

npm run test:e2e-v2

will build both the frontend and backend then run our end-to-end tests. The tests are located at __tests__/e2e. You will need to stop the Docker dev container to be able to run the end-to-end tests.

If you do not need to rebuild the frontend and backend, you can run

npx playwright test

Cross-browser testing

This project is tested with BrowserStack.

Architecture

The architecture overview is here.

MongoDB Scripts

Scripts for common tasks in MongoDB can be found here.

Support

Please contact FormSG (support@form.gov.sg) for any details.

Database Alternatives

Migrating from MongoDB to FerretDB

FerretDB is an open source MongoDB alternative built on PostgreSQL. MongoDB can be swapped out of FormSG for FerretDB. In order for this to be done, certain changes to the code should be made as described below:

Add postgres to the list of services in the docker.compose file e.g.

  image: postgres:15.3-alpine3.18
  environment:
    - POSTGRES_USER=<pguser>
    - POSTGRES_PASSWORD=<pgpassword>
    - POSTGRES_DB=<pgdbname>
  volumes:
    - pgdata:/var/lib/postgresql/data
  ports:
    - '5432:5432'

In the same file, change the "database" image from MongoDB to FerretDB and update the database section to include the lines below:

 image: ghcr.io/ferretdb/ferretdb:1.17.0
 environment:
   - FERRETDB_TELEMETRY=disable
   - FERRETDB_POSTGRESQL_URL=postgres://pg:5432/formsg?user=<pguser>&password=<pgpassword>
 ports:
   - '8080:8080'
 depends_on:
   - pg

Lastly, add the pgdata volume

    volumes:
        mongodb_data:
            driver: local
        pgdata:

FerretDB currently has some limitations and certain database features are not supported, these include TTL, database transactions and some aggregration pipelines which are all features used by FormSG.

The following changes can be made to mitigate the limitations of FerretDB:
- Add the autoRemove: 'interval' property to the initializing of the session object in the session.ts file.
- Remove the unsupported aggregration pipeline stages e.g. lookup and project, in the submission.server.model.ts file.
- Replace the findOneAndUpdate code block in the user.server.model.ts file with code similar to the one below:
```
 const user = await this.exists({ email: upsertParams.email })
 if (!user) {
  await this.create(upsertParams)
}
return this.findOne({
  email: upsertParams.email,
}).populate({...
```

Migrating from Mongoose ODM to Prisma ORM

FormSG uses Mongoose as the Object-Document Mapping (ODM) to MongoDB. This means that our code is strongly coupled with MongoDB as Mongoose solely supports it.

In order to use a different database with FormSG you will have to first migrate from Mongoose to other object modelling libraries. One of which is Prisma.

Prisma is an Object-Relational Mapping (ORM) library that can also be used as the object model for MongoDB. Prisma is compatible with various other relational databases like Cockroach DB.

Follow this guide by Prisma to migrate from Mongoose.

The guide has 4 primary steps:

Install Prisma CLI
Introspect the current MongoDB for the data model
1. For this section, Prisma’s introspection should be able to create prisma models that will replace your server.model.ts for each collection
2. Additionally, as Prisma is relational, you could add relations between the various documents. One good relation to add will be form many to one user on the [form.email](http://form.email) field.
Install Prisma Client
Replace Mongoose Queries with Prisma Client
1. This step will likely take the most refactoring efforts
2. This will include most files in formsg/src ending with service.ts
3. Including test files ending with service.spec.ts

Replacing MongoDB with CockroachDB

Thereafter, you could set up CockroachDB which is a distributed SQL DB. Follow the quick start guide by CockroachDB to create a CockroachDB Serverless cluster.

To replace the local development instance, you can follow this guide. As FormSG uses Docker for local development, you will have to replace the mongoDB container from docker-compose.yml to the cockroachDB version.

Then connect to CockroachDB by changing the DB url in .env to the one from your CockroachDB DATABASE_URL="YOUR_COCKROACH_DB_URL".

For local development, if the DB is replaced as above, you should not need to modify the ports as it will still be hosted on localhost:27017.

Other Prisma supported DBs

MongoDB can be replaced with other various relational databases supported by Prisma in this list.

Other potential DB migrations

It is also possible to migrate from Mongoose to Ottoman, which is another ODM.

The process will be simpler than migrating to Prisma, but Ottoman is more restrictive and can only be used together with Couchbase, which is also a noSQL DB like MongoDB.

Refer to this guide to migrate from Mongoose to Ottoman and then replace MongoDB with Couchbase.

Acknowledgements

FormSG acknowledges the work done by Arielle Baldwynn to build and maintain TellForm, on which FormSG is based.

Contributions have also been made by:
@RyanAngJY
@jeantanzy
@pregnantboy
@namnguyen08
@zioul123
@JoelWee
@limli
@tankevan

SuriyaRuk/FormSG

Table of Contents