/opencrvs-core

The open source software product that supports Civil Registrations and Vital Statistics (CRVS)

Primary LanguageTypeScriptOtherNOASSERTION

OpenCRVS

A global solution to civil registration
Report an issue · Case studies · Implementations · About us

Build Status codecov License: MPL 2.0

OpenCRVS

We are on a mission to ensure that every individual on the planet is recognised, protected and provided for from birth.

OpenCRVS (Civil registration & Vital Statistics) is a scalable, offline/low connectivity supporting, full stack, microservice, HL7 - FHIR standardised private/public cloud solution for the civil registration* needs of a country.


What is Civil Registration?

Civil registration is the system by which a government records the vital events (births, marriages, and deaths) of its citizens and residents. The resulting repository or database has different names in different countries. It can be called a civil registry.

Given the realities of population migration and covering the international scope of vital events affecting a human lifetime, a civil registration recordset is the single source of truth of a country's population data.

OpenCRVS supports any country scale data storage regulatory requirements.


Why OpenCRVS?

Plan International is challenging the current gap in the market for a user-centric and rights-based CRVS system by leading the development of OpenCRVS, an open-source digital CRVS solution that is free to use, adaptable to the country context, interoperable with other government systems (e.g. health and ID systems), and rights-based to ensure it protects and provides for those most vulnerable.

An estimated 1 billion people around the world cannot officially prove their identity, and 47% of these are children. World Bank, 2018

For people to count, they must first be counted, and that’s what a Civil Registration and Vital Statistics (CRVS) system does, recording the details of all major life events, such as births and deaths. Birth registration is the first step in securing legal identity and accessing other basic rights like education, healthcare and social protection.

As the sole continuous source of population data, it provides the foundation for human rights, government service delivery, and the measurement of U.N. development goals. Despite this, over 100 countries around the world do not have functioning CRVS systems.

OpenCRVS has the potential to go well beyond the scope of traditional CRVS systems. We see OpenCRVS as a foundational identity and population data system which will support and maintain the integrity of many other service delivery, statistical and identity management functions. In particular, OpenCRVS will ensure that these functions are above all inclusive, providing a digital response to the global call to “Leave No One Behind”.

We believe in the open source principle and its potential to achieve universal civil registration and advance children’s rights, and equality for girls.

From the outset we have been committed to the creation of a digital public good that is:

  • Freely available with no license fees or ties to software vendors.

  • Safe and secure using best-practice security features.

  • Fully interoperable with other government systems.

  • Data enabled for fast decision-making.

  • Based on international CRVS standards.

  • Easily configured and deployed in all country contexts.

  • Rights-based, empowering all individuals to access their basic human rights.

  • User friendly, designed by and for the people it serves.

  • Accessible and inclusive

To create OpenCRVS we have partnered with registration authorities, leading health system providers, human-centered design & software development experts, urban and remote, rural communities in developing nations to design and build a global digital product that will serve the needs of end users and those being registered. We have achieved this by following international standards, digital principles for development and agile methodologies.


How secure is OpenCRVS?

OpenCRVS has been independently security tested to equivalent U.K. Government I.T. standards by the CREST certified organisation MDSec. - authors of "The Hacker's Handbook".


What is in the OpenCRVS Core repository?

The core of OpenCRVS is a monorepo organised using Lerna package. Each package is covered by Jest unit tests. Every package is independently scalable in docker containers, and deployed in an automatically "round robin" load balanced cluster using Docker Swarm.

Docker swarm was chosen for simple container management in a private or public cloud.

We wanted to democratise the containerisation benefits of AWS/Kubernetes style public cloud deployments for developing nations.

Some nations may be located far from a developed world datacentre. Many nations may not be able to legally support international data storage of citizen data, and they may not have access to the development skills necessary to manage a complicated distributed cloud deployment.

Docker Swarm makes it easy to commence service distribution privately and then migrate publically when an organisation is ready to do so.


What are the key OpenSource dependencies of OpenCRVS?

The following dependencies are automatically provisioned alongside the OpenCRVS Core on your infrastructure in docker containers.


Hearth MongoDB Database layer

In order to support endless configuration for limitless country scale, OpenCRVS was designed for NoSQL, built on MongoDB, and aligned to a globally recognised healthcare standard.

Massively scalable and extensible, Hearth is an OpenSource NoSQL database server built by the OpenCRVS founding member Jembi Health Systems, using interoperable Health Level 7 FHIR v4 (ANSI Accredited, Fast Healthcare Interoperability Resources) as standard.

We innovatively extended FHIR to support the civil registration context. Our civil registration FHIR standard can be contributed to in this repository at Jembi.


ElasticSearch

De-duplication management to ensure data integrity is essential to any respectable civil registration system. A fast search engine lowers operational costs and improves the user experience for frontline staff.

OpenCRVS uses ElasticSearch, an industry standard, NoSQL document orientated, real-time de-duplication & search engine. Lightning fast, intelligent civil registration record returns are possible, even with imprecise, “fuzzy” search parameters.


InfluxData

Hyper efficient and optimised, Influx is a time series database for big data insights. Millisecond level query times facilitate civil registration statistical queries over months of data, disaggregated by gender, location and configurable operational and statistical parameters.


OpenHIM enterprise service bus, interoperability Layer

The OpenHIM (Health Information Mediator) is a NodeJS enterprise service bus designed to ease interoperability between OpenCRVS and external systems such as Health & National ID. It provides external access to the system via secure APIs. OpenHIM channels and governs internal transactions, routing, orchestrating and translating requests into FHIR between services and the database layer.


OpenCRVS microservice business layer packages

The OpenCRVS’ back end microservice architecture enables continuous evolution of its business requirements.

The microservices are written in TypeScript (a strictly typed superset of JavaScript that compiles to JavaScript) and NodeJS using the fully documented HapiJS framework.

Each microservice in OpenCRVS has no knowledge of other services or business requirements in the application, and each exposes it’s capabilities via JWT secured APIs.

Microservices:

  1. auth - the authentication microservice for OpenCRVS, JWT token generation and management in Redis.

⋅⋅⋅Our client applications are protected by SMS 2-Factor Authentication. Our apps and microservices utilise OAuth best practices for JWT tokens.

  1. commons - a shared library package for Node methods used by multiple microservices.

  2. gateway - the GraphQL and Apollo API gateway for the OpenCRVS client.


Using GraphQL allows OpenCRVS to perform much faster and more responsively in remote areas by drastically reducing the number of HTTP requests that are required to render a view in the presentation layer.⋅⋅

The OpenCRVS GraphQL Gateway is a JWT protected Apollo server that requests and resolves FHIR resources from Hearth via OpenHIM into GraphQL, for easy consumption in the client applications.

  • metrics - the civil registration metrics and analytics microservice using the Influx time series database.

  • notification - the microservice that manages SMS communications from OpenCRVS, including content management and SMS supplier details.

  • search - the search microservice for OpenCRVS using ElasticSearch

  • user-mgnt - the user management microservice for the OpenCRVS client. User permissions and roles can be centrally managed, supporting IT organisations that conform to ISO27001 certification.

  • workflow - the OpenCRVS business process orchestration microservice, mediating civil registration vital event status and audit updates.


OpenCRVS client application packages

  • login - the login UI client built in React.

  • client - the OpenCRVS UI client for civil registration built in React.

Client npm dependencies and enablers include:


Using an Android progressive web application for our client app means that we can take advantage of offline functionality and native mobile features using Workbox, without the TCO overhead of maintaining multiple web and mobile codebases and respective App/Play Store releases.

In remote areas, registrars can save a configurable number of registrations offline on their mobile phone using IndexedDB.


OpenCRVS component library package

components - a UI component library package for the clients using React Styleguidist.


OpenCRVS end-to-end and performance testing packages

  • e2e - automated end to end testing for OpenCRVS using Cypress.
  • integration - performance tests for OpenCRVS using the K6 framework.

How do I set up a development environment?

  1. First, make sure your environment is prepared by installing these pre-requisites:

    On Linux you will also need to:

    • increase your file watch limit using: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
    • increase vm max heap for ElasticSearch using: echo vm.max_map_count=262144 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

    On Mac you will need:

  2. Next, clone this repo! :) git clone https://github.com/opencrvs/opencrvs-core.git

  3. cd into the repo and run chmod 775 data/elasticsearch from root of the project to allow ElasticSearch access

  4. Run yarn to install deps

  5. Run docker swarm init - your localhost has to be a Docker swarm manager in order to use the "overlay" network.


How do I configure OpenCRVS?

OpenCRVS depends upon your own country specific "resources" module of reference data (addresses, employees, offices, facilities), custom configuration of vital event registration forms and your own integrations.

A test "resources" module for Zambia is released for free in order to illustrate how this can be done. It contains documentation of how you would configure and deploy your own integrations, customise birth and death registration forms, add your own country specific address structure, and how you would populate OpenCRVS with FHIR standardised reference data.

  1. cd ../ back up and clone this example Zambia resources module. You can iterate upon it for your needs.

  2. git clone https://github.com/opencrvs/opencrvs-zambia.git

    You will likely want to store your custom resources module in a private repo.

    We have a lot of experience of customising OpenCRVS resources to any scale now that OpenCRVS has been deployed in Bangladesh, so if you need system integration help, please get in touch.

  3. cd opencrvs-zambia into the resources repo and run yarn to install deps

  4. Run yarn start to run the resources module

  5. cd ../opencrvs-core to return to the core module.

  6. Run yarn dev to up the OpenCRVS core dev environment (frontend and backend services in this repo start as local dev servers that will autoreload and dependencies are started via docker-compose) OR alternatively you may run the dependencies and the services in this repo separated in two diffrent terminal with yarn compose:deps (dependencies) and yarn start (services in this repo). We also provide tmux commands.


Starting the dev environment (necessary for OSX):

Docker For Mac can affect OpenCRVS ability to find containers on localhost. Find your local IP address and start the dev environment, replacing the IP address with yours, like this: LOCAL_IP=192.168.0.5 yarn dev

  1. Populate the database with test data:**

  2. Once your environment is running, you will now need to populate your database with test data using the Zambia implementation.cd ../opencrvs-zambia to return to the Zambia resources module

  3. Run yarn db:backup:restore to populate the database with reference data and test users.

That's it! You should be running OpenCRVS with test users and test locations. Apps can be found running at the following URLs:

Login using using the following Zambia user accounts.

  • Field agent: kalusha.bwalya
  • Registration Agent: felix.katongo
  • Registrar: kennedy.mweene
  • System Administrator: emmanuel.mayuka

In development, the password is "test" and the 2-factor auth SMS code is "000000".

Please get in touch with us to find out more about the available business functionality for each user type.

After logging in you should be redirected to: Client: http://localhost:3000/

UI component library will be running here: Styleguide: http://localhost:6060/


How can I test OpenCRVS locally on an Android device?

You can run a PWA on an Android device and connect to OpenCRVS running on your local network. This is the best way to work and debug service workers and do actual device testing.

  1. Go to ngrok.conf and add an auth token. Register to https://ngrok.com/. The free plan is enough for this.

  2. Run ngrok start -config infrastructure/ngrok.conf --all. Go to https://dashboard.ngrok.com/auth and get yourself an auth token

  3. Open https://ocrvs-client.ngrok.io on your device. Create a .env file to packages/mobile-proxy

AUTH_TOKEN=THE_AUTH_TOKEN_YOU_GOT_HERE

  1. Start the development environment as you normally would, but add --ignore @opencrvs/client to the start command. So yarn start:bgd --ignore @opencrvs/client

  2. Run yarn start in the mobile-proxy package

  3. You should now have an ngrok url that can be used remotely on any device. It's still required to be in the same network as the host machine, as some services (login) aren't used through ngrok.

  4. Build the client app NODE_ENV=production yarn build

  5. Serve the client up in 3000 port by running yarn serve

  6. You can now open up the address you got from the mobile proxy in your mobile browser


How can I set up continuous integration and delivery?

We provide an example Travis configuration to automate unit and end-to-end testing integration & deployment.


How can I install and manage an OpenCRVS server cluster?

OpenCRVS should be deployed on a minimum cluster of 3 nodes, each with the following minimm specification:

8 GB Memory (preferrably 16 GB) / 160 GB Disk / Ubuntu 18.04.3 (LTS) x64

To prepare your server cluster and manage the Docker Swarm, some pre-requisites and instructions are documented here

An Ansible playbook script is provided here to automate the vast majority of your server cluster setup.


Clearing and creating your own reference data

  1. cd ../opencrvs-zambia to return to the Zambia resources module

  2. Run yarn db:clear:all to delete the entire local database

  3. Log into the OpenHIM at here to load the initial base config - default password is root@openhim.org:openhim-password (login will fail a security check as we are using self signed certs by default, follow the instructions in the error message so that Chrome recognises the SSL cert.)

  4. Once logged in click Export/Import then drop the file infrastructure/openhim-base-config.json into the import box and click 'Import'

  5. Click Channels and check all have loaded successfully.

  6. Click edit, and then go to routes tab and change the value of host from service name to your local IP address.

  7. Test the setup with curl http://localhost:5001/fhir/Patient/123 you should get some JSON with a 'Not found' error.


Create a new metadata db dump

Start the development environment as described above, then:

  1. Start the dev environment - as explained above.

  2. Code reference data for your country requirements following instructions inside the README of your resources package. Then populate the database like this: cd packages/resource && yarn populate:<<insert alpha3 country code>> && cd ../..

  3. Create new backup zip files to act as your baseline. cd packages/resources && yarn db:backup:create <<insert country code>>

  4. Commit and push the new db dump archive files that have been created in your country folder in resources package to your private repo.


Docker scripts

There are a number of docker scripts available via yarn. yarn dev is the easiest command to run to get started (see above) but if you need to manage the docker containers some of these scripts may be useful.

The yarn compose:* scripts only setup the dependencies in docker containers and not the applications in this repository.

  • *:build scripts which just build the images

  • *:up scripts which just run pre-build images in containers

  • *:down scripts which stop and remove the containers (along with data not stored in a volume!)

A number of other useful Docker commands you will need to manage the swarm are accessible here


How can I deploy to a staging environment cluster?

To deploy to a staging environment we combine docker-compose files that are used in the docker setup above with a few others to configure the stack.

The deployment uses Docker Swarm and sets up an OpenCRVS stack containing each service with a number of replicas defined in the docker compose files. Note: This deployment is currently automated so that every time we push to master the build will be deployed during the CI process.

The deploy is easily executed by just running: yarn deploy:staging --clear-data=yes --restore-metadata=yes <<insert host>> <<insert version>> - you will need ssh access to the server for this to work.


How can I deploy to a QA environment cluster?

Deploying to QA is much the same as above, however you may specify a version to deploy. The version can be any docker image tag. Each time master is build on CI docker images are created for that commit hash. Any of these hashes may be used as the version. In addition any time a git tag is created and pushed all the docker images will automatically build. Once complete the name of this tag can be used to deploy to the QA environment as well.

yarn deploy:qa --clear-data=yes --restore-metadata=yes <<insert host>> <<insert version>>


How can I deploy to production?

Deploying to Production is much the same as deploying to QA.


How do I export recent registrations?

You can export registrations (both death and birth) from the last 7 days by running a script from infrastructure/export-last-7-days.sh <<insert user>> <<insert user password>>.

Would create a new .zip file export.zip with 2 CSV files inside of it. You can tweak the time period from inside the script if needed.


How does OpenCRVS back up registration data?

OpenCRVS backs up all of its data by default and the Ansible playbook script here allows you to configure a remote server in which all data will be zipped and copied to on a nightly cron job. Follow the instructions here


How do I configure OpenCRVS for my country needs?

Just follow the instructions in the Zambia resources package or please get in touch with us for help. We'd love to hear from you!


Meet the people we are building OpenCRVS for

Samira and Anir, Parents

Registering their daughter’s birth is now affordable because they no longer have to skip days of work to make multiple visits to the registration office. The Health Worker comes to them to start the registration process and they can track progress through their mobile phone so they know when and where to collect the birth certificate.

Sophia, Community Health Worker

A community health worker (field agent) can easily register the births for all pregnant and lactating mothers in her remote, rural catchment area and provide quality healthcare. This is now possible because she isn’t spending the majority of the consultation filling out paperwork aswell as administering vaccinations, whilst worrying about connectivity.

Raman, Civil Registrar

Civil registrars can delegate responsibility to registration agents to validate documentation and perform data entry tasks. Registrars can approve applications easily previewing documentation alongside the data and generate PDF certificates instantly. They can view potential duplicates in real-time.

Performance management reports can be auto-generated, presenting real-time disaggregated registration rates to government policy makers. This leads to improved national decision making amd planning, saving time and money.


Become part of the OpenCRVS Community

We want to see OpenCRVS implemented across the world. We can’t do this alone. Through the OpenCRVS Community, we are uniting experts in civil registration and other interested parties.

Join the community


Contributing

You may view/add issues here: https://github.com/opencrvs/opencrvs-core/issues

To contribute code, please review the CONTRIBUTING.md file https://github.com/opencrvs/opencrvs-core/blob/master/CONTRIBUTING.md, fork the repository and submit a pull request. The authors will review the code and merge it in if all is well.

By contributing to the OpenCRVS code, you are conforming to the terms of the license below.


How to tag an OpenCRVS release?

So you have contributed to core and want to make a new release as an OpenCRVS core repo admin?

  1. Update all packages with the new version number according to semver. All packages will have the same version for simplicity as they are all designed to be used together. Update all dependencies to point to the newly created versions. E.g. client depend on components, so update the dependency: Do a find and replace for 1.0.0-alpha.2 and replace with 1.0.0-alpha.3

  2. Run yarn to ensure there are no version errors.

  3. Run yarn test and ensure all passed.

  4. Run git tag v<version_number> e.g. git tag v1.0.0-alpha.2.0

  5. Run git push origin v<version_number>

  6. Create a new release on Github using the tag you just pushed and including any release notes.

  7. Dockerhub should automatically build the images when a new release tag is created in Git. Howver Dockerhub can sometimes timeout and you may need to compose and push the release tagged images locally. To do that, run yarn compose:push:release


License

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this

file, You can obtain one at https://mozilla.org/MPL/2.0/.

OpenCRVS is also distributed under the terms of the Civil Registration & Healthcare Disclaimer located at http://opencrvs.org/license.

Copyright (C) The OpenCRVS Authors. OpenCRVS and the OpenCRVS graphic logo are (registered/a) trademark(s) of Plan International.