This is the implementation of the Digital Citizenship API, a set of services that enable Public Administrations to deliver modern digital services to citizens.
The functionality exposed by the Digital Citizenship API focus on end-to-end communication between the Public Administrations and the citizens and the delivery of personalized digital services based on the citizen's preferences (e.g. location, language, etc...).
For further details about the Digital Citizenship initiative, checkout the main Digital Citizenship repository.
Public administration agencies receive millions of requests every year from citizens anxious to find out about the progresses of their application or whether a payment has been received. Citizens have to spend time on hold, which wastes their time and costs government a lot of money in running call centres. Moreover, citizens forget about or miss payment deadlines costing them overtime fees.
The messages service makes it easier to keep citizens updated, by helping service teams across public administration agencies to send text messages, emails or letters to the citizens.
Public administration services can send notifications to citizen by calling the messages API from their web applications, back office systems or batch jobs. The messages service provides flexibility and resilience by having a number of SMS, email and post providers. It’s straightforward for us to swap these providers in and out, based on price, performance etc, with no effort or impact on government service teams.
The messages service is for sending transactional messages, not for marketing. There is a risk that marketing messages may be reported as spam, which would affect delivery rates.
Modern digital services are designed for delivering personalized experiences to the users. Today, a citizen that wishes to provide personal information and preferences to the services he uses, has to provide his preferences over and over again to all services; that's because most public digital services don't share any information.
The preferences service makes it easier for the citizen to provide his personal preferences (i.e. contacts information, payment preferences, language, etc...) in a central repository that digital services across public administration can use to provide a more personalized digital experience to citizens.
Public administration services can query a citizen preferences by calling the preferences API from their web applications. The preferences service provides fine control on what preferences attribute a certain application can read or write, making handling user provided information safe and painless.
The preferences service is for delivering personalized digital services, not for collecting citizens emails or mobile numbers. For any transactional communication need, the messages service must be used.
All new accounts on the Digital Citizenship API starts off in trial mode.
This means:
- you can only send messages with email notifications to yourself
- you can only send 50 messages per day
As soon as you’re ready, we can remove these restrictions.
If a notification fails for a certain notification channel because the user has not configured that channel and you haven't provided a default address for that channel, nothing can be done.
Our delivery states are:
- Sending
- Delivered
- Phone number or email address not provided
- Technical failure
All messages start in the Sending state.
This means that we have accepted the message. It’s waiting in a queue to be sent to our email or text message delivery partners.
This means the message is in the person’s email inbox or on his/her phone.
We can’t tell you if they’ve read it — to do so would require invasive and unreliable tracking techniques.
You haven't provided any address to reach the citizen (email phone number), and the citizen you're trying to contact doesn't have any contact preferences in his profile.
This means there is a problem with the connection between the messages API system and our email or text message delivery partners.
Notifications still being retried are marked as Sending. We mark notifications
as Technical failure once we’ve given up.
The design of the system strives to follow the functional paradigm as much as possible.
Most data structures are immutable and the vast majority of data models are versioned and immutably stored in the database.
The business logic is designed to be purely functional and segregated from the non-functional, non-typesafe parts (interface to the Azure Functions framework, the database and the Express framework).
The application is structured in three layers:
| Layer | Responsibilities |
|---|---|
| Presentation | Handling of HTTP requests |
| Domain | Business logic |
| Data Source | Communication with databases, messaging systems and 3rd party APIs |
The presentation layer follows the front controller pattern and handles the HTTP requests for the API resources.
The main API controller is based on Express.js (that handles the routing and the request parsing phases).
The controller relies on shared middlewares for implementing shared functionalities exposed to the domain layer:
| Middleware | Responsibility |
|---|---|
| Azure API Authentication | Extracts and validates authentication information from the request |
| Azure User Attributes | Extracts and validates metadata related to the API user |
| Azure Context | Extracts the Azure Functions context from the request |
| Fiscal Code | Extracts and validates the Fiscal Code parameter of the request |
| Required Parameter | Checks the presence of a certain required parameter in the request and extracts its value |
The middlewares can be configured to be executed before each request and will either pass through the extracted information or stop the processing of the request by returning an error to the API client.
The domain layer follows the service layer pattern and handles the business logic exposed through the API.
The service layer is composed of several specialized controllers that handle operations for specific resources:
| Controller | Responsibilities |
|---|---|
| Messages | Handles operations on the Message resource |
| Profiles | Handles operations on the Profile resource |
| Info | Provides system runtime information |
| OpenAPI | Provides the API specification in OpenAPI format |
The specialized controllers are designed to follow the Zalando RESTful API guidelines.
The data source layer follows the table data gateway pattern and handles the operations on the resources stored in the database.
Operations specific to the database technology have been encapsulated in a gateway object that abstracts the DocumentDb API exposed by CosmosDB. This approach decouples the underlying data store semantic from the application, making possible to migrate the application to a different data store with little effort.
The data model is designed around several common techniques like identity field, foreign key mapping and value object.
The data model is composed of the following core entities:
| Entity | Description |
|---|---|
| Profile | Represents the profile of a citizen (i.e. his preferences) |
| Message | Represents a message sent to a citizen by a service owned by an Organization |
| Notification | Represents a notification to the citizen, triggered by a Message |
| Service | Represents the Service, owned by an Organization, that sends the messages or access citizen's profiles |
The data model defines also several non-core entities used to describe events happening inside the application:
| Entity | Description |
|---|---|
| Created Message Event | Gets triggered by the Messages controller when a new Message gets created |
| Notification Event | Gets triggered for each notification channel when a new Notification gets created |
The following diagram describes the relationship between the above entities:
The system is designed around loosely coupled components that communicate through REST interface or asynchronous messaging queues.
Some components are provided by Azure services while other are custom.
| Component | Responsibilities |
|---|---|
| API Management | API gateway, client and permission management, authentication and pre-processing of requests |
| Functions | Runtime environment for custom application logic |
| CosmosDB | Database for structured data |
| Queue Storage | Messaging infrastructure |
| Blob Storage | Unstructured storage |
| Application Insights | Application instrumentation and analysis |
| Component | Responsibilities |
|---|---|
| Public API controller | Front controller for public APIs |
| CreatedMessage handler | Processor for CreatedMessage events |
| EmailNotification handler | Processor for EmailNotification events |
What follows is a brief description of how requests and events get processed by the components.
- An API client sends a request to the public endpoint of the API.
- The public endpoint forwards the request to the Azure API Management system.
- The Azure API Management system looks up the credentials provided by the client and validates them, it will also lookup the groups associated with the client.
- The Azure API Management system forwards the request to the API Function implementing the REST API, enriching it with authentication data (e.g., the client user ID and the associated groups).
- The API Function processes the requests. Most CRUD requests will need to interact with the data store.
- If the request created a new
Message, a new message event gets pushed to the new messages queue. - A function that maps the new message to the notifications gets triggered for each new event consumed from the new messages queue.
- For each new
Message, the function will lookup the notification preferences for theProfileassociated to the recipient of theMessageand create a pendingNotification. If the user enabled the message inbox, the content of theMessagewill also be persisted and associated to theMessagerecord in a blob container namedmessage-content. - In case one or more notification channels have been configured in the
Profilepreferences, a new notification gets pushed to each configured channel queue (e.g., email, SMS, push notification, etc...). - A function responsible for handling new notification for a specific notification channel gets triggered.
- Each new notification event triggers a call to a channel endpoint
(e.g., an MTA, a 3rd party API, etc...) that will send the content of the
Notificationto the recipient through the channel. - The result of the call is stored in the
Notification.
Authentication of requests is handled by the Azure API Management service.
Currently the system relies on a custom API token for authenticating clients.
The token is transmitted in the HTTP request custom header Ocp-Apim-Subscription-Key
and is tied to one user account belonging to an Organization.
Access rights to the resources is based on
scopes.
Each scope has a corresponding custom group in the Azure API Management service
(e.g., the ProfileRead scope has a corresponding ProfileRead group).
Most resources have read and write scopes (e.g. ProfileRead, ProfileWrite).
API clients can be allowed to any scope by adding the client to the scope's
group in the Azure API Management console (i.e., a client that is part of the
ProfileRead and the ProfileWrite groups will have read and write rights on
the profiles resource).
The currently supported scopes can be found in the Azure API Authentication middleware.
Authorization of API requests can be restricted by client IP.
This access permission can be configured in the Service objects, through the
authorizedCidrs attribute.
This attribute must contain all the allowed CIDRs and IPs.
See the documentation of cidr-matcher
for example values.
If the authorizedCidrs attribute for a service is not set (default), the API
will accept request from any IPs for that Service.
If the authorizedCidrs attribute for a service is set, the API will respond
with a Not Authorized response to requests originating from IPs that don't
match at least one of the provided CIDRs or IPs.
For monitoring and auditing purposes, the system emits application events to the Azure Application Insights service.
Currently the system emits the following events:
api.messages.create: when a message gets created (metadata includessenderOrganizationId,senderUserIdandsuccessstatus).notification.email.delivery: when an email notification gets delivered ( metadata includesaddressSource,messageId,notificationIdandmta).
The API is developed in TypeScript, all code is under the lib directory.
Each Azure Function is declared in host.json (in the functions key) and has
a top level directory associated (e.g., PublicApiV1).
Each function directory contains a function.json that configures the bindings
and the reference to the function entrypoint from lib/index.ts.
See the development documentation to run the application for development.
To gain the most from TypeScript's type safety we rely on compile-time code generation of models from the OpenAPI specs. This is still a work in process but eventually all code for data models of requests and responses will likely be generated from the API specs at compile time.
The OpenAPI specs are located under api.
The code is generated by the script api/generate_models.ts that uses simple
Jinja templates to translate the specs into TypeScript code.
The generator can be executed with:
$ yarn generate:models
// the command will output the models that have been generatedNote: You'll have to install -h ts-node first to make it work.
The generated code will be stored in lib/api.
For improved readability on as many devices as possible, we rely on MJML responsive email framework.
The MJML templates live under templates/mjml.
The MJML templates gets compiled to Typescript code by the Makefile
located in the root folder:
gulp generate:templatesUnit tests gets execute using Jest and are located in the __tests__
sub-directory of the module under test.
The release process is implemented as a gulp task named release.
To cut a new release you simply run:
$ gulp release
[11:16:39] Using gulpfile ~/src/digital-citizenship-functions/gulpfile.js
[11:16:39] Starting 'release'...
...
RELEASE FINISHED SUCCESSFULLY
[11:17:28] Finished 'release' after 49 sThe release task does the following:
- Runs some sanity checks on the repository.
- Runs the unit tests.
- Bumps the version to the next release version and adds a version tag.
- Compiles the Typescript code into Javascript and runs
azure-functions-pack to
create a deployable asset - the result is stored in a version-specific branch
named
funcpack-release-vX.Y.Zthat is also pushed toorigin. - Syncs a remote branch named
funcpack-release-latestto the content of thefuncpack-release-vX.Y.Zbranch. - Bumps the version to the next snapshot version.
Currently the deployment of the Azure Functions is triggerred by the
GitHub continuous deployment
trigger, that is linked to the funcpack-release-latest branch.
Thus, a deployment gets automatically triggered when by the
funcpack-release-latest branch gets updated by the release task.
The rollback process is manual right now, if you want to revert to version
x.y.z you must reset the funcpack-release-latest branch to the release
branch you want to revert to:
git fetch
git checkout -t origin/funcpack-release-latest
git reset --hard origin/funcpack-release-x.y.z
git push -f