Node 14 Update
davidrkupton opened this issue · 10 comments
Upgrade the containers which provide the AWS ECS deployed webapps from Node10 to Node14.
This requires a deploy of each webapp, (more specifically the container in which it is deployed on AWS).
The deployment can be done in sequence, with less critical webapps being done during the day, and those which affect CoB/DoIT staff being done out of hours.
CoB authored code changes have been kept to a minimum, with only the changes necessary to make new versions of old contributed packages function within the existing code-base being applied and deployed.
Below is the list of applications that are affected, and a proposed time. Within each time slot the apps are ranked in increasing criticality:
-- During business hours --
- Internal Slackbot
- NginX proxy
- Commissions App
- Commissions Search
- Permit Finder
-- Out of Hours --
- Payment-webhooks
- Registry Certs
- Group Management
- Access Boston
Dependency
This update to the AWS infrastructure activity must be completed before this activity. That update
- provides additional hard disk volume which is required or else the deployments of containers will fail, and
- provides a change to infrastructure which ensure a smooth and speedy deploy given the additional size of the container images with Node14.
Risk
The risk is minimal. The containers/apps have been tested on staging for some time without issues.
Mitigation:
- The deployment can be rolled back to the previous Node10 version.
@phillipbentonkelly can you pls review this and make sure I have it right? Thanks.
@phillipbentonkelly @davidrkupton Are any of the apps being updated during business hours constituent facing? Though there is minimal risk should move any apps that people touch to after hours just in case?
The thing is that out of hours eg 6pm is often busier than in hours for constituent facing. For example we really don't see a drop off in registry traffic until after 10pm. So unless we are going to schedule for very late, I would just do in business hours. What I usually do is have James draft a service disruption for Boston.gov, and then if we have an issue I put it live and then remove it afterwards - but we can push the Commmisions +permit finder webapps into the eve too, it just makes the eve process longer.
If we were to deploy a non-business face that would look like this
-- During business hours --
- Internal Slackbot
- Payment-webhooks
- NginX proxy
-- Out of Hours --
- Registry Certs
- Group Management
- Access Boston
- Commissions App
- Commissions Search
- Permit Finder
OK @davidrkupton that makes sense. Tagging @juliagtzp
@davidrkupton @phillipbentonkelly
We can keep the original schedule you proposed.
-- During business hours --
Internal Slackbot
NginX proxy
Commissions App
Commissions Search
Permit Finder
-- Out of Hours --
Payment-webhooks
Registry Certs
Group Management
Access Boston
Infrastructure was updated successfully so this task can go ahead as planned.
@phillipbentonkelly The database connection string for commissions app needs to be changed. At the moment the commissions app fails if the DB goes into fail-over mode.
At the moment it is:
COMMISSIONS_DB_SERVER=zpcobsql71.cityhall.boston.cob
I believe it should be:
COMMISSIONS_DB_SERVER=vsql71.cityhall.boston.cob
Can you incorporate this change alongside the node14 update.
At the same time, we should check all DB strings for the webapps to be sure they are pointing at the cluster (vsqlXX) and not the node (zpcobsqlxx).
@duffy-james FYI.
@phillipbentonkelly - I can close this, correct?