Rollout 2024-03-27
Closed this issue · 1 comments
premun commented
Purpose
This issue tracks the arcade-services
repository rollout. On top of the Rollout instructions described on the wiki, it provides the person responsible for the rollout checklist of the steps that need to be performed to rollout services in this repository. All relevant information, including the rollout PR, issues encountered during the rollout and steps taken to resolve them should be linked or added to this issue to keep full audit trail of changes rolled out to production.
Process
Build status check (Monday)
- Check the status of the dotnet-arcade-services-weekly pipeline
- Rotate any secrets that need manual rotation
- Check the status of the arcade-services-internal-ci pipeline. Try to fix issues, if any, so that we have a green build before the rollout day.
- Check the
Rollout
column in the Product Construction board - move any issues rolled-out last week intoDone
Rollout preparation (Tuesday)
- Check that the vendor prepared the rollout:
- Thread on the Rollout channel
- Rollout issue in AzDO
- Rollout PR in
arcade-services
- In case there is a problem with the CI build, notify the Rollout channel
- Link the rollout PR to the Rollout PRs section of this issue
- Double-check that the release notes contain all information
- Merge the already prepared rollout PR (
⚠️ DO NOT SQUASH) - Link the rollout build to the Rollout build section of this issue
- Verify that Maestro opened a production => main PR in
arcade-services
with the rollout merge commit (example). There should be no changes in the PR to any files. Do not merge the PR yet. - Ensure the build is green and stops at the
Approval
phase
Rollout day (Wednesday)
- Approve the
Approval
stage of the rollout build (that has been already started the day before) - Monitor the rollout build for failures.
- Note: this Maestro exceptions query might help in diagnosing issues.
- Keep track of any issues encountered during the rollout either directly in this issue, or in a dedicated issue linked to this issue
- Update the rollout stats in the Stats section below. The statistics will be available in Kusto a few minutes after the build was finished
- Notify the Rollout channel
- Merge the production => main Maestro PR in
arcade-services
(⚠️ DO NOT SQUASH) - Move rolled-out issues in the
Rollout
column of the Product Construction board intoDone
. Add a link in to this rollout issue in the comments before closing them (example) - Close this issue with closing comment describing a high-level summary of issues encountered during the rollout
- In case of rollback, uncomment the Rollback section below and follow the steps there
Rollout data
Rollout PRs
- The main PR:
https://github.com/dotnet/arcade-services/pull/3449
Rollout build
- Rollout AzDO build:
https://dev.azure.com/dnceng/internal/_build/results?buildId=2414726
Rollout times
Use the following Kusto query to gather data about rollout times:
- Pre-Approval run time:
01:03
- Post-Approval run time:
01:24
Useful links
premun commented
The rollout finished with no problems