Write EVM Endpoint Upgrade Runbook

[kj4ezj]

Last week, the EVM team experienced friction trying to perform software upgrades to EVM endpoints because it was not clear to them how to do this without causing service interruptions while operating within the constraints of the existing cloud network infrastructure.

From eos-evm issue 517:

The Automation team provides services relating to cloud infrastructure, DevOps, CICD, security, and other tasks. Currently, @kj4ezj is the only member of Automation. The EVM team wanted changes made to their AWS account by Automation last week but @kj4ezj was out-of-office for Consensus 2023. The EVM team found a workaround to meet their needs, and it turns out they already had the necessary permissions to make those changes on their own without a workaround.

This issue is to create a runbook explaining exactly how to perform upgrades to the EVM endpoints from start to finish, including but not limited to the "switcharoo" where network infrastructure is pointed to upgraded instances, tested, then out-of-date instances are removed; all without any service interruptions to customers. This ticket includes educating all EVM engineers on the process, and testing it in the testnet environment to verify it works as expected.

See Also

1. evm-public-docs issue 3 - Write EVM Endpoint Upgrade Runbook
   1. eos-evm-internal pull request 17 - Document AWS Regions and AZs
   2. evm-public-docs pull request 5 - Document Network Endpoint Health Checks
   3. evm-public-docs pull request 4 - EVM Endpoint Network Switcharoo Runbook
   4. evm-public-docs pull request 6 - Document Network Endpoint Smoke Tests

[kj4ezj]

Runbook is here.