openstreetmap/operations

Equinix Power Maintenance @ AM6 - 25 May 2024 (10 hours)

Closed this issue · 6 comments

Dear Equinix Customers,

DATE: 25-MAY-2024 - 26-MAY-2024

SPAN: 25-MAY-2024 - 26-MAY-2024

LOCAL: SATURDAY, 25 MAY 20:00 - SUNDAY, 26 MAY 06:00
UTC: SATURDAY, 25 MAY 18:00 - SUNDAY, 26 MAY 04:00

IBX: AM6

DESCRIPTION: Please be advised that Equinix and our approved contractor will be performing 5Y preventive maintenance on its main low-voltage distribution panels MDP-D, at our data center.

This maintenance involves switching the primary or redundant power supply(s) to multiple cabinets off for a time frame of 10 hours. After the maintenance is completed the power supply will be switched back on.

MAINTENANCE ACTIVITY: This maintenance will cause a temporary service interruption of one (1) of the power supplies to your cabinet(s) for a maximum duration of ten (10) hours.
During these works only one feed remains available and will be UPS backed.

RECOMMENDED ACTION: It’s important that, prior to the works, you have checked all your cabinets for the presence of single fed equipment, the presence of a correct installed redundant power cable configuration, and the presence of any defective power supply units or other equipment that could jeopardize your processes during the isolation, and that all issues have been resolved or mitigated.

OUTAGE DURATION: 10 Hours

There is also additional maintenance to the UPS @ AM6 earlier on the same day:

LOCAL: MONDAY, 25 MAR 07:00 - MONDAY, 25 MAR 17:00
UTC: MONDAY, 25 MAR 06:00 - MONDAY, 25 MAR 16:00.

"No loss of resilience is expected with N+1 redundancy maintained during this period with no transfer of load being undertaken and all cabinet supplies remaining UPS and Generator backed."

Although this is not expected to cause any power redundancy loss. The power feeds are "at risk".

Let's do a test in April where we shut off that side for a few minutes. This will verify that everything is working so if we find something that doesn't work, we know before it causes a 10 hour outage.

We will do a test outage of 1 side on Friday 17th May 2024, managed via PDU.

I tested a full 4 min power outage on PDU 2/B which is Feed MDP-D (Redundant). All appeared ok.
I adjusted the power thresholds on the switches for warn and overload to match max observed.

Maintenance started over an hour ago. PDU2 is offline as expected. Other than expected PSU alerts all servers appear to functioning correctly.

Maintenance has been completed with no degradation in our service. All alerts have cleared.

Screenshot 2024-05-26 at 08 22 18