Xibo Cloud CDN Errors
Closed this issue · 3 comments
Starting around 2023-02-15 19:10 UTC, some customers may have experienced issues with Players downloading content from their CMS instances.
This has been traced to an invalid configuration that was applied. We are taking steps to resolve this now.
Starting around 2023-02-15 19:10 UTC, some customers may have experienced issues with Players downloading content from their CMS instances.
This has been traced to an invalid configuration that was incorrectly applied.
A fix has been deployed to resolve this issue. We are monitoring, but we are seeing affected Players coming up to date as they download content now.
We believe the issue is fully mitigated and are not seeing errors on downloading files from the CDN any longer.
A post-mortem and further information will follow shortly.
Post-mortem
Around around 2023-02-15 19:10 UTC a configuration change was made to the CDN to support a new service for a customer.
Unfortunately a configuration error was made, and our routine monitoring did not pick that up since it would only affect certain requests.
In response to customer support requests, the issue was identified and rectified, and further monitoring showed that had resolved the issue fully.
We have since added additional monitoring which would have caught this edge case earlier and would have prevented disruption to service.
We apologise for the disruption this has caused, and feel confident that the measures we have put in place will prevent a repeat of this issue occurring again in the future.