ISSUE IDENTIFIED // Some websites are currently offline
Incident Report for Citrus-Lime Service Status
Postmortem

At approximately 13:45, one of our Hyper-V hosts suffered a software crash, which caused the machine to 'blue screen'. This caused all the virtual machines (VMs) running on this host to power off, causing the loss of some Ecommerce sites.

Immediate steps were taken to restart the Hyper-V host and the machine came back, but then crashed again with the same issue.

We then took the decision to invoke our recovery process and start to boot up replicas of the affected VMs on other Hyper-V hosts. Some of these replicas took additional work to get them running, but all VMs were available again by 16:00.

The failure of the Hyper-V host was unexpected. We have established that it was a component of Hyper-V that crashed, but we will be undertaking an investigation to determine the root cause and any necessary follow-up actions.

Posted Jan 24, 2023 - 17:52 GMT

Resolved
This issue has now been resolved and all of our Ecommerce sites are online.

A full issue report will be added to this page in due course.
Posted Jan 24, 2023 - 15:49 GMT
Update
We are continuing to work on a fix for this issue. This is our highest priority.

Please subscribe to the Status Page to receive updates via email.
Posted Jan 24, 2023 - 15:07 GMT
Identified
We are aware of an issue affecting websites on two of our servers.

Our Infrastructure Team have identified the cause and are already applying the fix. Further updates will follow as soon as they are available.
Posted Jan 24, 2023 - 13:52 GMT
This incident affected: eCommerce Platform - Standard.