ACI Learning - Degraded www.practice-labs.com performance – Incident details

What went wrong
Users were receiving intermittent timeout issues whilst using the Practice Labs platform before ultimately being unable to login or access the platform.
Who was impacted
During this outage many users who were logged into the Practice Labs platform may have encountered intermittent timeouts followed by a platform outage.
Why it went wrong
We encountered a database index corruption on one of the core tables, slowing traffic down to the point of application queries reaching their timeout limits.
How did we fix it
We validated there were no other corruptions throughout the database, rolled back incomplete data, and rebuilt the affected tables indexes.
The following morning we scheduled emergency maintenance to take the database offline to perform some rigorous maintenance work in an effort to strengthen the platform against this kind of incident in the future.

We are continuing to monitor the platform and will be performing Emergency Scheduled Maintenance:
Emergency Scheduled Maintenance - 14th July 2024 06:00 - 08:00 UTC (Platform downtime) - Maintenance details - ACI Learning - Status
To resolve the degraded performance of the api reporting.

We have implemented a fix and are currently monitoring the results. Initial tests indicate that the platform is functioning as expected, though API reporting will be degraded.