ACI Learning - Notice history

Please be advised, the incident Sunday morning UTC where labs were not accessible from either ATL or CDC locations has been successfully resolved on 2024-02-04 11:30am UTC.

What went wrong

An automated maintenance task overran, causing locks on the production database. Subsequently the database failed over and due to large trans log growth, we were unable to recover in a timely fashion.​

Who was impacted

All Users were unable to login to the Practice Labs platform.​ It’s possible that activity from users already logged into the platform may not be complete.​

Why it went wrong

Automated task was introduced to production, executing a trans log clean-up and index defragmentation task. The size of the database caused the index to rebuild and trans log reduction jobs to overrun causing locks in the production database.

How did we fix it

We began a database replication from last known working backup ​and built a physical node with twice the resources whilst new primary is building​.

This incident is now closed. We appreciate your patience and understanding during this process.

Please be advised, the incident where labs are not accessible from either ATL or CDC locations has been resolved.

This incident has been placed into monitoring until further notice.

We have successfully identified issues affecting our services and are currently working on bringing them back online. Our team is diligently working to resolve the situation, and we appreciate your patience during this time.

We will keep you updated on our progress and let you know as soon as the services are fully operational again. Thank you for your understanding.

We are currently engaged in ongoing investigations to determine the underlying cause of the access issues affecting the labs in these locations. Our technical team is currently focused on restoring Practice Labs databases, which appear to be the source of the problem.

We recognise that this situation may be causing inconvenience and disruption, but please rest assured that our engineers are diligently working to resolve it as swiftly as possible.

We will provide regular updates regarding any developments and progress in resolving this issue. Thank you for your patience and understanding.

We are still actively investigating the root cause behind the inability to access the labs from these locations. Our technical team is diligently working to identify and resolve the issue to ensure that normal operations can resume promptly.

I understand that this situation may be causing inconvenience and disruption. Please rest assured that we are treating this matter with the utmost priority and doing everything within our capacity to address it as swiftly as possible.

We will continue to keep you updated on any developments and progress made in resolving this issue.

Symptoms

Users attempting to use Labs from our CDC datacentre may have noticed a decrease in availability, devices not displaying within the platform when connected or device disconnections.

What Went Wrong

A planned maintenance change overran outside the window as changes to our Lab configurations were being carried out to allow for added redundancy.

Who Was Impacted

Any user accessing the Practice Labs platform trying to access or use some CDC Labs between 7:30 - 12:00 UTC.

Why It Went Wrong

The modification added extra redundancy in our CDC datacentre by splitting all Labs into either a CDC01 or CDC02 location identifier. The Lab changes took longer than expected and overran the maintenance window.

How We Fixed It

The change was completed as soon as possible and tested to ensure that all Labs were available in either half of the CDC datacentre. We will continue to test and monitor the situation throughout the day.

After further investigation we noticed that this issue was also affecting some our CDC lab availability.
This will require a service restart to resolve which will take place at 13:35 UTC.