ACI Learning - Degraded V4 performance – Incident details

Symptoms

Users accessing the Practice Labs v4 interface would have experienced either slow or long delays in accessing the platform. In some circumstances; due to the platform taking so long to connect, if a customer was being proxied other errors may have appeared. This behaviour was intermittent throughout the day.

What went wrong

Our v4 interface accessed files in a different data centre when logging in. The latency on this link exceeded normal operations and caused these files to be pulled slowly, and in some cases very slowly when accessing the platform. As the latency of the link was fluctuating, this caused the intermittent nature of the problem.

Who was impacted

A small number of customers who use our v4 interface may have experienced issues, although the majority of customers now use the v5 platform. Users may have also experienced this issue when saving their settings in the v5 interface.

Why it went wrong

This went wrong due to a link between our data centres becoming abnormally congested with increased latency.

How did we fix it

We have now moved the files that are in question to the local data centre with them being replicated asynchronously to the previous data centre.

We implemented a fix and are continuing to monitoring the platform.

Next update will be tomorrow 09/14/2023 9am UTC.

We are still investigating and monitoring the V4 platform at this time.
Next update will be tomorrow 09/13/2023 10am BST.