ACI Learning - Notice history

Symptoms

Users attempting to use Labs from our CDC datacentre may have noticed a decrease in availability, devices not displaying within the platform when connected or device disconnections.

What Went Wrong

A planned maintenance change overran outside the window as changes to our Lab configurations were being carried out to allow for added redundancy.

Who Was Impacted

Any user accessing the Practice Labs platform trying to access or use some CDC Labs between 7:30 - 12:00 UTC.

Why It Went Wrong

The modification added extra redundancy in our CDC datacentre by splitting all Labs into either a CDC01 or CDC02 location identifier. The Lab changes took longer than expected and overran the maintenance window.

How We Fixed It

The change was completed as soon as possible and tested to ensure that all Labs were available in either half of the CDC datacentre. We will continue to test and monitor the situation throughout the day.

After further investigation we noticed that this issue was also affecting some our CDC lab availability.
This will require a service restart to resolve which will take place at 13:35 UTC.

Symptoms
Users integrating via LTI 1.0, 1.1 or 1.2 into api.practice-labs.com may have noticed an increase in loading times, eventually resulting in a HTTP 503 'Service Unavailable' status code when accessing Practice Labs or Assessments.

What Went Wrong
A package called "RestSharp" was identified as not correctly disposing of used TCP connections and potentially leading to a socket exhaustion scenario which eventually meant the process was no longer able to accept incoming connections.

Who Was Impacted
Any customer or users using LTI 1.0, 1.1 or 1.2 integration methods from their LMS.

Why It Went Wrong
Our monitoring platform reported a large increase in established TCP connections on 2 separate occasions leading to the investigations into the cause. However, the issue had already occurred and caused the process to crash.

How We Fixed It
We have removed any references to the "RestSharp" package and replaced with a more robust HTTP client and retry logic to further prevent this issue occurring again.

A potential root cause has been identified within the LTI gateway, a package used in this application called "RestSharp" has been known to excessively consume server resources leading to instability.

This package has since been removed and we are completing further testing in our Staging environment prior to an expected Production release on January 17th 2024.

An automated workaround has been implemented to help remediate potential recurrences of this incident. Further investigations into the root cause are continuing and updates will be provided as more information becomes available.