ACI Learning - Notice history

Symptoms
Users integrating via LTI 1.0, 1.1 or 1.2 into api.practice-labs.com may have noticed an increase in loading times, eventually resulting in a HTTP 503 'Service Unavailable' status code when accessing Practice Labs or Assessments.

What Went Wrong
A package called "RestSharp" was identified as not correctly disposing of used TCP connections and potentially leading to a socket exhaustion scenario which eventually meant the process was no longer able to accept incoming connections.

Who Was Impacted
Any customer or users using LTI 1.0, 1.1 or 1.2 integration methods from their LMS.

Why It Went Wrong
Our monitoring platform reported a large increase in established TCP connections on 2 separate occasions leading to the investigations into the cause. However, the issue had already occurred and caused the process to crash.

How We Fixed It
We have removed any references to the "RestSharp" package and replaced with a more robust HTTP client and retry logic to further prevent this issue occurring again.

A potential root cause has been identified within the LTI gateway, a package used in this application called "RestSharp" has been known to excessively consume server resources leading to instability.

This package has since been removed and we are completing further testing in our Staging environment prior to an expected Production release on January 17th 2024.

An automated workaround has been implemented to help remediate potential recurrences of this incident. Further investigations into the root cause are continuing and updates will be provided as more information becomes available.

Please be advised that the memory hardware issue in our ATL storage server has been successfully resolved. This incident is now closed. We appreciate your patience and understanding during this process.

One of our storage servers in ATL is undergoing hardware repairs scheduled for 10 AM EST today. We'll vigilantly monitor progress to ensure it won't impact platform performance or availability. Thank you for your understanding and trust.