Coediting issues

Duration: 42 minutes

Impact

The Beefree SDK builder was unavailable for users on plans with the co-editing feature. These users experienced an infinite loading loop with a rotating spinner after the editor progress bar and toolbar had loaded, indicating a failure to retrieve templates.

Summary

A failure in the co-editing application led to 42 minutes of editor unavailability. The issue was resolved by restarting the Redis, Lightstreamer, and adapter services.

Root Cause (Under Investigation)

Initial analysis suggests an interruption in traffic flow between the frontend and the co-editing application. A comprehensive log analysis of system, application, and Lightstreamer-specific logs during the incident period ruled out a Lightstreamer license issue as a contributing factor, as no warnings or errors were found. Further investigation is ongoing.

Timeline

19:30 UTC: Co-editing application ceased receiving traffic, leading to editor loading issues.

19:35 UTC: The Beefree team reported the issue due to multiple user complaints.

19:50 UTC: Analysis identified that Redis and workers were not processing traffic.

20:06 UTC: Redis was manually restarted (without immediate impact).

20:12 UTC: Lightstreamer and adapter services were restarted, fully resolving the issue.

Next Steps

• To quickly resolve the issue, we restarted all co-editing-related services.
• After the situation was stabilized, a series of tests was performed to simulate the issue and better understand the root cause. Some of them are still in progress.
• Monitoring was improved by adding new dashboards for monitoring hardware resources used by the containers and the license status.