Elevated authentication errors
Incident Report for BEE Plugin
Postmortem

On March 25th, 2019, the BEE Plugin system experienced a rare downtime event. System up-time is of paramount importance to us, and we are very sorry for what happened.

As some of you know, many applications within MailUp Group (BEE's parent company) rely on BEE Plugin - just like you do. When there is any degradation of performance or outage of any kind, our own company is negatively affected, just like you are. So we owe it to you - and to us - to aim for the highest level of availability and performance. And we are very much committed to that.

The March 25th issue occurred within - and was isolated to - legacy infrastructure, which had been scheduled for an upgrade on March 9th. As previously communicated, the March 9th upgrade was successful, but had to be rolled back to assist customers using BEE Plugin authorization tokens in undocumented ways. We decided to roll back because we are committed to supporting each and every BEE Plugin customer with a smooth migration when the system is upgraded.

As Murphy's Law dictates, one of the very portions of the system that was scheduled to be upgraded on March 9 - which relies extensively on Redis for in-memory data storage - failed on March 25. Despite very extensive research, we have so far been unable to exactly pinpoint why Redis became unresponsive within our AWS environment.

This portion of the BEE Plugin system will be replaced with a new-generation infrastructure on April 6. Extra resources have been deployed in the meantime to ensure that no further downtime events occur between now and April 6.

Once again, we sincerely apologize for the downtime.

If you have any questions, certainly reach out.

All the best,

The BEE Team

Posted Apr 02, 2019 - 07:59 PDT

Resolved
The incident has been resolved and all our systems are fully operational.
Posted Mar 26, 2019 - 06:27 PDT
Update
All our systems are operational.
We are continuing to monitor for any further issues.
Posted Mar 26, 2019 - 04:13 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 26, 2019 - 03:42 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted Mar 26, 2019 - 03:36 PDT
Update
Our team is still investigating the origin of the authentication issue.
Posted Mar 26, 2019 - 03:15 PDT
Investigating
We're experiencing an elevated level of authentication errors that causes the editor to stop loading.
Posted Mar 26, 2019 - 02:26 PDT
This incident affected: Authorization.