We found that one of the load-balanced machines in our AWS environment had become corrupted, which led to the intermittent issues. The AWS environment has been updated and the issue appears to have been resolved.
In the short term, we implemented a few changes to minimize any chances of recurrence of this problem:
- We increased the authorization token life to 120 seconds.
- We modified the UID validator endpoint so that it will trigger a retry request if authorization fails.
In the medium term, we plan to review the entire authorization system to make it even more robust and scalable.
Posted Dec 20, 2017 - 00:22 PST
Monitoring
The issue was identified as a problem with AWS instances that had been replaced. Sice that, the loading issue stopped in our tests.
Posted Dec 19, 2017 - 02:04 PST
Investigating
We received reports of intermittent authorization issues related to the UID validation. The editor loads normally when the page is refreshed.