This issue is now resolved.
Error rates have remained at normal levels following the mitigation. This incident is now resolved.
Error rates and latency have remained at normal levels following the mitigation. This incident is now resolved.
The issue affecting project lifecycle operations, including project creation, pausing, and restoration, has been resolved. We have observed a reduction in project lifecycle job failures following the fix and confirmed that operations have returned to normal.
All affected projects have be fixed. We have not seen recurrence of the issue and we are resolving the incident.
We've rolled out a fix for this issue and re-enabled upgrades in the dashboard. We apologize for the inconvenience.
A fix for this issue has been implemented and our data shows previous impacted users have recovered.
This issue has been resolved and our error rates have returned back to normal.
During this incident users would have observed unauthorized/5xx failures across multiple operations on the project operations and dashboard functions. We have since rolled out a fix and all functionality is restored.