Our supplier has provided updates to indicate that the root cause has been resolved and errors are no longer occurring
Posted Dec 02, 2023 - 15:59 GMT
Update
We are continuing to monitor for any further issues.
Posted Dec 01, 2023 - 14:50 GMT
Monitoring
We have received an update from our partner: As of 30 November 2023, to increase storage capacity within the region and mitigate the impact being experienced, engineers continue to replace drives, pulling in inventory from Rackspace data centers globally. As part of this process, engineers are utilizing drives with higher capacity as possible. Given the type of storage configuration in this environment, the drive replacement process necessitates that drives are replaced as a group, rather than individually. As of 21:41 UTC, engineers confirmed they successfully completed a group replacement and are monitoring the environment.
Additionally, engineers have identified some drives, once installed, are not being recognized by the configuration, and must either be re-introduced or replaced once again, adding to the time of recovery. Once properly installed (a process that must be carefully managed by engineers so as not to introduce additional impact) and incorporated into the logical volume, this additional capacity enables installation, deletion, replication and synchronization processes operate in a healthy environment.
While affected customers may continue to experience impact, engineers are observing incremental improvement in the environment (and a reduction in the rate of errors experienced by customers). They continue work to fully remediate the issue in the best possible time.
Posted Dec 01, 2023 - 09:28 GMT
Investigating
We are aware of an issue that is affecting the uploading of files and photos.
This is caused by a technical issue with one of our partners. Their team is working on this as a priority.
We will post further updates here as soon as we have more information.