Monkey Tree Hosting - Notice history

August 23, 2024 20:13 BST

[Resolved] This incident is now closed. Should any customers still be experiencing any connectivity issues, please raise a support ticket via your regular support channels. A RFO will be circulated via the ServerChoice portal as soon as it becomes available.

The teams investigating have now provided an update regarding this. Please find it below:

August 23, 2024 18:50 BST

[Monitoring] Resilient ports have been brought back online for most customers We will be monitoring the network over at least the next hour before closing this incident.

The engineer teams have reported back in to notify us that they are still working on this:

>August 23, 2024 18:36 BST

>[Identified] Engineers are still working on bringing all ports back online in batches.

We do apologise for the delays in resolving this on their behalf.

Please find the latest set of updates below regarding this matter. We thank you for your patience so far:

August 23, 2024 16:56 BST

[Identified] When bringing up resilient ports, further connectivity issues were seen and these have been now been brought down. A further period of monitoring of no less than 30 minutes will take place before any further changes are made during this fix.

August 23, 2024 17:31 BST

[Identified] Engineering team are starting to bring resilient ports online in batches now.

August 23, 2024 18:01 BST

[Identified] The engineering team are still working on bringing back resilient ports in batches.

Please find the latest update below regarding this:

August 23, 2024 16:43 BST

[Identified] Customer ports on a single upstream router have now been enabled again. The team will monitor this for a further 15 minute period before bringing up resilient connections in batches.

The teams have provided us with another update just now. Please find it below:

August 23, 2024 16:32 BST
[Identified] Some customers primary links have been shutdown during this monitoring period. Engineers are still working through the connectivity issues

The teams that are investigating have now provided this update. Please find it below:

August 23, 2024 16:29 BST

[Identified] All customer ports on a single upstream router have now been enabled. The team will monitor this for a 15 minute period before bringing up resilient connections in batches.

This is the newest update provided at 16:15. The engineers there are currently working to bring these services back online. We expect this to be resolved shortly. Further confirmation will be provided soon:

August 23, 2024 16:15 BST

[Identified] Engineers are still working on restoring full connectivity services. Customers ports are currently being brought back online in batches.

The relevant teams have posted another update regarding this. Please find this below. Once we have further information regarding this, we will post it here:

August 23, 2024 15:55 BST

[Identified] Engineers are still working to restore connectivity to all customers.

Microsoft have confirmed through extending monitoring and internal validations that the issue is now resolved.

Microsoft have completed our additional mitigation actions to address the adverse impact. We're now performing final validations to confirm the issue is fully resolved.

Microsoft have confirmed that the updated mitigation approach is successfully addressing the remaining impact. We're applying these changes in the final regions and will continue to monitor service health to ensure the issue is fully resolved once completed.

As Microsoft investigate reports of specific services and regions that are still experiencing intermittent errors, we believe that our network configuration changes have successfully mitigated the impacts of the usage spike, but that these changes are causing some inadvertent side effects to certain services. Microsoft are updating our mitigation approach to minimize these side effects and applying these changes in phases to ensure no additional impact is observed.

Microsoft have isolated a small amount of network packet drops and residual impact. They are adjusting the networking configuration changes to mitigate impact. Availability for most services remains stable and we continue to monitor to ensure full mitigation.

Telemetry shows that the service has remained stable. We're continuing to monitor the service for an extended period of time to confirm resolution.

Microsoft have implemented a networking configuration change, and some Microsoft 365 services have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability, and we're continuing to monitor to ensure full recovery.

Microsoft are analyzing traffic patterns within a section of a networking infrastructure to assist our investigations. Additionally, they are reviewing mitigation options, including potential failovers, to provide relief.

These lists will change as we confirm that different services are or aren’t affected.

Impacted services include, but are not limited to:

- Microsoft 365 admin center

- Intune

- Entra

- Power Platform

Initial indications are that the following services are not impacted:

- SharePoint Online

- OneDrive for Business

- Microsoft Teams

- Exchange Online

Microsoft have confirmed that the issue is only occurring within a section of their networking infrastructure and doesn't impact all Microsoft 365 services. Microsoft are working on a list of affecting services.

Microsoft have identified a potential networking issue and are investigating this further.