Duration: 17 hours

Affected Components: MTH 365, Plesk, Hosted Apps, VPS', Plesk, cPanel, cPanel

Jun 1, 16:30:00 GMT+0
Identified - Each affected host has been isolated, to avoid network congestion. .

Jun 1, 15:30:00 GMT+0
Investigating - We are currently investigating this incident..

Jun 1, 15:50:15 GMT+0
Identified - A subset of servers have been affected by a DOS, and have been shutdown to prevent a larger issue..

Jun 2, 08:30:00 GMT+0
Resolved - This incident has been resolved..

Duration: 4 days

Affected Components: MTH 365, Plesk, Hosted Apps, VPS', Plesk, cPanel, cPanel

May 24, 00:30:00 GMT+0
Investigating - Our Monitoring Platforms have alerted to a full service outage of our MTH-1 Datacenter..

May 24, 01:30:00 GMT+0
Investigating - Engineers are due on-site by 3am to investigate..

May 24, 03:00:00 GMT+0
Identified - A Power Feed fault has been identified. Remedial works are underway, whilst Emergency Electricians are due by 5am..

May 24, 05:00:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

May 24, 09:30:00 GMT+0
Monitoring - We are seeing the majority of services now responding..

May 24, 11:00:00 GMT+0
Monitoring - A small subset of services are still affected, and remedial works are underway..

May 24, 14:30:00 GMT+0
Resolved - Shared Hosting services are now fully resolved..

Duration: 23 hours

Feb 3, 12:00:00 GMT+0
Identified - A restore is in progress as of 12:00.

Feb 3, 15:30:00 GMT+0
Identified - A status update will be posted at 17:00 confirming restore progress..

Feb 3, 17:30:00 GMT+0
Monitoring - Account restorations are ongoing. Further updates will continue to be posted when they are available..

Feb 4, 09:00:00 GMT+0
Resolved - This incident has been resolved and all services have been restored..

Feb 3, 10:00:00 GMT+0
Investigating - Our monitoring has shown catastrophic failure of one of our Shared Hosting nodes. Emergency migration is automatically taking place to fail-over websites to our active Shared Hosting nodes..

Duration: 5 hours and 38 minutes

Affected Components: Hosted Apps, cPanel, VPS', Plesk, cPanel, Plesk

Nov 20, 09:52:15 GMT+0
Investigating - We are currently investigating this incident..

Nov 20, 10:35:46 GMT+0
Investigating - The Network Connection is intermittent and we are experiencing 50% packet loss..

Nov 20, 12:38:17 GMT+0
Identified - MTH Engineers are working to resolve this. Whilst also raising with upstream providers..

Nov 20, 15:30:00 GMT+0
Resolved - This incident has been resolved..

Nov 20, 14:03:25 GMT+0
Identified - Openreach Network engineers have been engaged for a site-visit.

Nov 20, 14:30:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Duration: 2 hours and 57 minutes

Nov 18, 13:30:00 GMT+0
Monitoring - CloudFlare are continuing working on restoring service for application services customers..

Nov 18, 14:34:00 GMT+0
Monitoring - CloudFlare have deployed a change which has restored dashboard services. We are still working to remediate broad application services impact.

Nov 18, 14:44:46 GMT+0
Resolved - CloudFlare confirms a fix has been implemented, and they believe the incident is now resolved. We are continuing to monitor for errors to ensure all services are back to normal..

Nov 18, 11:48:00 GMT+0
Investigating - Cloudflare is aware of, and investigating an issue which impacts multiple customers: Widespread 500 errors, Cloudflare Dashboard and API also failing. We are working to understand the full impact and mitigate this problem. More updates to follow shortly..

Nov 18, 12:21:00 GMT+0
Monitoring - Please see the below update from Cloudlare: > We are seeing services recover, but customers may continue to observe higher-than-normal error rates as we continue remediation efforts..

Duration: 2 hours and 26 minutes

Affected Components: MTH 365, , Managed IT →

Oct 9, 09:08:50 GMT+0
Investigating - Microsoft are investigating reports of an issue in which users may be unable to access the Microsoft 365 admin center or leverage Microsoft Entra to access other Microsoft 365 services. We're reviewing dependent service pathways to identify our initial investigating actions. Next update by 12:00.

Oct 9, 10:57:07 GMT+0
Identified - Users may be unable to access their cloud PCs via the Windows app web client..

Oct 9, 10:15:00 GMT+0
Identified - Please find the latest update below: > We've identified that an issue within the Azure Front Door (AFD) service is causing intermittent access issues to Microsoft 365 admin portals leading to wider impact. We're reviewing recent changes that may be contributing to the issue and are actively working on mitigation strategies..

Oct 9, 10:51:00 GMT+0
Monitoring - Please find the latest quick update from Microsoft below: > We identified a loss of capacity within the Azure Front Door (AFD) service causing impact to the Microsoft 365 service. We're restoring the service by bringing the AFD instances back online. Current telemetry shows signs of recovery, and we’re closely monitoring the service to ensure full restoration. .

Oct 9, 09:39:00 GMT+0
Investigating - Microsoft have posted: "We're looking at diagnostic data to isolate the cause of the issue. Currently our investigation is focussed on the load balancing infrastructure.".

Oct 9, 11:30:00 GMT+0
Monitoring - Please find the below update: > We've restored approximately 98% of the AFD service. We’re actively monitoring telemetry to confirm full recovery. In parallel, we’re initiating failover for the Microsoft 365 Portal service to accelerate the restoration process..

Oct 9, 11:35:00 GMT+0
Resolved - The failover of the Microsoft 365 portal service is complete, and we've validated that the service has fully recovered. Some of the affected users have also confirmed resolution..

Duration: 5 hours

Affected Components: cPanel, , Plesk, , cPanel, , VPS', Plesk, Hosted Apps, Web Hosting → Server Hosting → Email Hosting →

Sep 29, 23:00:00 GMT+0
Identified - Services are considered at risk during this maintenance window, with 3x 20 minute outages expected..

Sep 29, 23:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 30, 04:00:00 GMT+0
Completed - Maintenance has completed successfully.

Affected Components: , VPS', Hosted Apps, Server Hosting →

Aug 12, 16:11:59 GMT+0
Resolved - Hi there, We have been alerted to a Server Node Failure. Services are failing over as expected and will be back online within 15-20 minutes. Apologies for any inconvenience caused..

Duration: 3 hours

Affected Components: cPanel, Plesk, , Web Hosting →

Jul 16, 14:30:00 GMT+0
Investigating - Our monitoring has shown catastrophic failure of one of our Shared Hosting nodes. Emergency migration is automatically taking place to fail-over websites to our active Shared Hosting nodes..

Jul 16, 17:30:00 GMT+0
Resolved - Emergency Migration was successful to active nodes. The failed node has been taken out of service and will be replaced. All services are showing as up on active Shared Hosting Nodes..

Duration: 30 minutes

Affected Components: Plesk

Jun 2, 15:30:00 GMT+0
Investigating - Our monitoring has shown catastrophic failure of one of our Shared Hosting nodes. Emergency migration is automatically taking place to fail-over websites to our active Shared Hosting nodes..

Jun 2, 16:00:00 GMT+0
Resolved - Emergency Migration was successful to active nodes. The failed node has been taken out of service and will be replaced. All services are showing as up on active Shared Hosting Nodes..

Duration: 1 hour

Affected Components: VPS', Plesk, cPanel, cPanel, Hosted Apps, Plesk

May 22, 16:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 22, 15:00:01 GMT+0
Identified - Maintenance is now in progress.

May 22, 15:00:00 GMT+0
Identified - Hi there, Emergency Maintenance is required within our MTH-1 datacentre for Network Upgrades. Some Hosted Servers are considered At Risk during this window and downtime is expected. Please raise any queries to [support@monkeytreehosting.com](mailto:support@monkeytreehosting.com) with any queries or issues before or after the maintenance window..

Duration: 5 hours

Affected Components: Hosted Apps, VPS'

Mar 21, 23:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 21, 23:00:00 GMT+0
Identified - Hi there, Further Emergency Maintenance is required within our MTH-2 datacentre for proactive remediation to boost infrastructure stability and reliance. Some Hosted Servers are considered At Risk during this window and downtime is expected. Please raise any queries to [support@monkeytreehosting.com](mailto:support@monkeytreehosting.com) with any queries or issues before or after the maintenance window..

Mar 22, 04:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 5 hours

Affected Components: Hosted Apps, VPS', Plesk, cPanel, cPanel, Plesk

Mar 13, 23:00:00 GMT+0
Identified - Hi there, Emergency Maintenance is required within our MTH-2 datacentre for proactive remediation to boost infrastructure stability and reliance. All Shared Web Hosting and Hosted Servers are considered At Risk during this window and downtime is expected. Please raise any queries to [support@monkeytreehosting.com](mailto:support@monkeytreehosting.com) with any queries or issues before or after the maintenance window..

Mar 13, 23:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 14, 04:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 44 minutes

Feb 24, 11:51:19 GMT+0
Monitoring - Note that we are aware of issues with the Virgin Media network at the present time causing issues for some customers of their services. Note that these issues are outside of the control of Monkey Tree Hosting. Customers are advised to contact Virgin Media for updates. [downdetector.co.uk/status/virgin-media](https://downdetector.co.uk/status/virgin-media/).

Feb 24, 12:34:59 GMT+0
Resolved - This appears to be resolved now. Please contact Virgin Media if you require further information..

Duration: 2 hours and 23 minutes

Affected Components: MTH 365

Sep 2, 09:37:28 GMT+0
Investigating - Users in the United Kingdom have reported issues using Microsoft 365 services..

Sep 2, 09:38:20 GMT+0
Identified - Microsoft are reviewing service monitoring telemetry and networking logs to isolate the root cause..

Sep 2, 12:00:00 GMT+0
Resolved - Microsft have determined through service telemetry that the Microsoft 365 service and environment was healthy. An external ISP networking issue may have resulted in access issues or latency. A service incident didn't actually occur and this communication will expire in 24 hours..

Duration: 4 hours and 30 minutes

Affected Components: Hosted Apps, MTH 365, VPS', Plesk, cPanel, cPanel, Plesk

Aug 23, 14:43:00 GMT+0
Investigating - We are currently aware of an issue with one of our datacentre that is affecting different servers that are provided by us. This may affect performance and access to any services, such as websites and remote applications that are currently hosted on these servers. This is currently being investigated and worked on as a top priority with the relevant teams. Any updates received regarding this will be posted here, including further information. Please find the latest update below: **August 23, 2024 15:35 BST** **\[Identified\]** Our engineers continue to investigate the incident and will provide further updates here..

Aug 23, 15:07:33 GMT+0
Investigating - The relevant teams have posted another update regarding this. Please find this below. Once we have further information regarding this, we will post it here: **August 23, 2024 15:55 BST** **\[Identified\]** Engineers are still working to restore connectivity to all customers..

Aug 23, 15:20:07 GMT+0
Identified - This is the newest update provided at 16:15\. The engineers there are currently working to bring these services back online. We expect this to be resolved shortly. Further confirmation will be provided soon: **August 23, 2024 16:15 BST** **\[Identified\]** Engineers are still working on restoring full connectivity services. Customers ports are currently being brought back online in batches..

Aug 23, 15:33:47 GMT+0
Monitoring - The teams that are investigating have now provided this update. Please find it below: **August 23, 2024 16:29 BST** **\[Identified\]** All customer ports on a single upstream router have now been enabled. The team will monitor this for a 15 minute period before bringing up resilient connections in batches..

Aug 23, 15:35:03 GMT+0
Identified - The teams have provided us with another update just now. Please find it below: **August 23, 2024 16:32 BST** **\[Identified\]** Some customers primary links have been shutdown during this monitoring period. Engineers are still working through the connectivity issues.

Aug 23, 15:59:28 GMT+0
Monitoring - Please find the latest update below regarding this: **August 23, 2024 16:43 BST** **\[Identified\]** Customer ports on a single upstream router have now been enabled again. The team will monitor this for a further 15 minute period before bringing up resilient connections in batches..

Aug 23, 18:07:36 GMT+0
Monitoring - The teams investigating have now provided an update regarding this. Please find it below: **August 23, 2024 18:50 BST** **\[Monitoring\]** Resilient ports have been brought back online for most customers We will be monitoring the network over at least the next hour before closing this incident..

Aug 23, 17:08:32 GMT+0
Identified - Please find the latest set of updates below regarding this matter. We thank you for your patience so far: **August 23, 2024 16:56 BST** **\[Identified\]** When bringing up resilient ports, further connectivity issues were seen and these have been now been brought down. A further period of monitoring of no less than 30 minutes will take place before any further changes are made during this fix. **August 23, 2024 17:31 BST** **\[Identified\]** Engineering team are starting to bring resilient ports online in batches now. **August 23, 2024 18:01 BST** **\[Identified\]** The engineering team are still working on bringing back resilient ports in batches..

Aug 23, 19:13:00 GMT+0
Resolved - **August 23, 2024 20:13 BST** **\[Resolved\]** This incident is now closed. Should any customers still be experiencing any connectivity issues, please raise a support ticket via your regular support channels. A RFO will be circulated via the ServerChoice portal as soon as it becomes available..

Aug 23, 17:42:25 GMT+0
Identified - The engineer teams have reported back in to notify us that they are still working on this: **\>August 23, 2024 18:36 BST** **\>\[Identified\]** Engineers are still working on bringing all ports back online in batches. We do apologise for the delays in resolving this on their behalf..

Duration: 8 hours and 49 minutes

Affected Components: MTH 365

Jul 30, 12:24:00 GMT+0
Investigating - Microsoft are investigating a potential issue and checking for impact. An update will be posted within 30 minutes..

Jul 30, 13:14:00 GMT+0
Identified - Microsoft have identified a potential networking issue and are investigating this further..

Jul 30, 13:47:00 GMT+0
Identified - Microsoft have confirmed that the issue is only occurring within a section of their networking infrastructure and doesn't impact all Microsoft 365 services. Microsoft are working on a list of affecting services..

Jul 30, 14:18:34 GMT+0
Identified - These lists will change as we confirm that different services are or aren’t affected. Impacted services include, but are not limited to: \- Microsoft 365 admin center \- Intune \- Entra \- Power Platform Initial indications are that the following services are not impacted: \- SharePoint Online \- OneDrive for Business \- Microsoft Teams \- Exchange Online.

Jul 30, 14:19:04 GMT+0
Identified - Microsoft are analyzing traffic patterns within a section of a networking infrastructure to assist our investigations. Additionally, they are reviewing mitigation options, including potential failovers, to provide relief..

Jul 30, 14:46:00 GMT+0
Monitoring - Microsoft have implemented a networking configuration change, and some Microsoft 365 services have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability, and we're continuing to monitor to ensure full recovery..

Jul 30, 15:51:00 GMT+0
Monitoring - Telemetry shows that the service has remained stable. We're continuing to monitor the service for an extended period of time to confirm resolution..

Jul 30, 16:52:00 GMT+0
Identified - Microsoft have isolated a small amount of network packet drops and residual impact. They are adjusting the networking configuration changes to mitigate impact. Availability for most services remains stable and we continue to monitor to ensure full mitigation..

Jul 30, 18:24:00 GMT+0
Monitoring - As Microsoft investigate reports of specific services and regions that are still experiencing intermittent errors, we believe that our network configuration changes have successfully mitigated the impacts of the usage spike, but that these changes are causing some inadvertent side effects to certain services. Microsoft are updating our mitigation approach to minimize these side effects and applying these changes in phases to ensure no additional impact is observed..

Jul 30, 19:23:00 GMT+0
Monitoring - Microsoft have confirmed that the updated mitigation approach is successfully addressing the remaining impact. We're applying these changes in the final regions and will continue to monitor service health to ensure the issue is fully resolved once completed..

Jul 30, 20:17:00 GMT+0
Monitoring - Microsoft have completed our additional mitigation actions to address the adverse impact. We're now performing final validations to confirm the issue is fully resolved..

Jul 30, 21:13:00 GMT+0
Resolved - Microsoft have confirmed through extending monitoring and internal validations that the issue is now resolved..

Duration: 3 hours and 1 minute

Affected Components: MTH 365

Jun 28, 12:27:17 GMT+0
Identified - From Microsoft 365 Direct: `While we're focused on remediation, users with access to a desktop build that is cached, may not experience the impact. We’re reviewing application logs to identify the root cause and determine a mitigation plan.`.

Jun 28, 12:01:37 GMT+0
Investigating - We have been made aware that some users may be unable to access Microsoft Teams, specifically on the web but we have noticed this on some desktop apps. Once we hear anything from Microsoft we will update you here. The next update is expected by: `Wednesday, June 28, 2023 at 3:00 PM GMT+1`.

Jun 28, 12:54:20 GMT+0
Identified - From Microsoft 365: `We’re reviewing a recent authentication configuration change which may have unexpectedly resulted in impact. We’re determining our next steps to resolve the impact.`.

Jun 28, 13:17:38 GMT+0
Monitoring - From Microsoft 365: `We’ve initiated reverting the configuration change which resulted in impact. Early monitoring of service telemetry indicates we’re seeing availability improving in America, Asia-Pacific and Europe, Middle East and Africa regions.`.

Jun 28, 15:02:51 GMT+0
Resolved - This incident has been resolved..