[General Notice] Scheduled Maintenance: Infoblox NIOS-X Server Platform Components
Between November 18 and 21, phased upgrades will be performed on NIOS-X server platform components. These updates will follow the ‘Schedule Software Updates’ configuration set in the Infoblox Portal.
Impact: During this period, NIOS-X servers connected to Infoblox Cloud [https://portal.infoblox.com & https://portal.eu.infoblox.com] may report an unhealthy status a couple of times, each lasting up to 2–3 minutes.
All other services will remain unaffected, and there will be no impact on protocol services.
Note: This update has also been communicated through the corresponding notification https://status.infoblox.com/incidents/f6ztbwhpg378.
Frankfurt Site DNS traffic re-routed
Postmortem
ISSUE SUMMARY
On Tuesday, 12 November 2019, the Frankfurt site was not serving DNS traffic for 1 hour and 10 minutes. It started at 11:41 PM PST and by 12:50 AM PST, 13 November 2019, the traffic to Frankfurt Site was re-routed to the nearest other Sites.
Customers may have experienced issues with DNS resolution and latency.We apologize to our customers whose services or businesses were impacted during this incident, and we are taking immediate steps to improve the platform’s performance and availability.
ROOT CAUSE AND REMEDIATION
Infoblox Cloud platform was affected due to network congestion issues reported by Amazon Web Services in their Frankfurt Availability Zone.In an event of traffic impact, the services are routed to other Sites, in this particular case AWS services were down hence the traffic was not rerouted immediately.
We are planning to put more automation and health checks in place so that there is no impact in similar situation again.