Intermittent DNS failures when connected to our California and Frankfurt PoPs

Incident Report for Infoblox

Postmortem

Summary

On Sunday, August 30th, from approximately 10:06 AM UTC to 3:37 PM UTC, customers may have experienced intermittent issues resolving recursive DNS queries when pointed to the BloxOne cloud. These issues were caused by an ISP outage and not due to any issues with the Infoblox cloud service.

Details

Infoblox started to receive few monitoring alerts at around 10.06 AM UTC August 30th. Infoblox’s Engineering team immediately began investigating the issue and did notice that there was an outage affecting connectivity across the internet. We did some changes from our end so the impact on our customers is minimal

Infoblox is committed to providing high available services when issues like this happen. We are continuously monitoring and improving our infrastructure to provide the best performance and security to our customers.

Posted Aug 31, 2020 - 17:26 UTC

Resolved

This incident has been resolved as of 3:37 PM UTC, once the internet backbone provider fixed issues from their side

Posted Aug 30, 2020 - 18:22 UTC

Update

Intermittent DNS resolution issues appear to be caused by a major Internet backbone provider outage which has disrupted access to multiple Internet destinations. Our engineers continue to work on this and are closely monitoring the situation

Posted Aug 30, 2020 - 14:45 UTC

Update

We are continuing to monitor for any further issues.

Posted Aug 30, 2020 - 14:38 UTC

Monitoring

We have made some routing changes and the DNS queries are working fine right now. We are further investigating and monitoring the situation

Posted Aug 30, 2020 - 12:00 UTC

Update

We are continuing to investigate this issue.

Posted Aug 30, 2020 - 11:55 UTC

Investigating

We have identified an issue with resolving DNS queries intermittently when connected to our California and Frankfurt PoPs. Our engineers are working to fix the issue.

Posted Aug 30, 2020 - 11:13 UTC

This incident affected: Infoblox Threat Defense (Business Cloud/Advanced) (DNS Resolution - California, US, DNS Resolution - Frankfurt, DE).