ISSUE SUMMARY:

There was a temporary degradation on Wednesday, 19th February 2020 which resulted in the inability to access our Cloud Services Portal. There were two separate occurrences recorded on the same day. The first interruption started at 5:31 AM PST and ended at 5:42 AM PST lasting for 11 minutes. The succeeding event triggered at 6:28 AM PST and resolved at 6:40 AM PST lasting for 12 minutes.

We apologize to our customers whose services or businesses may have impacted during this incident. We have taken immediate steps to improve the platform’s performance and availability.

ROOT CAUSE AND REMEDIATION:

Infoblox Cloud CSP platform was affected due to high CPU usage. We immediately increased the number of Identity Service pods at the time and the average CPU utilization dropped, restoring service availability.

We will be increasing the CPU and memory limits in Production as the original values were low. We will also be adding horizontal auto-scaling to bring the pod counts up/down or as needed on demand.