Cloud Services Portal (CSP) Offline
Incident Report for Infoblox
Postmortem

ISSUE SUMMARY:

There was a temporary degradation on Wednesday, 19th February 2020 which resulted in the inability to access our Cloud Services Portal. There were two separate occurrences recorded on the same day. The first interruption started at 5:31 AM PST and ended at 5:42 AM PST lasting for 11 minutes. The succeeding event triggered at 6:28 AM PST and resolved at 6:40 AM PST lasting for 12 minutes.

We apologize to our customers whose services or businesses may have impacted during this incident. We have taken immediate steps to improve the platform’s performance and availability.

ROOT CAUSE AND REMEDIATION:

Infoblox Cloud CSP platform was affected due to high CPU usage. We immediately increased the number of Identity Service pods at the time and the average CPU utilization dropped, restoring service availability.

We will be increasing the CPU and memory limits in Production as the original values were low. We will also be adding horizontal auto-scaling to bring the pod counts up/down or as needed on demand.

Posted Mar 06, 2020 - 23:39 UTC

Resolved
This incident has been resolved.
Posted Feb 19, 2020 - 18:27 UTC
Update
We are continuing to monitor for any further issues.
Posted Feb 19, 2020 - 13:44 UTC
Monitoring
The issue with Cloud Service Portal(CSP) login has been resolved. We are monitoring the services.
Posted Feb 19, 2020 - 13:44 UTC
Investigating
We are experiencing a temporary outage which may result in inability to access our Cloud Services Portal. We are working to resolve the outage as soon as possible. Thank you for your patience.
Posted Feb 19, 2020 - 13:37 UTC
This incident affected: BloxOne Threat Defense (Business Cloud/Advanced) (Cloud Services Portal (CSP)), BloxOne Threat Defense (Business On-prem) (Cloud Services Portal (CSP)), BloxOne DDI (Cloud Services Portal (CSP)), and BloxOne Threat Defense (Essentials) (Cloud Services Portal (CSP)).