Service Health
For incidents related to Google Security Products, visit https://status--cloud--google--com.mawarmekar.com/security. For incidents related to Looker (original), visit https://status--cloud--google--com.mawarmekar.com/looker.
Incident affecting AlloyDB for PostgreSQL, Apigee, Cloud Filestore, Cloud Run, Cloud Spanner, Google App Engine, Google BigQuery, Google Cloud Bigtable, Google Cloud Dataflow, Google Cloud SQL, Google Compute Engine, Google Kubernetes Engine, Hybrid Connectivity, Looker (Google Cloud core), Virtual Private Cloud (VPC)
Multiple products in us-central1-b are experiencing network service degradation.
Incident began at 2026-09-01 07:44 and ended at 2026-09-01 11:52 (all times are US/Pacific).
Previously affected location(s)
Iowa (us-central1)
| Date | Time | Description | |
|---|---|---|---|
| | 29 Sep 2026 | 09:09 PDT | Addendum to Incident ReportThis addendum extends the Detailed Description of Impact from the previous message. Though only two clusters in a datacenter in us-central1-b and us-central1-f experienced network isolation, a few regional products with a dependency on the affected datacenter were also affected. Between 07:41 and 11:52 US/Pacific on Tuesday, September 1, 2026, the following products were affected:
Examples of products that are based on Compute Engine VMs include: Apigee, Cloud SQL, AlloyDB for PostgreSQL, Dataflow, Cloud Filestore, and Looker.
|
| | 10 Sep 2026 | 14:20 PDT | Incident ReportSummaryOn Tuesday, 1 September 2026, some customers in us-central1 experienced network service degradation and instance isolation for a duration of 4 hours and 11 minutes. To our customers whose businesses were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. Root CauseThe event was triggered by the inadvertent physical disconnection of network fiber-optic cables during routine hardware maintenance. A technician was performing a scheduled capacity upgrade on data center routers that support a fraction of capacity in the us-central1-b zone and a small fraction of capacity in the us-central1-f zone. During the hands-on maintenance, optical fibers connecting the data center routers to the network fabric were unplugged and the optical transceivers were replaced with a transceiver that supports higher-density connectivity. This transceiver was not compatible with the transceivers still in place in the data center fabric, causing a loss of connectivity for each fiber. The network is designed with redundant routers in separate data center rooms within the same building so that failure of one router does not interrupt service. The maintenance was intended to upgrade one router at a time over several days, with traffic diversions at each step and verification between steps that the network has returned to a fully-connected state. However, a procedural error in the manually orchestrated upgrade process caused the complete list of transceiver replacements across all routers to be issued to the technician without instructions to sequence the work one router at a time. The maintenance workflow also did not include the expected human and software verification steps for detecting unintended disruption. As a result, the technician sequentially unplugged all fiber paths across the affected devices within 13 minutes. The speed and nature of the error prevented warnings of incorrect action from reaching the engineer before connectivity was lost. Additionally, a standing procedure to halt if light is detected on any fiber-optic cable after unplug was not followed. This resulted in compute capacity in the impacted zone being isolated from the network. Customers were unable to reach their virtual machines, and those virtual machines could not establish connections outside their zone. Remediation and PreventionThe issue was detected immediately by automated network loss monitoring systems as well as proactive probes, which rapidly engaged the network engineering and incident response teams. To mitigate the immediate customer impact, engineering teams actively moved traffic away from the impacted infrastructure. Concurrently, hardware operations technicians on site identified the disconnected optical links and physically re-inserted the original optical transceivers. Once the physical links were fully restored, traffic flow rates normalized and the traffic was redirected back to return the capacity to service. Google is committed to preventing a repeat of this issue in the future and is completing the following actions:
Detailed Description of ImpactOn Tuesday, 1 September, from 07:41 to 11:52 US/Pacific, a portion of the us-central1-b and us-central1-f zones experienced severe network degradation and resource isolation.
Affected Services and Features
Customer Impact Customers with resources hosted in the specific affected clusters within us-central1-b and us-central1-f experienced a complete loss of network connectivity starting at 07:41 US/Pacific. During this time, virtual machines and associated services became unreachable from the internet and from other Google Cloud regions, and those internal resources could not initiate outbound connections. The issue was detected almost immediately and engineers took traffic diversion actions at 07:45 and 08:00 US/Pacific to reduce impact to regional products. By 08:50 US/Pacific, the majority of physical connections were restored and traffic began recovering, and the traffic diversion actions were removed at 09:19 US/Pacific. Full recovery across nearly all affected services and long-running operations was confirmed by 11:52 US/Pacific. Because the network isolation was strictly limited to specific clusters within us-central1-b and us-central1-f, multi-zonal deployments correctly utilizing redundancy across other zones in us-central1 were largely able to bypass the physical hardware failure and continue serving traffic. Customers with projects in both us-central1-b and us-central1-f would not have seen multi-zonal impact - at most one of their zones would have capacity in the affected data center. |
| | 3 Sep 2026 | 10:47 PDT | Preliminary Incident ReportWe sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your environment. We are working to address the root cause and prevent this from occurring in the future. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud--google--com.mawarmekar.com/support Date/Time of the Issue (All time US/Pacific)Incident Start: 1 September 2026 07:41 Incident End: 1 September 2026 11:52 Duration: 4 hours, 11 minutes SummaryOn Tuesday, 1 September 2026, some customers in us-central1-b experienced network service degradation and instance isolation for a duration of 4 hours and 11 minutes. To our customers whose businesses were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. Preliminary Root CauseThe immediate technical trigger for this event was the inadvertent physical disconnection of network fiber-optic cables during a routine hardware maintenance procedure. An engineer was performing a scheduled capacity upgrade on datacenter routers that support a fraction of capacity in the us-central1-b zone. The network architecture for each datacenter is designed with redundancy across multiple routing devices. The system is designed to be resilient to all single device or fiber-path failures, and most double- or triple-failures do not affect customer traffic. To ensure this, the devices and fiber paths are physically separated in each datacenter, with diverse power sources. A procedural error meant that the physical maintenance action sequentially unplugged 100% of fiber paths across all devices within 13 minutes. The nature of the error, combined with the speed of the action, prevented warnings of incorrect action reaching the engineer before complete disconnection. This resulted in compute capacity in the impacted zone being isolated from the network. Customers were unable to reach their virtual machines, and those virtual machines could not establish connections outside their zone. RemediationThe issue was detected immediately by automated network loss monitoring systems as well as proactive probes, which rapidly engaged the network engineering and incident response teams. To mitigate the immediate customer impact, engineering teams actively moved traffic away from the impacted infrastructure, automatically shifting compatible workload traffic to healthy capacity elsewhere in the region. Concurrently, hardware operations technicians on site identified the disconnected optical links and physically reseated the fibers. Once the physical links were fully restored, traffic flow rates normalized and the traffic was redirected back to return the capacity to service. Description of ImpactOn Tuesday, 1 September, from 07:41 to 11:52 US/Pacific, a portion of the us-central1-b zone experienced severe network degradation and resource isolation.
Affected Services and Features
|
| | 1 Sep 2026 | 12:12 PDT | Summary Multiple products in us-central1-b experienced network service degradation. Description The issue was affecting a portion of the us-central1-b zone. The remaining zones in the region were unaffected. All the impacted products have recovered. From preliminary analysis, routine network fabric path maintenance triggered unexpected issues in one cluster in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We thank you for your patience while we worked on resolving the issue. Customer Symptoms Customers experienced elevated packet loss and errors. Workaround The issue has been mitigated. |
| | 1 Sep 2026 | 11:36 PDT | Summary Multiple products in us-central1-b are experiencing network service degradation. Description The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered. The impacted products Cloud Run and App Engine are also seeing recovery and the errors have lowered pre-incident levels. Engineering continues to monitor and validate. Routine network fabric path maintenance triggered unexpected issues in two clusters in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We will provide more information by Tuesday, 2026-09-01 12:30 US/Pacific with current details. Customer Symptoms Customers are experiencing elevated packet loss and errors.
Following products have recovered
Workaround
|
| | 1 Sep 2026 | 11:01 PDT | Summary Multiple products in us-central1-b are experiencing network service degradation. Description The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered and the engineering team is still validating a few remaining product impacts. Routine network maintenance triggered issues in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We will provide more information by Tuesday, 2026-09-01 12:30 US/Pacific with current details. Customer Symptoms Customers are experiencing elevated packet loss and errors.
Following products have recovered
Workaround
|
| | 1 Sep 2026 | 10:09 PDT | Summary Multiple products in us-central1-b are experiencing network service degradation. Description The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered and the engineering team is still validating a few remaining product impacts. We will provide more information by Tuesday, 2026-09-01 11:00 US/Pacific with current details. Customer Symptoms Customers are experiencing elevated packet loss and errors.
Following products have recovered
Workaround
|
| | 1 Sep 2026 | 09:19 PDT | Summary Multiple products in us-central1-b are experiencing network service degradation. Description The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Following network-level mitigation efforts, the underlying network infrastructure has fully recovered. Product engineering teams across Cloud are currently validating the recovery status of individual services. We will provide more information by Tuesday, 2026-09-01 10:00 US/Pacific with current details. Customer Symptoms Customers are experiencing elevated packet loss and errors.
Workaround
|
| | 1 Sep 2026 | 09:03 PDT | Summary Multiple products in us-central1-b are experiencing network service degradation. Description We are experiencing a networking issue starting Tuesday, 2026-09-01 07:41 US/Pacific. The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Our network engineering team has put in mitigations and we are seeing services starting to recover. We will provide more information by Tuesday, 2026-09-01 09:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Customer Symptoms Customers are experiencing elevated packet loss and errors.
Workaround
|
- All times are US/Pacific