Resolved -
Webex Events infrastructure depends on AWS and they had increased delays associating Route 53 Private Hosted Zones to VPC.
Our customers had intermittent issues accessing the Platform and the Web App.
This is their explanation on the status page:
-----------
Oct 24 12:01 PM PDT Beginning around 10:05 AM PDT customers began to see issues with creating new hosted zone to VPC associations for Route 53 Private Hosted Zones. This issue does not affect Public or Private DNS record updates, which are working normally. All DNS queries are also being answered at this time.
This issue is also affecting certain workloads for some other AWS services:
Elastic Kubernetes Service ClusterCreate operations are resulting in errors. There is no impact to existing EKS clusters.
Parallel Computing Service (PCS) CreateCluster operations are resulting in errors. There is no impact to existing PCS clusters.
VPC PrivateLink is experiencing delays for Private DNS enabled interface VPC Endpoints for CreateVpcEndpoint and ModifyVpcEndpoint API calls in all regions. Existing PrivateLink resources are working as normal.
HealthOmics is experiencing increased error rates for ongoing runs for both STATIC and DYNAMIC run storage types.
Engineers are engaged on the issue and are actively working towards mitigation. We will provide another update within 30 minutes.
-----------
Oct 24 12:36 PM PDT We have identified the root cause of the issue causing delays creating new hosted Zone to VPC associations. Engineers are currently engaged and working on multiple mitigation paths. We have confirmed that this issue only affects new hosted zone to VPC associations for Route 53 Private Hosted Zones. This issue does not affect Public or Private DNS record updates, which are working normally, and all DNS queries are being answered at this time.
Elastic Kubernetes Service CreateCluster operations are resulting in errors. There is no impact to existing EKS clusters.
Parallel Computing Service (PCS) CreateCluster operations are resulting in errors. There is no impact to existing PCS clusters.
VPC PrivateLink is experiencing delays for Private DNS enabled interface VPC Endpoints for CreateVpcEndpoint and ModifyVpcEndpoint API calls in all regions. Existing PrivateLink resources are working as normal.
HealthOmics is experiencing increased error rates for ongoing runs for both STATIC and DYNAMIC run storage types.
FSx is experiencing issues affecting the ability to connect to recently-created Amazon FSx file systems. Existing file systems, and customers connecting to file systems using IP addresses directly, are not impacted.
We will provide another update within 30 minutes.
-----------
Oct 24 1:05 PM PDT Work is ongoing to mitigate the issue causing delays creating new hosted Zone to VPC associations. Engineers continue to pursue multiple mitigation paths. We want to reassure customers that this issue does not affect Public or Private DNS record updates, which are working normally, and all DNS queries are being answered at this time.
Elastic Kubernetes Service CreateCluster operations are resulting in errors. There is no impact to existing EKS clusters.
Parallel Computing Service (PCS) CreateCluster operations are resulting in errors. There is no impact to existing PCS clusters.
VPC PrivateLink is experiencing delays for Private DNS enabled interface VPC Endpoints for CreateVpcEndpoint and ModifyVpcEndpoint API calls in all regions. Existing PrivateLink resources are working as normal.
HealthOmics is experiencing increased error rates for ongoing runs for both STATIC and DYNAMIC run storage types.
FSx is experiencing issues affecting the ability to connect to recently-created Amazon FSx file systems. Existing file systems, and customers connecting to file systems using IP addresses directly, are not impacted.
Based on the current status of the event we expect to provide an update within 60 minutes.
-----------
Oct 24 1:45 PM PDT Between 10:05 AM and 1:42 PM PDT we experienced increased delays creating new hosted Zone to VPC associations. Engineers were engaged at 10:09 AM and began investigating. By 10:45 AM root cause was understood and the team set out on parallel mitigation courses. By 1:05 PM we began seeing the first signs of recovery and by 1:42 PM our telemetry indicated we were recovered. We want to reassure customers that this issue did not affect Public or Private DNS record updates, and all DNS queries were being answered throughout the course of the event. During this time, services which create and associate Hosted Zones to VPCs on behalf of customers also experienced impact. Examples of these services have been listed in previous updates to this Health Event.
The issue is resolved and all services are operating normally.
Oct 24, 15:00 EDT
