Tell HN: AWS outage in us-west-2 AND us-east-1
Last ~40 mins or so we've been getting intermittent internal DNS failures in us-west-2 in our Lambda-hosted app. AWS status board now shows "Lambda invoke issue: We are investigating increased invoke error rates in the us-west-2 Region."
arn:aws:health:us-west-2::event/LAMBDA/AWS_LAMBDA_INVOKE_ISSUE/AWS_LAMBDA_INVOKE_ISSUE_ab21a726-de18-5a2f-b2ac-5cefd2794db9 on https://health.aws.amazon.com/health/home
edit: Status page now says "Operational issue - Multiple services (Oregon)" and "Operational issue - Multiple services (N. Virginia)"
We started seeing evidence of high packet loss in us-east-1 at 17:38 GMT. All seems good now.
Last DNS failure (us-west-2) in my logs was 18:52 UTC; something's stabilizing.
Side note: Twitter used to be great for "is it just me?" on this sort of stuff, but their search no longer shows very good "latest" results for search queries. Quite the bummer, it was always comforting to see confirmation.
That’s where I went and I saw your tweet. So thank you!
I feel ya, I use this now https://downdetector.com/status/aws-amazon-web-services/
Snowflake not working well either..
Yeah... very painful!
We're experiencing the issuew. AWS just reported over the phone that Route53 DNS and “quite a few other services” experiencing outages.
I'm seeing issues with SSHing into new ec2 instances and getting those new instances to reach apt to fetch packages. I'm using packer. My last successful build was 4 hours ago. This is in us-west-2. AWS service health dashboard doesn't show anything for me.
EDIT: Health dashboard shows all services as green, not that the dashboard is down.
The only power developers have over companies who lie on status boards is to hit them in their wallets and file a support request for every failure, so they have to triage that instead of just being honest and upfront.
They seem to hide issues from folks they don't think are impacted.
Logged in (1 issue) versus logged out (0 issues) health dashboard: https://imgur.com/a/QRhZMf2
Thanks for the tip... that is mildly infuriating =/
Just for those following along. Even after I login, I still have 0 issues. I'm not sure how AWS determines when to hide a outage from a user.
The issue I'm shown has "Service: Lambda" on it, so I wonder if it's only shown to folks with Lambda functions live. Someone tweeted at me that they're having issues on EKS, and that they use private DNS inside their VPC as I do.
Not sure. We have lambdas in this account, but not that have triggered in a several months.
I'm not sure if we use Private DNS. We have some custom resolvers and rules. I see now they are calling it network issues.
Ooop, just had "Operational issue - Multiple services (Oregon)" show up.
edit: Aaaaand "Operational issue - Multiple services (N. Virginia)" now.
yup, finally seeing it on my health page.