Technology Explorations in Data & AI

In this special episode on the AWS outage, Stijn De Haes explains what happened during the AWS October 2025 Outage. He then zooms in on the limited effect it had on Dataminded and its product Conveyor. Finally, Stijn gives 4 tips on how to protect yourself from this kind of outage.

Resources:

Chapters:
  • (00:00) - Intro
  • (00:33) - What happened in the AWS outage?
  • (05:04) - How the outage affected Dataminded
  • (08:31) - 4 mitigations to protect yourself
  • (11:53) - Tabletop exercises for preparedness
  • (13:51) - Summary

Data & AI: Technology Explorations is a biweekly show from Dataminded. Each episode a Dataminded engineer demos a tool or technique worth knowing about -- working code, honest takes, no hype.

Music by Aleksandr Karabanov from Pixabay

Creators and Guests

Host
Jonny Daenen
Head of Knowledge at Dataminded
Guest
Stijn De Haes
Partner at Dataminded

What is Technology Explorations in Data & AI?

Deep dives and practical demos on the technologies shaping modern data and AI development. Join the Dataminded team as we explore, unbox, and critically review the latest tools, from building AI agents and RAG systems to optimizing cloud costs and accelerating data pipelines. We cut through the hype to show you what actually works in real data engineering practice, complete with demo code!

Hi everyone. This week you might have noticed your favorite apps, Outlook, Strava, Steam, Slack, being out for some time. This was caused by an AWS outage. Today, we'll have a look at what happened, how did it affect us as Dataminded, and what can you do to prevent all of this. And with me to explain all that is Stijn De Haes, our technical lead at Conveyor. Welcome, Stijn. Hello. How are you doing? Better than Monday. Yeah, busy week?

Well, it's okay. We had mostly under control. Today, we'll discuss three major questions like what happened at AWS, how did it hit us, and what can people do to prevent such an outage or to at least mitigate it? So what happened at AWS is they messed something up. As we engineers, we push things through production, right? And sometimes you make a mistake. So they made a small mistake with a DNS configuration. And then they made a mistake with a DNS configuration. And then they made a small mistake with a DynamoDB endpoint. And apparently, almost all of AWS has some connection with DynamoDB.

You couldn't launch virtual machines in the North Virginia region. You couldn't do anything with IAM globally. You couldn't update your IAM roles, etc. So a lot of people experienced some hardship this Monday. Yeah. And how did it affect us? Mostly, we noticed it. Through our alerting system. We couldn't pull any new images on our Azure cluster.

And then we also, some of our customers contacted us. We can't do Conveyor builds. So they couldn't deploy new packages on Conveyor. Everything on AWS was actually running smoothly for us. But it's just they couldn't deploy new projects at that point. Yeah. You mentioned already IAM failing in Virginia. What actually happened there? So IAM is Identity and Access Management. So it identifies virtual machines as having access to certain actions, right?

So it gives you the power to interact with, I don't know, SQS and SQS messages to interact with S3. So your virtual machines or your ECS containers or your EKS containers, they need rights to do certain things in the AWS client. IAM is responsible for that. It has two parts.

It has the configuration part where you configure which role can do which actions. So that's the configuring your IAM roles and policies. And it has a second part when your service is running, it fetches credentials so you can authenticate to do those actions with S3. So the global IAM configuration was completely down because that's hosted in the North Virginia region, in AWS.

However, getting credentials and doing actions, there are regional endpoints for that. However, you have to either use an SDK that had a major version, I think around July 2022 or 2023. From that point on, every SDK who had a new major version then used those regional endpoints by default.

But all other older SDKs still use the old North Virginia endpoint. Yeah. So you have to configure them to just use the regional endpoints. And that way you can't do updates to your IAM roles, but your services can still use other AWS services, right? So essentially IAM is hosted in Virginia, and this is for the configuration of your roles.

So let's say I have a data job, it gets a specific role, which it can use to contact other services. And that configuration is part of IAM. My data job has the rights to this path on S3, or this bucket. Yes, correct.

And the second thing you explained, whenever I do those actions and my role needs to get some tokens or credentials, this is fetched in a different way. This is not depending on this Virginia. So from North Virginia, this configuration is synced to all AWS data centers across the world.

So this is globally synced. So updates that you were doing, you couldn't do updates, but even if you could, they couldn't be synced. So you're not syncing to the regional locations, but they still have the latest update and rights that you have.

Okay. And this is why it was still functioning. But then if you would change your rights in IAM, that would not be propagated or not happen even. Correct. Correct. Yeah. Okay. I see. And the reason for this scale is, as I understand, many of AWS services, of course, rely on access management, all of them.

And so you could not change any rights anymore. So it kept on working, but no changes anymore to your rights to use these services. So yeah, the annoying part about that one is that it is very hard to roll out updates or roll out new versions of your application to fix then the issues that were going on. So for example, we use Terraform.

When you do a Terraform apply, well, it will try to see if the role needs to be updated. So we'll try to check the current configuration of the role. That wouldn't work. So depending on your configuration management tool, you weren't able to roll out a new release even of your application.

Yeah, I see. But your application could keep on running. So at Dataminded, our product Conveyor allows you to run data workloads and schedule data workloads. How did this affect our product? So there was an outage. I don't remember when exactly, but years ago with regards to IAM in global.

So at that point, we picked up the best practices of always setting the SDKs to talk to the regional endpoints. Luckily, this can be done with an environment variable. And as you know, Conveyor is a scheduler and runner for data applications using containers. Since it can be set with an environment variable, we configured an environment variable for all the jobs that our customers run. So all our customers automatically run the environment variable.

So all our customers automatically have this configuration set correctly. So that means that most of almost all of the running jobs of our customers could properly run without issues. Yeah. Luckily, we also made a conscious decision a couple of years ago to copy our container images to our regional registry. So for our AWS customers, Yeah.

we were also pulling containers from a local ECR registry and not from ECR public. So they didn't notice any issue there as well. Apparently during immigration, we forgot one minor service. So there was a bit of a hindrance there for our users. But mostly things were running fine. Yeah.

So things that could have gone wrong were like we launch a data job, which is a Docker container essentially. Mm-hmm. And then you're pulled to your cluster. And if you would then have a dependency on, let's say, public ECR, which is hosted in Virginia, then you would not be able to fetch it.

So if I understand correctly, the only fix that you needed to do was like an outdated container image that you were still referring to in public ECR. That was for our AWS customers. So for our Azure customers, we sadly didn't. There was one, we had a similar issue. A component, the networking component was pulling, not from public ECR, that wouldn't make sense, but from K.IO, which is the Red Hat Container Registry, I believe.

Yeah. And they also had an outage because it's hosted on AWS. Yeah. So on Azure, we're going to do the exact same thing that we already did on AWS. We're going to ensure that we have every image replicated in the same region as our clusters are running so that we don't depend on an external service going down.

Yeah. Okay. Let's maybe then talk a bit about what people can do to avoid this or to at least mitigate or mitigate the risk to be affected by an AWS outage. So what are the things you would recommend people do or take action on? So the first one is for this US East 1 outage, it's very clear that IAM is a global service, but you can mitigate the issue with using those regional endpoints.

And that's rather easy. Second thing, you can easily remove that dependency on public ECR. If you're running on AWS, configure your EKS add-ons to use a local copy of public ECR. There are multiple ways to achieve this. You can manually copy them or you can ECR pull through caching.

It's a service that AWS offers. Does that mean that when you pull in an image that it creates a copy? Yeah, it creates a copy on the fly. And the third thing is you also don't want to depend on external container registries too much. So again, you can think of this in two phases for the running or the building.

For running, for sure, don't depend on external registries. We all know the thing that Docker Hub did a couple of years ago, introduced rate limiting. And I know why they did it right. It was costing them, it was a free service. It was costing them way too much.

It makes sense. So you want to duplicate those as well to ensure that when they have an issue, your clusters don't have an issue. The biggest work is mostly in, if you're using the pull through cache, you still need to update where you're pulling your image from. And if you're manually copying them, you also need to update where you're pulling your image from.

That is actually the most work in our experience. For every Helm chart you install in your Kubernetes cluster, you need to update those images. So that's the most work. That's the most work. That's the most work actually.

Okay. And you had a fourth mitigation, I believe, right? Fourth one is prepare. It's basically prepare for outages. Outages will happen. And somebody, everybody can make a mistake.

So you need to prepare and you need to ensure. What we like to do is we have a disaster recovery playbook. So basically it highlights the steps that we want to do. As a company, when a disaster strikes. For us, it's a three step approach. So basically the first one is the notification and activation phase.

Basically, we notify the correct people that an incident is going on. Yeah. That means both internally and externally. And then we're investigating how bad is the issue? What is the impact? Do we need to go to our second phase, which is, recover to another region in our case?

And the third phase is everything is open over now. It's called the reconstitution phase. But basically the simple thing is if you recover to another region, we need to go back to our primary region because that's where we want to host everything. And also we write a postmortem. So we prepare for the future.

And how do you then test all of this? I mean, you can devise a playbook on disaster recovery. But how do you really prepare? That is another very important part of what we do is we test this every couple of months, minimal once a year using a tabletop exercise. Do you know D&D?

Dungeons and Dragons? Dungeons and Dragons. Yeah, you basically all sit together on a table and you imagine that you're playing in this fantasy world. Well, this is a lot more boring. We fantasy that we're playing an issue that is going on. So we also have some kind of a dungeon master like role.

We call it the tabletop facilitator, which prepared a session and said like, hey, this is the outage I prepared that is going to happen. And we're going to role play that. The idea is that they get the reflex of taking that runbook, following it, making a checklist, and it becomes a habit. It becomes easier.

Secondly, it also helps with mental health. Mentally preparing for when an issue is happening, getting in that mindset of, well, we need to first ensure things are mitigated a bit. Do we need to go to that reconstitution phase? That's like a fire exercise that you do and you take some learnings. And in this case, you do it to prevent or to know how you would respond to an outage and follow the whole procedure. Right.

So I'll try to summarize what you recommend is indeed a regional endpoints to have less dependencies. Mm-hmm. Global service. Then also the container registry that you want to pull locally. Mm-hmm. Thirdly, removing your dependencies on external dependencies like Docker Hub and these kinds of services that also rely indirectly on AWS.

And then finally, the big thing, prepare for these things and do the tabletop exercise. Mm-hmm. Yes. Yeah. Okay. If we zoom out, we learned that there were many services or many products being out because more than 140 services of AWS Mm-hmm.

were affected by the IAM outage. Then at Dataminded, we already took some mitigation steps before because we learned from the past that these things can go out. Mm-hmm. And then you mentioned four recommendations on how to mitigate that. And I believe you wrote all of them down in a blog post for people who are interested in more information. So I'll add them to the comments.

And then the only thing remaining, I think, is to thank you, Stijn, for giving us a look into how you dealt with the AWS outage. And then, of course, to share your insights. So thanks a lot. Thank you too, Jonny, for being a great host. Thank you. All right.

Thank you, everybody, for watching. Check out our other videos that we have published in our channel. And we'll see you next time. Bye-bye. Bye. Bye.