Skip to main content
Updated:
Original:

How did Amazon screw up the internet this week? A typo

Typing is hard, so we can relate.

An Amazon cloud service outage disrupted chunks of the internet Tuesday, rendering some apps and website useless, while causing chaos behind the scenes for some companies.

Amazon Web Services is now explaining why its S3 service went out that day: A human typed something wrong.

That morning, workers were trying to debug an issue that was slowing down payment processing. Just after 11:30 a.m. CST, one of those employees punched in a command that was supposed to take a "small number" of S3 servers offline to help figure out what was causing the slowdown.

"Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended," the explanation says.

So an accidental typo ended up taking down servers that are needed to manage metadata and handle certain commands.

Amazon had to restart those servers, and it "took longer than expected." During that time, a bunch of services weren't available.

And that's why things got all crazy.

Amazon Web Services says it's built to handle certain failures without any customer impact, writing they operate "with the assumption that things will occasionally fail." But they haven't done a full restart like was required this time, and didn't know it'd be such a process.

They've made some changes to how things work, including tweaking that command tool that the typo was punched into, so in future cases it'll remove systems slowly and prevent large numbers of servers from being taken offline.

"Finally, we want to apologize for the impact this event caused for our customers," the message read. "We will do everything we can to learn from this event and use it to improve our availability even further."

So there you have it. All it took was a typo to bring down apps and websites for an afternoon.

Next Up

940 Margaret St, St Paul, MN 55106, United States - May 2019 (1)

4-year-old girl struck by hit-and-run driver in St. Paul

The girl was struck by the driver around 5 p.m. Monday.

Minneapolis police

Baby found safe inside car that was stolen in south Minneapolis

The incident happened near 32nd Street and Chicago Avenue around 7 p.m.

Anthony Edwards

Ant puts on a show to help Timberwolves beat Pacers

A third-quarter surge gave the Timberwolves their seventh win in their past eight games.

Mikko Koivu

Wild to retire Mikko Koivu's No. 9 jersey

Koivu will be the first player in franchise history to have his number retired.

Boebert-Omar - Flickr Gage Skidmore

Rep. Boebert refuses to apologize for Islamophobic comment toward Rep. Omar

A phone call Monday between the two lawmakers ended abruptly.

Patrick Peterson

Vikings place Patrick Peterson on COVID-19/reserve list

The Vikings cornerback said he was vaccinated in August.

flickr - thin ice warning - USFWS Midwest

'Numerous' reports of eager anglers falling through ice

Conditions are still unpredictable, and can vary even across a single body of water.

covid

Minnesota health officials watching closely for omicron variant

Health leaders do not yet know how transmissible or severe the new variant is.

school bus stop pixabay

School bus driver charged in fatal hit-and-run will plead guilty

Another motorist told the driver to call 911, but he instead got on the bus and drove off, charges say.

Franconia Sculpture Park - Lorie Shaull - Flickr

5 outdoor destinations to explore around the Twin Cities

There's something for everyone on this list of overlooked spots.

Child mental health counseling

As pandemic continues, so do efforts to improve child mental health access

Children's Minnesota has announced it will open its first inpatient mental health facility for under 18s.

Dalvin Cook

Mixed reports on Dalvin Cook's injury status

The Vikings running back could miss two games...or he could play Sunday.

Related

Update: Amazon says everything's fixed, the internet should be normal

It's not just you. Lots of websites and apps were having trouble today.

Amazon's HQ2: How it could dramatically transform the Twin Cities tech scene

"Amazon is pushing the boundaries in just about every direction," one local tech leader said.

Report: Minnesota's Amazon tech office is working on a secret Uber-like truck app

There's a reason Amazon is interested in the trucking business.

Net neutrality supporters will try to 'break the internet' Tuesday

Net neutrality supporters are taking action Tuesday – here's what to expect while you're browsing.

A cyber attack is holding computers around the world for ransom

One malware site says this WannaCry attack has been detected in 99 countries.

The internet is having problems today because of a cyber attack

A second round of cyber attacks has led to more reported outages in Minnesota.

Can a new partnership with Amazon and Alexa save Sears?

Alexa, help rescue Sears from doom. And start my laundry while you're at it.