Skip to main content

How did Amazon screw up the internet this week? A typo

Typing is hard, so we can relate.

An Amazon cloud service outage disrupted chunks of the internet Tuesday, rendering some apps and website useless, while causing chaos behind the scenes for some companies.

Amazon Web Services is now explaining why its S3 service went out that day: A human typed something wrong.

That morning, workers were trying to debug an issue that was slowing down payment processing. Just after 11:30 a.m. CST, one of those employees punched in a command that was supposed to take a "small number" of S3 servers offline to help figure out what was causing the slowdown.

"Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended," the explanation says.

So an accidental typo ended up taking down servers that are needed to manage metadata and handle certain commands.

Amazon had to restart those servers, and it "took longer than expected." During that time, a bunch of services weren't available.

And that's why things got all crazy.

Amazon Web Services says it's built to handle certain failures without any customer impact, writing they operate "with the assumption that things will occasionally fail." But they haven't done a full restart like was required this time, and didn't know it'd be such a process.

They've made some changes to how things work, including tweaking that command tool that the typo was punched into, so in future cases it'll remove systems slowly and prevent large numbers of servers from being taken offline.

"Finally, we want to apologize for the impact this event caused for our customers," the message read. "We will do everything we can to learn from this event and use it to improve our availability even further."

So there you have it. All it took was a typo to bring down apps and websites for an afternoon.

Next Up

George Floyd, protest

Twin Cities authorities preparing for community reaction to Tyre Nichols video

The ATF division in St. Paul sent out an alert out of "an abundance of caution."

Screen Shot 2023-01-27 at 2.12.05 PM

Changes at K102's Chris Carr & Company, with Sam Sansevere joining as co-host

She is the daughter of former KQRS morning show personality Bob Sansevere.

Screen Shot 2023-01-27 at 1.16.21 PM

Minneapolis' Royal Foundry Craft Spirits announces closure

The distillery set out to infuse the Minneapolis craft cocktail scene with British flavor.

ShutterflyGoogle

Shutterfly to cut almost 100 jobs at Shakopee facility

The layoffs are expected to impact multiple positions within the company.

millcitymuseum_hr1

Mill City Museum in the running to be named best history museum in the U.S.

The museum offers scenic views of the Mississippi River.

HolmgrenMugRamseyCo

Man charged with attempted murder of White Bear Lake police officer

Daniel Holmgren Jr., 33, was being served an arrest warrant at the time of the incident.

image

Gallery: 1,000-acre Wisconsin resort listed for nearly $4.3M

The lush landscape features a creek and private lake.

school bus

Alert after Woodbury students approached by strangers asking if they want candy

Valley Crossing Elementary School urged parents to go over safety information with their children in response to the incident.

basketball unsplash stock

North St. Paul basketball cancels Eden Prairie game after opposing coach's reinstatement

Head Coach Cornelius Gilleylen said he let his players decide amongst themselves.

Pixabay - school - classroom

These are the nominees for 2023 Minnesota Teacher of the Year

Education Minnesota has nominated 131 teachers from across the state.

Related

Update: Amazon says everything's fixed, the internet should be normal

It's not just you. Lots of websites and apps were having trouble today.

Report: Minnesota's Amazon tech office is working on a secret Uber-like truck app

There's a reason Amazon is interested in the trucking business.

Amazon's HQ2: How it could dramatically transform the Twin Cities tech scene

"Amazon is pushing the boundaries in just about every direction," one local tech leader said.

A cyber attack is holding computers around the world for ransom

One malware site says this WannaCry attack has been detected in 99 countries.

Net neutrality supporters will try to 'break the internet' Tuesday

Net neutrality supporters are taking action Tuesday – here's what to expect while you're browsing.

The internet is having problems today because of a cyber attack

A second round of cyber attacks has led to more reported outages in Minnesota.

You can get local, handmade gifts delivered in 2 hours with Amazon Prime Now

This sure seems like another swipe at craft competitor Etsy.