How did Amazon screw up the internet this week? A typo

Typing is hard, so we can relate.
Author:
Updated:
Original:

An Amazon cloud service outage disrupted chunks of the internet Tuesday, rendering some apps and website useless, while causing chaos behind the scenes for some companies.

Amazon Web Services is now explaining why its S3 service went out that day: A human typed something wrong.

That morning, workers were trying to debug an issue that was slowing down payment processing. Just after 11:30 a.m. CST, one of those employees punched in a command that was supposed to take a "small number" of S3 servers offline to help figure out what was causing the slowdown.

"Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended," the explanation says.

So an accidental typo ended up taking down servers that are needed to manage metadata and handle certain commands.

Amazon had to restart those servers, and it "took longer than expected." During that time, a bunch of services weren't available.

And that's why things got all crazy.

Amazon Web Services says it's built to handle certain failures without any customer impact, writing they operate "with the assumption that things will occasionally fail." But they haven't done a full restart like was required this time, and didn't know it'd be such a process.

They've made some changes to how things work, including tweaking that command tool that the typo was punched into, so in future cases it'll remove systems slowly and prevent large numbers of servers from being taken offline.

"Finally, we want to apologize for the impact this event caused for our customers," the message read. "We will do everything we can to learn from this event and use it to improve our availability even further."

So there you have it. All it took was a typo to bring down apps and websites for an afternoon.

Next Up

scratch lottery ticket

State lottery: Don't give scratch-offs as gifts to minors

The Minnesota Lottery participates in an annual responsible gambling campaign focusing on underage lottery play during the holidays.

Marijuana, cannabis

Minnesota adds 2 more qualifying conditions for medical marijuana

There will be 17 conditions that qualify people to obtain medical marijuana in Minnesota.

Matt Birk

Ex-Viking Matt Birk confirms interest in future run for governor

The next gubernatorial election in Minnesota is Nov. 8, 2022.

Screen Shot 2020-12-01 at 10.59.59 AM

Handsome Hog in St. Paul to close temporarily

Executive chef Justin Sutherland cited inaction by state and federal leaders during the COVID-19 pandemic.

Classroom

Dept. of Education adding more gender options when collecting student data

The public can comment on the implementation of the nonbinary gender options until Dec. 13.

police tape, crime scene

Police ID couple found dead in home, murder-suicide suspected

The bodies were found in St. Louis County after the pair failed to show for work.

State Capitol

Minnesota's budget forecast improves, shows $641M surplus for 2020-21

"Higher general fund revenues and lower expected spending result in a projected surplus," MMB said.

Screen Shot 2020-08-14 at 6.45.53 PM

Brooklyn Park shooting leaves 2 injured

Two victims sustained non-life threatening injuries during the Monday evening incident.

Screen Shot 2019-10-14 at 10.38.01 PM

Worthington settles police brutality lawsuit for $590,000, agrees to reforms

The lawsuit was filed on behalf of Kelvin Francisco Rodriguez, who spent five days in the ICU after being arrested.

Michael Munoz

Rochester superintendent admits he plagiarized letter to staff

Med City Beat broke the story, prompting the superintendent to apologize.

Related

Update: Amazon says everything's fixed, the internet should be normal

It's not just you. Lots of websites and apps were having trouble today.

Amazon's HQ2: How it could dramatically transform the Twin Cities tech scene

"Amazon is pushing the boundaries in just about every direction," one local tech leader said.

Report: Minnesota's Amazon tech office is working on a secret Uber-like truck app

There's a reason Amazon is interested in the trucking business.

Net neutrality supporters will try to 'break the internet' Tuesday

Net neutrality supporters are taking action Tuesday – here's what to expect while you're browsing.

A cyber attack is holding computers around the world for ransom

One malware site says this WannaCry attack has been detected in 99 countries.

The internet is having problems today because of a cyber attack

A second round of cyber attacks has led to more reported outages in Minnesota.

Can a new partnership with Amazon and Alexa save Sears?

Alexa, help rescue Sears from doom. And start my laundry while you're at it.