June 30: 23:59:60, The Second That Made Servers Sweat

Show Notes

A leap second sounds harmless until the world’s machines have to live through one. On June 30, 2012, official UTC time briefly displayed 23:59:60, stretching the final minute of the day to keep human clocks aligned with Earth’s uneven rotation. For people, it was a strange timestamp. For servers built around ordinary minutes, sleeping processes, scheduled jobs, and predictable time comparisons, it exposed a fragile assumption hiding deep inside modern infrastructure. The disruption was not a cinematic internet collapse, but it was unnerving: Reddit became nearly unusable, then went offline, while other services including Mozilla, LinkedIn, Foursquare, Yelp, and Gawker Media reported their own leap-second-related instability. The failures varied across Linux timers, Java systems, Cassandra, Hadoop, Tomcat, and other stacks, but the lesson was shared. Rare code paths are still real code paths, and time — before it becomes a timestamp — is a physical thing that does not always move the way software expects. Read the original article: https://medium.com/@dia_91230/june-30-23-59-60-the-second-that-made-servers-sweat-d00bc57383eb Subscribe to The Dead Internet Almanac: https://buttondown.com/dead-internet-almanac

What is Dead Internet Almanac?

Old games, dead platforms, forgotten memes, vanished websites, and the strange little artifacts that somehow survived.

For one second, the official clock showed a time most wall clocks will never admit exists.

Twenty-three fifty-nine and sixty seconds.

It looked like a typo. The kind of broken timestamp you get from a cheap script. Then parts of the web began to grind.

That strange little digit came from a real procedure. A leap second is a one-second correction, added now and then to official time so the world's clocks do not drift too far from Earth's uneven rotation. On June thirtieth, twenty twelve, the last minute of the UTC day lasted sixty-one seconds. The Paris Observatory's IERS leap-second page still says it with beautiful bluntness.

The best surviving image is tiny. A clock captured on Wikimedia Commons, showing twenty-three fifty-nine sixty in the box where a normal person expects the minute to end. It is a public-domain government screenshot, only three hundred and forty pixels wide, and it carries the whole joke.

For one second, official time looked like it had failed a validation check.

Computers are supposed to be calmer about this than people are. They synchronize with time sources. They compare timestamps. They sleep, wake, schedule jobs, and assume the clock is boring.

The leap second put a rare event into systems built for ordinary minutes. Enough production machines entered a strange state that engineers had to remember time is physical before it is digital.

The failure was smaller than apocalypse and stranger than a simple outage.

WIRED followed the twenty twelve incident through Reddit, where administrators first thought they were seeing a network problem after an Amazon Web Services outage earlier that weekend. The trouble turned out to be inside their own stack.

WIRED reported that Reddit was nearly unusable for thirty to forty minutes, then entirely offline for about an hour and a half. Affected servers eventually needed reboots.

The bug was subtle in the way infrastructure bugs like to be subtle.

Reddit's servers were running Linux, Java, and Apache Cassandra. As WIRED explained it, the leap second interacted badly with Linux's high-resolution timer machinery. Processes woke in ways that sent CPU usage wild.

Linus Torvalds gave the cleanest diagnosis. Leap-second code is almost never run, so users almost never test it under normal conditions.

That is the cold little horror of the story.

The code path was real. It was waiting. Most days it might as well have been a locked door in the basement.

The disruption spread beyond one site. CBS News and CNET reported problems or instability at Mozilla, Gawker Media, LinkedIn, Foursquare, and Yelp, while being careful not to treat every service as if it had the same failure or the same duration.

Mozilla's issue involved Hadoop. Gawker mentioned Tomcat servers. Reddit had Java and Cassandra symptoms.

The shared villain was not one product so much as a cultural assumption: time moves forward in the ordinary way.

Some companies had learned to sand the edge off the event. Google described its leap smear approach, spreading the extra second out in tiny increments instead of taking the full step at midnight.

It is a very Google solution. Make a cosmic bookkeeping adjustment disappear into millisecond dust before anyone has to look at it.

The old web loved big visible failures. A front page down. A whale image. A database fire. A line of red text in a status post.

The leap second was quieter. It hid inside schedulers and timers, inside machines that looked fine until they were asked to experience one legal second that normal software rarely meets.

The June twenty twelve leap second was announced like infrastructure often is, in plain text. The IANA time-zone mailing list preserved the Bulletin C wording.

Twenty twelve June thirtieth, twenty-three hours, fifty-nine minutes, fifty-nine seconds.

Then twenty-three hours, fifty-nine minutes, sixty seconds.

Then July first, zero hours, zero minutes, zero seconds.

It reads like a spell for making computers uncomfortable.

There is something almost tender about that official sequence. The world is messy, so timekeepers add a second. Programmers try to make machines behave as if the world is clean.

Somewhere between those two jobs, a server room fills with fans, alarms, SSH sessions, and a timestamp that looks impossible but is, for one second, absolutely correct.

The web kept going.

But twenty-three fifty-nine sixty stayed behind like a trapdoor under the clock.