By

Years ago, I worked at Gameloft as a game server developer.
Our server needed to handle player actions in real-time. Its main loop ran at a fixed speed of 10 FPS, meaning the server would process data every 100 milliseconds. If the processing finished in less than 100 ms, the server would sleep until the start of the next 100 ms period.

To achieve this, the server called the Linux times() function before and after the processing step to calculate elapsed time, as shown below:

int64 beginTime = getCurTime();
// hadling player data
int64 endTime = getCurTime();
int64 timeElapsed = endTime - startTime;

You can see that timeElapsed is always expected to be a positive number.

One day, all the servers crashed simultaneously.
Upon inspecting the core dump, I discovered the cause: timeElapsed was calculated as -900 ms, meaning the server’s clock had somehow gone backward by a second. Every single server was affected.

Was this real-world time travel?

I then recalled a news article I had recently read: About Leap Seconds.
We all know about leap years: the Earth’s orbit around the Sun isn’t exactly 365 days, it is closer to 365.24 days.
Leap years help ensure our calendars remain aligned with the Earth’s orbit.
Similarly, the Earth’s rotation isn’t perfect 24 hours, it varies due to climatic and geological factors.
To keep the world clock aligned with the Earth’s rotation, adjustments are made when the difference approaches one second.
These adjustments are called leap seconds.

The most recent leap second, at the time, had been applied on 2016-12-31, the exact date our servers crashed.
At midnight, all server’s time moved back from 00:00:00 to 23:59:59 of the previous day.

The bug was easy to fix but hard to forget.

Contents