Facebook Switches to New Timekeeping Service

Facebook recently switched millions of its own servers and consumer products (including Portal and Oculus VR headsets) over to a new timekeeping service. The company says the new service, built in-house by the company's engineers using open-source tools, is more scalable than the one it used previously. What's more, it will improve the accuracy of device's internal clocks from 10 milliseconds to 100 microseconds.

To figure out what time it is, Internet-connected devices look to timekeeping services maintained by companies or government agencies such as the U.S. National Institute of Standards and Technology (NIST). There are dozens of such services available. Devices constantly ping them for fresh timestamps formatted in the Network Time Protocol (NTP), and use the info to set or recalibrate their internal clocks.

With the announcement, Facebook joins other tech companies including Apple and Google that operate publicly-available timekeeping services of their own. Facebook's service is now available for free to the public at time.facebook.com.

To maintain products and organize customer data, tech companies rely on large databases. Engineers must ensure that the time of each entry is recorded in a consistent way. In many cases, such companies prefer to build and operate their own time services rather than rely on publicly-available options. Owning a time service eliminates delays and interruptions that may occur as data travels between a company's servers and an external provider.

Pat Diamond, principal consultant at Diamond Consulting and ATIS (Alliance for Telecommunications Industry Solutions) workshop chair, says Facebook's announcement is likely a sign of the company's shift to using more distributed databases and services. When data is stored across multiple machines, it's especially important that all systems share the same value for time. "Facebook is starting to move into that space more, and the further they get into it, the more they learn about what they need to have," he says.

As is true of other time services, Facebook's relies on atomic clocks to keep the time. These clocks produce a signal once they've measured radiation indicating cesium atoms (or in Facebook's case, rubidium atoms) have changed energy states a certain number of times-a chain of events whose duration is fixed in nature.

Satellite constellations that provide global navigation services (including Galileo, GLONASS, and GPS) carry atomic clocks and broadcast that signal to receivers below. Time services such as Facebook's translate signals from those broadcasts, and their own on-site atomic clocks, into time of day.

When distributing time to connected devices, these services must minimize distortions or delays created by networks or operating systems along the way. Facebook's new service performs this task better than its old one, thereby keeping time more accurately.

Photo: Facebook Facebook built its own testing rig to evaluate the company's new time service (based on chrony) compared to its predecessor (based on ntpd).

The biggest technical change Facebook made was to switch its timekeeping service over to a new time daemon. A time daemon is a software tool that coordinates time across many connected devices.

The time daemon that Facebook had used was a popular one called ntpd, for Network Time Protocol Daemon. The one it switched to, called chrony, is newer and not as widely used. Chrony consumes less memory (about 1 mebibyte) and less power than ntpd. "It's much more modern than ntpd," says Diamond.

Timur Snoke, a senior network defense analyst in the CERT Division of the Software Engineering Institute at Carnegie Mellon University, says the chrony-based service also appears to be more secure. In switching to chrony, Facebook abandoned features of ntpd known to be vulnerable to hacks, including a problematic authentication option.

Since chrony is a smaller application, Snoke adds, "there are less opportunities for there to be issues with the code" that could produce security flaws. And he says "it does look like they made a bunch of really good decisions to get rid of the components that could be misconfigured."

Chrony is also more tightly coupled to a device's hardware. With chrony, a time service can use hardware timestamps, which are derived from a system's network interface card, to eliminate errors introduced by its operating system. Facebook found that these hardware timestamps helped chrony keep time more accurately than ntpd.

"We proved that chrony is a lot more precise," says Oleg Obleukhov, a production engineer in Facebook's Dublin office. "By our measurements, it's 10 to 100 times more precise than ntpd."

Facebook's time service also preforms a function called leap-second smearing. Due to irregularities in the Earth's rotation, timekeepers must add or remove a second from the world's clocks every few years-a controversial practice known as the leap second.

Facebook's service, like those operated by other tech companies, includes a layer that "smears" a leap second out, gradually adding or subtracting it over an 18-hour period. With ntpd, a leap second was smeared ahead of the actual leap second-with the switch to chrony, the smearing will take place afterward.

Facebook's internal tests show that its new time service is at least as accurate as those offered by Apple and Google. However, none of the time keeping services provided by tech companies are certified as accurate to Coordinated Universal Time (UTC)-the gold standard that banks, network providers, and grid operators rely on.

Image: Facebook Facebook's internal tests show the company's new time service (blue) is comparable to services operated by Apple (yellow) and Google (orange) in deviation from "true time" (represented as zero in the chart).

The reason, says Diamond, is that tech companies don't really need time to be that accurate, so long as all their systems and devices are in sync. "Is it the same as NIST time? Probably not," he says. "Is it close enough? Yeah." He adds that Coordinated Universal Time deals with leap seconds instantaneously, with no smearing required.

Diamond was impressed with the Facebook team's work, which they describe today in a blog post. "I was encouraged by how thorough they were," he says. "The engineers they had working on it clearly understood what they were trying to accomplish."

The switch to chrony was completed in December 2019. Obleukhov says Facebook's engineers are now considering whether to adopt Precision Time Protocol (PTP) across its systems, which is generally considered more accurate than NTP.