Synchronizing Networks with NTP

by Glenn Graham
01/02/2003

Is Your Network Running On Time?

Most system administrators don't consider the importance of accurate network timing. Under Unix, log daemons such as syslogd write their information to the central log files in 00:00:00 (hour:minute:second) format. tcpdump writes information in the 00:00:00.000000 format. Cron jobs depend on an accurate system software clock to execute processes. Apache timestamps its logs.

Why Do We Need Accuracy?

If your server doesn't keep accurate time, your log files are useless in the event of an incident that requires log-dependent information, including security breaches. E-mail servers and other clients depend on accurate time to relay, send, and receive data. What good is the date stamp contained in an e-mail if the server that passed that information is inaccurate? These programs all must be timed precisely to within 1/100 of a second.

Last year our network suffered a power outage. Until I started reviewing the data in the log files, the exact time wasn't clear. I was pleased to find accurate information that pinpointed the exact time of failure on each separate A/C line and UPS in the facility for each server.

Around the same time, another network I administered was breached. In this case, each server reported to a remote log server via syslogd that was timed to a Network Time Server. After the system was attacked, the perpetrator attempted to cover his tracks by altering the local logfile on the compromised hosts. What he didn't realize was that another log existed -- "the real McCoy" on the remote monitor, containing unaltered accurate information and synchronized via NTP. The information I obtained proved invaluable and helped to find and prosecute the intruder. When presenting evidence to law enforcement, the first question they asked was, "How accurate is the time on your logs?" I was quickly able to prove the accuracy of the system's time.

Many organizations use a time standardazation protocol called NTP (Network Time Protocol). A team of professionals at the University of Delaware maintains it.

Related Reading Essential System Administration Pocket Reference Commands and File Formats By Æleen Frisch

The Time Protocols

NTP is based on its own unique protocol. Today, NTP has become the de facto standard as it has evolved through various RFCs.

The most recent release is a complete implementation of version 4. It retains compatibility with version 3, as defined by RFC 1305, and versions 1 and 2, defined by RFC 1059 and RFC 1119, respectively. NTP v.4 implements asymmetric encryption, to prevent malicious users from spoofing time packets over port 123. There is no formal RFC for version 4 yet, though there is a draft for using public key cryptography with NTP.

The NTP Chain of Commands

An NTP primary server, or stratum 1, is connected to a high precision reference clock and equipped with NTP software. Other computers (stratum 2s), equipped with similar software automatically query the primary server to synchronize their system clocks. The stratum 2 computers can synchronize other computers, called stratum 3s, and so on, to 16 stratums. The further a computer is from stratum 1, the less accurate its timing information. In this example, each computer can be a client of a server in a higher stratum and a server for computers in a lower stratum. A single primary server can indirectly serve an unlimited number of clients.

To increase reliability, each client may query multiple servers in the upper stratum. In this case, the NTP software continuously monitors the stability and accuracy of all configured servers, switching dynamically to the server with the best figures.

Where does a stratum 1 get its initial data? A stratum 1 requires a radio or satellite receiver. Modern systems retrieve their data from the NIST radio broadcasts or GPS data.

Implementation

NFS, the Network File System, has long been known to have remote exploits and vulnerabilities. Even secure NFS has weaknesses. However, NFS is also a very reliable means of copying data to a central backup server. Several systems I work with implement NFS for that very reason. Here's where the NTP comes in handy. Each server runs a cron job at a specified time to execute its backup. The backup job has the ability to enable and to disable the NFS server. Although I incorporate NFS, I don't allow it to be on all the time - only during automated backups.

Here's the order of operations. First, the cron job fires on the client. It starts NFS. The server knows when this should happen, so it mounts the client share. The server backs up the files on the share and unmounts it. Finally, the client stops NFS.

Once the backup server and clients have synchronized, the script executes on each machine. This is all based on extremely accurate, split-second timing. After the backup is complete, the scripts disable NFS. If one machine is out of sync, that machine won't be able to perform a backup to the central server.

Related Reading Hardening Cisco Routers By Thomas Akin Table of Contents Index Sample Chapter 10: NTP

The Importance of Choosing an Accurate NTP

Three years ago, I had several servers timed to a single NTP server located at a university. Unbeknownst to me, the administrators were checking for the Unix 2038 time overflow bug. During their test, they advanced their real time clock to the year 2038. Suddenly, each of my log files started writing the year 2038 instead of 1999. This caused massive problems for systems across the network. The lesson learned was that most NTP clients have the ability to query multiple stratums, writing the average of difference to the software clock. Where the difference is significantly out of sync, the client will either compensate to the clock closest to the last known value or exit, leaving the local clock(s) unchanged.

Choosing an NTP Architecture

Arguments abound on how to implement NTP on a local network. Some say that a (single) local server should be designated to poll the upstream stratums, synchronize, then broadcast to the local network. Others believe the local area hosts should (directly) poll the upstream stratums. Most modern systems use the first approach.

The advantage of an internal stratum is so that all machines on your LAN may be timed to the local network. Only one machine needs to be timed to the outside. The disadvantage is that downstream systems may be affected if your local stratum goes off sync.

We'll use a Unix host running ntpd to connect to a stratum 2 across the Internet. Our local machines will each synchronize to the local stratum.

The ntpd Server

The ntpd daemon can operate as a server, a client, or a relay from a set of servers to a dependent client population on a local net.

To configure ntpd, edit /etc/ntp.conf. As the ntp.conf man page is very long, listing every option is outside the scope of this article. However, useful configuration commands include server, peer, broadcast, manycastclient, and options include autokey, burst, iburst, minpoll and maxpoll.

Here is the basic ntp.conf for "timekeeper", a stratum 3 on our internal network. Note each server listed is an official stratum 2.

# File /etc/ntp.conf
# Localhost timekeeper.foo.com
server tick.uh.edu
server time.nist.gov
server tick.usno.navy.mil
driftfile /etc/ntp.drift

The next configuration configures our stratum 4 to check timekeeper, then a stratum 2 machine. It will write the difference to the drift file and the system clock.

# File /etc/ntp.conf
# Localhost yogi.foo.com
server timekeeper.foo.com
server time.nist.gov
driftfile /etc/ntp.drift

In each example, you must touch the file /etc/ntp.drift, which contains the offset differences of each upstream stratum. Before attempting to retrieve data, wait until ntpd has had time to synchronize.

Start ntpd by running it from the command line without any arguments. Be sure to add the daemon to your local start script. Once the server has been started, check the offset by tailing your messages log. You should see a line resembling:

Nov 13 03:16:52 striker ntpd[31962]: time set -0.096919 s

If you are running in a production environment, I suggest reading the man section pertaining to restricted access and encryption. ntpd runs on port 123. Be sure other applications such as PortSentry are not bound to this port.

Running NTP Clients

I recommend running NTP software hourly from your cron. Most Unixes provide ntpdate. (Some Linux distributions provide a program called netdate, but it does not support version 4 of the protocol.) Both commands follow the same syntax:

$ ntpdate <primary time server> <secondary time server>

For a Windows server or workstation, you can download and install a program called Dimension 4. This program provides several options for checking and correcting the time, and has an excellent, up-to-date list of time servers.

Recent versions of Mac OS X provide a simple checkbox under the Network Time tab of Date & Time settings.

Summary

Time servers are a science in themselves. As we speak, scientists are working on improvements. If you take time synchronization for granted, then I suggest you read more about this fascinating technology. For those of you worried about the 2038 bug, relax. In 36 years most Unixes as we know them will have been improved; many of us will be retired on a beach in Bermuda or so we hope.

Glenn Graham has been working with telecommunications since 1977.

---

Return to the Linux DevCenter.