A month or so ago all of my multifactor authentication tokens for my administrator access to our Totem servers suddenly stopped working. All of them. As a result, I could not log into any instances of our tools. Being a paranoid cybersecurity guy, I imagined the worst: all of the tools had somehow been hacked, and the attackers were denying administrator access so we could not combat the problem. So, I asked a colleague with administrator access to one of the servers if he could still log on. He could, with no issues. So, why had I been singly targeted? Turns out there was no hack, and no problem at all with the tools. The problem turned out to be an old nemesis of mine (and of many a system administrator over the years): unsynchronized network time. In this post, we’ll explore why network time synchronization (we’ll call it “time sync” for short) is crucial for proper functioning of an IT system. We’ll also explore why the National Institutes of Standards and Technology (NIST) considers time sync important enough to include explicit requirements for it in the 800-171 standard for Department of Defense (DoD) contractors to protect Controlled Unclassified Information (CUI). And since the DoD Cybersecurity Maturity Model Certification (CMMC) is built around NIST 800-171, network time sync is critical in CMMC as well.
Brother CMOS, Sister OSI, Mother Token, Father Time
The reason my Totem multifactor authentication (MFA) stopped working was subtle. My Totem MFA tokens are generated by an app on my phone. These one-time-password tokens (six-digit codes) are generated every 30 seconds by a highly time-dependent algorithm. The day before this issue, I had traveled from the East Coast to our Totem headquarters in Utah. When I landed, for some reason my phone did not automatically update to Mountain Time like usual. A restart didn’t help, so I assumed it was a bug with a recent software update. So, I manually set my phone to Mountain Time and thought nothing of it.
The problem with this is that I was relying on my phone to keep accurate time, and computers are relatively shoddy timekeepers. (A smart phone is a powerful, but compact, computer). Their internal clocks are maintained on little CMOS chips on the motherboard, and a variety of factors causes the time kept by these things to “drift” pretty quickly. They’re worse timekeepers than a bored kid on a long car ride. OK, I’m exaggerating: computers are amazing timekeepers if we compare them to a 19th century pocketwatch. But computing network protocols, such as the Transmission Control Protocol (TCP), rely on consistent synchronized timekeeping at the millisecond level. If one networked computer’s internal time is off by a few seconds, communication on the network can grind to a halt. Synchronized time is important enough that there exists an entire protocol for managing it: the Network Time Protocol (NTP). Our computers are configured by default to sync with NTP servers on the network (or the Internet) to ensure that other network protocols stay in working order. By manually configuring the time on my phone, I had inadvertently turned off my phone’s ability to use NTP.
I was stymied by my problems for a while, until I decided to go back to network troubleshooting basics and work my way up the OSI model.
The OSI model describes the logical layers that make up a modern-day network-based application. When troubleshooting using this model, one starts by checking the foundational level – the physical layer – first and moving one’s way “up” the layers one-by-one, ruling out potential culprits. The physical layer wasn’t the problem in my case, as we knew the Totem servers had network connectivity, and a physical connection for my phone wasn’t required to generate MFA codes.
The next layer in the OSI sequence is the link layer, but I learned the hard way in the beginning of my career to, at this point, detour up three layers to the application layer and check network time sync. (I distinctly remember in my early days troubleshooting for two days why a server would not mount windows-based network fileshares. I tried everything I could think of –twice– when finally I asked for help. A senior system administrator asked me if system time was correct. When I checked, the NTP service had stopped working. Rookie mistake.) It was when contemplating this detour that I remembered I had set my phone’s time manually. I decided to toggle the phone back to network time, which this time, for whatever reason, the phone seemed to be fine with. I immediately tried logging into a Totem server, and to my immense relief, my MFA token was accepted. I tried another server: success! After a third successful login I realized I had fixed my problem. *whew* But time sync had struck again *sigh*. The problem was this: as it turns out, over the course of 12 hours or so my phone’s clock had drifted enough that the MFA tokens were out of sync with what the Totem server’s mirrored algorithm was expecting. As a result, the server was rejecting my MFA, and I could not log in. Once I resolved the time drift, MFA worked again.
Clearly, time sync is critical for MFA to function properly. There are several MFA requirements in NIST 800-171 / CMMC:
- 3.5.3: Use multifactor authentication for local and network access to privileged accounts and for network access to non-privileged accounts.
- 3.7.5: Require multifactor authentication to establish nonlocal maintenance sessions via external network connections and terminate such connections when nonlocal maintenance is complete.
Since we are required to implement MFA to protect CUI, then by corollary we must make sure that time is synchronized. However, I mentioned above that network time sync is not just implied by NIST/CMMC, but is explicitly required. Where is this requirement and why? We’ll explore this next.
Where is network time sync required in CMMC?
The NIST 800-171 explicit requirement for network time sync can be found in the Audit and Accountability family:
- 3.3.7: Provide a system capability that compares and synchronizes internal system clocks with an authoritative source to generate time stamps for audit records.
The rest of the controls in this family require us to hold users – both authorized and unauthorized – accountable for their actions in our covered systems by:
- Configuring all system components to generate logs of events of interest (e.g. failed logins) with sufficient logged information to identify the user or process that triggered the event
- Gathering and aggregating these logs in a central Security Information and Event Management (SIEM) tool that facilitates:
- Event correlation, reduction, analysis, and reporting, i.e. looking for the “needle in the haystack”
- Preventing unauthorized access to the logs at any point in the journey from generation to reporting, including limiting access to the SIEM to an authorized subset of privileged users
The spirit of this family of controls is to identify anomalous behavior in the environment so that a response process can be triggered if the anomaly appears malicious. Figuratively, we are expected to build fire towers in our environment, and staff those fire towers with skilled fire watchers looking for forest fires.
So, why is the NTP requirement included in this family of controls that is so focused on identifying and responding to anomalies in our environment? The answer is in the word correlation.
Crime Scene Investigation, D.C. style
In the fall of 2021 I attended a conference in Washington, D.C. I parked my car in the hotel garage for the duration of my three-day stay. When I went to fetch the car to head back home, to my chagrin I noticed the passenger side had been swiped by another car, which severely damaged the body panels and destroyed the mirror. No note or other notification had been left to me by the offender. The damage was extensive enough that whoever did it must have felt the collision, so this was a hit and run. I was pissed.
But I noticed that despite the standard “Not responsible for vehicle damage” signs, the garage did have several cameras in the area where I parked my car. I was hopeful I could use the recorded video to get a look at least at the type of car that had been next to me.
I went to the hotel management and asked if we could take a look at the video. The facilities manager was obliging, but not confident. He mentioned that the cameras were fisheye, so there would be distortion and not a lot of resolution. That was OK though, because when I asked him if there was a camera that had a view of vehicles’ license plates as they entered the garage, he said yes.
We spent about half an hour fast forwarding through the stored video from the camera that had a view of the area where my car was parked. In sped-up motion I watched me park my car, get out, and head to the elevator. Nothing noteworthy happened the first day. Cars came and went; several parked next to me without incident. Eventually, we spotted the dirty deed. On the second day, a dark-colored SUV had tried to back into the spot next to me, and the driver wildly misjudged both the vehicle’s turn radius and their own ability to manage a multi-point turn. We rewound and slowed the video so I could watch in real-time my poor Hyundai’s passenger side evisceration. Only after shearing off the mirror did the driver stop backing, presumably after realizing their miscalculation. Then, he or she — the camera resolution didn’t show the driver well enough to discern — paused for a few seconds, and sped out of the camera’s view.
The camera was far enough away from the area that we couldn’t determine the make and model of the SUV, and we could see that there was no front license plate. Still, I was hopeful I could turn to the entrance camera to try to make out the rear license plate of an SUV entering the garage near that point in time on the second day. That way, I might be able to track down the culprit. So, we started checking out the camera at the garage entrance. As we started rewinding the video, I noticed the time stamp in the bottom right corner of the video feed: 00:00:00.00 1 JAN 1969. My heart sank. The entrance camera’s timekeeping system was broken. It was going to be very difficult to correlate the time any vehicle entered the garage with the time my car got wrecked.
“Well, just scroll back the video until you see a dark-colored SUV enter, and that’s probably the one” you say.
My reply: “You don’t know how many dark-colored SUVs there are in the world until you watch parking garage video.” I.e.: there were a lot. Dozens at least just in the several hours of video I watched.
I contemplated how I might be able to work around the lack of time synchronization. Could I just scrutinize for the transition between daylight and nighttime to work my way back a day and a half? Perhaps, but would I be sure any one of the probably several dozen SUVs were the one? I didn’t have that kind of time, and the facilities manager had other things to do. I wasn’t going to be able to find the person that caused the damage to my car. I wasn’t going to be able to hold anyone accountable. As Bill and Ted would say: “Bogus”.
Network time sync helps us correlate sequences during cyber incidents and fix root causes
As you can see from my car damage analogy, accurate and correlated time keeping was crucial for my investigation. Or, I should say, lack of time accuracy and correlation was crucial to the failure of my investigation.
When it comes to protecting CUI, the DoD doesn’t have room for failure. NIST 800-171 and CMMC require us to ensure network time synchronization so that we can correlate any number of a multitude of events in our environment. This will help the DoD hold the bad guys accountable when those events are associated with malicious activity.
The DoD also wants us to be able to use synchronized logs from multiple sources to retrace the adversarial activity backwards in time, to discover the original point(s) of entry and fix the root cause(s) that allowed the adversary to attack in the first place.
For example, we’ll use a SIEM to monitor logs to alert us if an adversary attempts to exfiltrate CUI from our environment. Let’s say that during the alert investigation these logs indicate that CUI was compromised on a particular workstation. We’ll then want to compare that workstation’s logs to other workstations and servers in our environment to see if similar activity took place anywhere else. We’ll continue our investigation of logs from multiple components — workstations, servers, firewalls, network switches — to determine how the adversary got access to that workstation in the first place. If those logs indicate that, say, an employee clicked on a link in a phishing email that launched malware that took advantage of a missing patch on that workstation, then we have some root cause fixing to do!
So how do you do network time synchronization?
The good news is that configuring your IT systems for synchronized network time is straightforward. Note that control 3.3.7 requires an “authoritative source”. So first, you’ll want to identify an authoritative source of time your network components can sync to, using the NTP protocol. We suggest the NIST Internet Time Service, which is accessed through a pool of Internet-based servers at time.nist.gov.
(Out of the box, Windows machines get NTP from time.windows.com, and Apple products get it from time.apple.com. These aren’t bad time sources, but not as central and authoritative as the NIST service. There are plenty of poor choices for NTP service out there, so pay attention to how your components come configured. For instance, we discovered that a very popular commercial-grade wireless access point router was configured by the manufacturer to sync time to a server pool in Russia. The horror!)
You could point each individual IT component (workstation, server, wireless router, firewall, etc.) independently to use the NIST time service, but a more common practice is to synchronize all internal components to an internal source, which is in turn sync’d to NIST as the authoritative external source. Commonly, a Windows Domain Controller is designated as the internal time server. The Domain Controller receives its time from the NIST servers (using NTP across the Internet), and subsequently all other IT components in the domain sync their network time with the Domain Controller (again, using NTP, but this time locally). The advantage of this scenario is that the Domain Controller is usually installed on server hardware that has a better CMOS clock than the average laptop, router, or firewall, and so if Internet connection is lost and the NIST servers are not reachable, the internal domain can all stay synchronized, at least for a few days. In this manner we can ensure network time synchronization for incident response, which as we’ve seen is a crucial capability for CMMC compliance.
A nice by-product of synchronizing network time this way is that you can use network traffic analysis to discover unauthorized or “rogue” devices on your network. (I learned this trick in a SANS Institute course, BTW) After you configure your domain NTP system as described above, ensure your Network Traffic Analysis (NTA) or Intrusion Detection System (IDS) is monitoring all NTP traffic. (You do have NTA / IDS in place, correct? If not, you’ll want to read up on the NIST 800-171 System and Information Integrity family of controls and get to work). If you’ve setup the NTP system properly, any NTP traffic that shows up in NTA / IDS that is not destined either for time.nist.gov or the Domain Controller is coming from a rogue device! Then you just have to track down that device by IP or MAC address and get it off your network 😉 “Excellent!”
Wrapping up
Synchronized network time is crucial for the proper function of security applications. It’s also a crucial enabler of cyber incident identification, response, and recovery. So crucial in fact, that network time sync is required by CMMC and the NIST 800-171 standard that CMMC is built around.
It’s pretty simple to setup network time synchronization, and we cover this topic in-depth in our DFARS/CMMC Workshops. So, if you’d like to learn more or have any questions about network time sync, we’d love you to join us.
If you have any questions on DFARS, NIST 800-171, or CMMC in general, please drop us a line.
Good Hunting!
–Adam
