Time Card and PTP on a Raspberry Pi Compute Module 4

Ahmad Byagowi, the project lead for Open Compute Project's Time Appliance, reached out to me a couple weeks ago and asked if I'd be willing to test the new Time Card Facebook had announced in mid-August on a Raspberry Pi Compute Module 4. Since I have a sort of obsession with plugging anything and everything into a Pi to see what works and what doesn't, I took him up on the offer.

The official specs had PCI Express Gen 3 on a x4 slot as a requirement, but it seems the Gen 3 designation is a little loose—the card and its driver should work fine on an older Gen 2 bus—like the one the Raspberry Pi Compute Module 4 exposes if you use the official IO Board:

Raspberry Pi Compute Module 4 IO Board PCI Express Slot

The slot is x1, but you can plug in any width card using an adapter like this one or by hacking an open end into it with a razor saw or dremel tool.

The Time Card

Time Card next to Raspberry Pi Compute Module 4 IO Board

I won't get deep into the details of the Time Card—check out Facebook's post linked in the first paragraph above, or the open source Time Card spec on GitHub—but it's basically a PCI Express card that turns any computer into a Stratum 1 time server.

That is, it has a built-in GNSS receiver that can acquire time from GPS, GLONASS, or any of the other positioning satellite networks, accurate to the tens of nanoseconds. Then it includes a Rubidium oscillator—the MAC from Microchip, in my card's case—to hold time accurate to a few nanoseconds for up to a couple days. And to tie everything together, there's an FPGA that interfaces with the PCIe bus and incorporates 4 PPS I/O ports.

I go into a lot more detail, along with a full hardware walkthrough, in the video I just published, The Time Card makes the most accurate Raspberry Pi clock EVER!.

But the big thing the Time Appliance Project brings to the table is the ability to build a 'grandmaster' clock source based on an open design. Basically, a server that distributes extremely precise time to other computers and devices on the same network, using the Precision Time Protocol (PTP). These kinds of devices existed before, but usually they are more expensive, less standardized (in terms of integration into a rack or data center), and run proprietary software that requires vendor support.

The Time Appliance Project (and the Time Card, by extension) aims to make a standard for 'time appliances' that is more inexpensive than existing solutions and runs on an open source software stack.

PTP and the Pi

PTP allows for a local network to have a time reference more precise than NTP, while not requiring every endpoint on the network to have its own GPS reference.

For the highest level of accuracy, PTP requires hardware timestamping support on the computer's NIC. Many popular NICs have timestamping support—for example, the Intel I340 that I've already tested on the Compute Module 4:

Intel I340 on Raspberry Pi Compute Module 4

But here's something you might not know: the Compute Module 4's built-in NIC, a Broadcom BCM54210PE PHY, is actually capable of hardware timestamping—unlike the BCM54213PE used in the Pi 4 model B!

CM4 vs Pi 4 model B Broadcom NIC Ethernet chip difference

However, Ahmad pointed me to this open issue: CM4 is missing IEEE1588-2008 support through BCM54210PE. So for now, hardware timestamping on the CM4 has to be routed through an external PCI Express NIC that supports it.

Time Card Driver

As part of my work testing the card for my Pi PCI Express Card Database, I discovered the driver in the Time Card project on GitHub won't compile on the Pi's default kernel version (5.10), due to some driver-specific functionality that requires 5.11 or newer just to compile. It seems that the driver recommends 5.12 or newer, and if you compile the latest 5.14 or 5.15 kernels, you can add support natively, as the ptp_ocp driver is now part of the kernel source!

And so that's what I did. Using my Linux cross-compile environment I cloned the Raspberry Pi Linux source tree, checked out the 5.14 branch, and compiled a kernel with the following enabled in menuconfig:

Device Drivers
  > PTP clock support
    > OpenCompute TimeCard as PTP clock

After copying over my custom kernel, I rebooted the Pi, and dmesg showed the driver initializing correctly:

pi@cm4:~ $ dmesg | grep ptp
[    4.425124] ptp_ocp 0000:01:00.0: enabling device (0000 -> 0002)
[    4.425224] ptp_ocp 0000:01:00.0: Time: 3603.563280960, UNSYNCED
[    4.427082] ptp_ocp 0000:01:00.0: Version 1.2.0, clock PPS, device ptp0
[    4.427123] ptp_ocp 0000:01:00.0: TOD Version 2.0.1
[    4.427143] ptp_ocp 0000:01:00.0: control: 10000001
[    4.427156] ptp_ocp 0000:01:00.0: TOD Protocol UBX enabled
[    4.427171] ptp_ocp 0000:01:00.0: GNSS ALL
[    4.427187] ptp_ocp 0000:01:00.0: status: 0
[    4.427202] ptp_ocp 0000:01:00.0: correction: 0
[    4.427216] ptp_ocp 0000:01:00.0: utc_status: 0
[    4.427229] ptp_ocp 0000:01:00.0: utc_offset: 0  valid:0  leap_valid:0

But as I mentioned, the CM4 itself doesn't support hardware timestamping yet. I'll be doing more testing with the card (I haven't even put it in a spot where I can get GPS reception yet!), and I'm hopeful the CM4 could be used as a Time Appliance / grandmaster clock so people could have an easier and less expensive option for highly accurate timing on their networks!

It's not only useful in the data center—a good clock source can be used in media production, industrial automation, 5G, heck, maybe even in coordinating turn signals someday!

Be sure to check out my YouTube video on the Time Card for more details and stay tuned—this isn't the last you'll see of the Pi and Time Card on the blog!

Comments

One can use the bounceback test to check if the clock's are not synchronized relative to the bb time. (you'll need to compile the latest iperf from master)

Below is an example:

root@raspberrypi:/usr/local/src/iperf2-code# iperf -c 192.168.1.33 -e -i 1 --trip-times --bounceback --bounceback-period 0
------------------------------------------------------------
Client connecting to 192.168.1.33, TCP port 5001 with pid 38489 (1/0 flows/load)
Bounceback test (req/reply size = 100 Byte/ 100 Byte) (server hold req=0 usecs & tcp_quickack)
TCP congestion control using cubic
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 85.0 KByte (default)
Event based writes (pending queue watermark at 16384 bytes)
------------------------------------------------------------
[  1] local 192.168.1.32%eth0 port 42258 connected with 192.168.1.33 port 5001 (prefetch=16384) (bb w/quickack req/reply/hold=100/100/0) (trip-times) (sock=3) (icwnd/mss/irtt=14/1448/265) (ct=0.43 ms) on 2023-10-18 14:49:24.047 (PDT)
[ ID] Interval        Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS(avg)
[  1] 0.00-1.00 sec   998 KBytes  8.18 Mbits/sec    10223=0.093/0.078/1.057/0.027 ms    0   14K/62 us    10729 rps
[  1] 1.00-2.00 sec  1.06 MBytes  8.93 Mbits/sec    11166=0.086/0.077/0.225/0.003 ms    0   14K/61 us    11631 rps
[  1] 2.00-3.00 sec  1.07 MBytes  8.94 Mbits/sec    11172=0.086/0.077/0.434/0.004 ms    0   14K/60 us    11633 rps
[  1] 3.00-4.00 sec  1.06 MBytes  8.87 Mbits/sec    11092=0.087/0.079/0.376/0.005 ms    0   14K/62 us    11547 rps
[  1] 4.00-5.00 sec   979 KBytes  8.02 Mbits/sec    10025=0.096/0.090/0.442/0.004 ms    0   14K/61 us    10402 rps
[  1] 5.00-6.00 sec   960 KBytes  7.86 Mbits/sec    9831=0.098/0.090/0.413/0.008 ms    0   14K/61 us    10213 rps
[  1] 6.00-7.00 sec   984 KBytes  8.06 Mbits/sec    10080=0.096/0.090/0.150/0.002 ms    0   14K/61 us    10461 rps
[  1] 7.00-8.00 sec   983 KBytes  8.06 Mbits/sec    10070=0.096/0.090/0.168/0.002 ms    0   14K/61 us    10452 rps
[  1] 8.00-9.00 sec   984 KBytes  8.06 Mbits/sec    10074=0.096/0.092/0.149/0.002 ms    0   14K/61 us    10455 rps
[  1] 9.00-10.00 sec   982 KBytes  8.04 Mbits/sec    10056=0.096/0.087/0.446/0.004 ms    0   14K/64 us    10434 rps
[  1] 0.00-10.01 sec  9.90 MBytes  8.29 Mbits/sec    103791=0.093/0.077/1.057/0.011 ms    0   14K/1729 us    10795 rps
[  1] 0.00-10.01 sec  OWD (ms) Cnt=103791 TX=0.770/-0.357/4.012/1.683 RX=-0.678/-3.709/0.789/1.686 Asymmetry=2.299/0.001/7.702/2.857
[  1] 0.00-10.01 sec  OWD-TX-PDF: bin(w=100us):cnt(103791)=1:14920,2:263,3:275,4:271,5:273,6:272,7:272,8:272,9:271,10:272,11:270,12:275,13:273,14:272,15:269,16:272,17:272,18:272,19:272,20:273,21:271,22:269,23:273,24:271,25:272,26:273,27:273,28:270,29:272,30:271,31:272,32:271,33:273,34:270,35:271,36:272,37:269,38:20475,39:625,40:2,41:1 (5.00/95.00/99.7%=1/100000/100000,Outliers=0,obl/obu=57994/0)
[  1] 0.00-10.01 sec  OWD-RX-PDF: bin(w=100us):cnt(103791)=1:12175,2:3139,3:3132,4:3136,5:51552,6:5,8:2 (5.00/95.00/99.7%=1/100000/100000,Outliers=0,obl/obu=30650/0)
[  1] 0.00-10.01 sec  BB8-PDF: bin(w=100us):cnt(103791)=1:100388,2:3136,3:258,4:4,5:4,11:1 (5.00/95.00/99.7%=1/1/2,Outliers=0,obl/obu=0/0)
[  1] 0.00-10.01 sec  Clock sync error count = 92028

Below is an example where the clock are sync'd - there is no Clock sync error count message

root@raspberrypi:/usr/local/src/iperf2-code# iperf -c 192.168.1.33 -e -i 1 --trip-times --bounceback --bounceback-period 0
------------------------------------------------------------
Client connecting to 192.168.1.33, TCP port 5001 with pid 38492 (1/0 flows/load)
Bounceback test (req/reply size = 100 Byte/ 100 Byte) (server hold req=0 usecs & tcp_quickack)
TCP congestion control using cubic
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 85.0 KByte (default)
Event based writes (pending queue watermark at 16384 bytes)
------------------------------------------------------------
[  1] local 192.168.1.32%eth0 port 46112 connected with 192.168.1.33 port 5001 (prefetch=16384) (bb w/quickack req/reply/hold=100/100/0) (trip-times) (sock=3) (icwnd/mss/irtt=14/1448/291) (ct=0.46 ms) on 2023-10-18 14:51:27.555 (PDT)
[ ID] Interval        Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS(avg)
[  1] 0.00-1.00 sec  1003 KBytes  8.22 Mbits/sec    10270=0.093/0.079/1.218/0.025 ms    0   14K/61 us    10775 rps
[  1] 1.00-2.00 sec  1.06 MBytes  8.93 Mbits/sec    11166=0.086/0.078/0.322/0.005 ms    0   14K/60 us    11628 rps
[  1] 2.00-3.00 sec  1.07 MBytes  8.94 Mbits/sec    11175=0.086/0.078/0.263/0.003 ms    0   14K/61 us    11635 rps
[  1] 3.00-4.00 sec  1.06 MBytes  8.93 Mbits/sec    11167=0.086/0.078/0.313/0.004 ms    0   14K/60 us    11630 rps
[  1] 4.00-5.00 sec   998 KBytes  8.18 Mbits/sec    10224=0.094/0.080/0.410/0.006 ms    0   14K/62 us    10614 rps
[  1] 5.00-6.00 sec   958 KBytes  7.85 Mbits/sec    9811=0.098/0.088/0.432/0.009 ms    0   14K/61 us    10187 rps
[  1] 6.00-7.00 sec   982 KBytes  8.05 Mbits/sec    10060=0.096/0.090/0.306/0.003 ms    0   14K/61 us    10437 rps
[  1] 7.00-8.00 sec   980 KBytes  8.03 Mbits/sec    10035=0.096/0.090/0.927/0.015 ms    0   14K/61 us    10409 rps
[  1] 8.00-9.00 sec   981 KBytes  8.04 Mbits/sec    10048=0.096/0.090/0.763/0.010 ms    0   14K/61 us    10424 rps
[  1] 9.00-10.00 sec   982 KBytes  8.04 Mbits/sec    10054=0.096/0.092/0.287/0.003 ms    0   14K/61 us    10432 rps
[  1] 0.00-10.01 sec  9.92 MBytes  8.31 Mbits/sec    104011=0.092/0.078/1.218/0.012 ms    0   14K/1009 us    10816 rps
[  1] 0.00-10.01 sec  OWD (ms) Cnt=104011 TX=0.040/0.030/0.680/0.005 RX=0.053/0.045/0.887/0.008 Asymmetry=0.013/0.000/0.847/0.006
[  1] 0.00-10.01 sec  OWD-TX-PDF: bin(w=100us):cnt(104011)=1:103967,2:31,3:9,4:2,6:1,7:1 (5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0)
[  1] 0.00-10.01 sec  OWD-RX-PDF: bin(w=100us):cnt(104011)=1:103769,2:226,3:13,4:1,8:1,9:1 (5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0)
[  1] 0.00-10.01 sec  BB8-PDF: bin(w=100us):cnt(104011)=1:100646,2:3146,3:198,4:10,5:4,6:1,7:1,8:2,10:2,13:1 (5.00/95.00/99.7%=1/1/2,Outliers=0,obl/obu=0/0)