Ahmad Byagowi, the project lead for Open Compute Project's Time Appliance, reached out to me a couple weeks ago and asked if I'd be willing to test the new Time Card Facebook had announced in mid-August on a Raspberry Pi Compute Module 4. Since I have a sort of obsession with plugging anything and everything into a Pi to see what works and what doesn't, I took him up on the offer.
The official specs had PCI Express Gen 3 on a x4 slot as a requirement, but it seems the Gen 3 designation is a little loose—the card and its driver should work fine on an older Gen 2 bus—like the one the Raspberry Pi Compute Module 4 exposes if you use the official IO Board:
The slot is x1, but you can plug in any width card using an adapter like this one or by hacking an open end into it with a razor saw or dremel tool.
The Time Card
I won't get deep into the details of the Time Card—check out Facebook's post linked in the first paragraph above, or the open source Time Card spec on GitHub—but it's basically a PCI Express card that turns any computer into a Stratum 1 time server.
That is, it has a built-in GNSS receiver that can acquire time from GPS, GLONASS, or any of the other positioning satellite networks, accurate to the tens of nanoseconds. Then it includes a Rubidium oscillator—the MAC from Microchip, in my card's case—to hold time accurate to a few nanoseconds for up to a couple days. And to tie everything together, there's an FPGA that interfaces with the PCIe bus and incorporates 4 PPS I/O ports.
I go into a lot more detail, along with a full hardware walkthrough, in the video I just published, The Time Card makes the most accurate Raspberry Pi clock EVER!.
But the big thing the Time Appliance Project brings to the table is the ability to build a 'grandmaster' clock source based on an open design. Basically, a server that distributes extremely precise time to other computers and devices on the same network, using the Precision Time Protocol (PTP). These kinds of devices existed before, but usually they are more expensive, less standardized (in terms of integration into a rack or data center), and run proprietary software that requires vendor support.
The Time Appliance Project (and the Time Card, by extension) aims to make a standard for 'time appliances' that is more inexpensive than existing solutions and runs on an open source software stack.
PTP and the Pi
PTP allows for a local network to have a time reference more precise than NTP, while not requiring every endpoint on the network to have its own GPS reference.
For the highest level of accuracy, PTP requires hardware timestamping support on the computer's NIC. Many popular NICs have timestamping support—for example, the Intel I340 that I've already tested on the Compute Module 4:
But here's something you might not know: the Compute Module 4's built-in NIC, a Broadcom BCM54210PE PHY, is actually capable of hardware timestamping—unlike the BCM54213PE used in the Pi 4 model B!
However, Ahmad pointed me to this open issue: CM4 is missing IEEE1588-2008 support through BCM54210PE. So for now, hardware timestamping on the CM4 has to be routed through an external PCI Express NIC that supports it.
Time Card Driver
As part of my work testing the card for my Pi PCI Express Card Database, I discovered the driver in the Time Card project on GitHub won't compile on the Pi's default kernel version (5.10), due to some driver-specific functionality that requires 5.11 or newer just to compile. It seems that the driver recommends 5.12 or newer, and if you compile the latest 5.14 or 5.15 kernels, you can add support natively, as the ptp_ocp driver is now part of the kernel source!
And so that's what I did. Using my Linux cross-compile environment I cloned the Raspberry Pi Linux source tree, checked out the 5.14 branch, and compiled a kernel with the following enabled in menuconfig:
Device Drivers
> PTP clock support
> OpenCompute TimeCard as PTP clock
After copying over my custom kernel, I rebooted the Pi, and dmesg showed the driver initializing correctly:
pi@cm4:~ $ dmesg | grep ptp
[ 4.425124] ptp_ocp 0000:01:00.0: enabling device (0000 -> 0002)
[ 4.425224] ptp_ocp 0000:01:00.0: Time: 3603.563280960, UNSYNCED
[ 4.427082] ptp_ocp 0000:01:00.0: Version 1.2.0, clock PPS, device ptp0
[ 4.427123] ptp_ocp 0000:01:00.0: TOD Version 2.0.1
[ 4.427143] ptp_ocp 0000:01:00.0: control: 10000001
[ 4.427156] ptp_ocp 0000:01:00.0: TOD Protocol UBX enabled
[ 4.427171] ptp_ocp 0000:01:00.0: GNSS ALL
[ 4.427187] ptp_ocp 0000:01:00.0: status: 0
[ 4.427202] ptp_ocp 0000:01:00.0: correction: 0
[ 4.427216] ptp_ocp 0000:01:00.0: utc_status: 0
[ 4.427229] ptp_ocp 0000:01:00.0: utc_offset: 0 valid:0 leap_valid:0
But as I mentioned, the CM4 itself doesn't support hardware timestamping yet. I'll be doing more testing with the card (I haven't even put it in a spot where I can get GPS reception yet!), and I'm hopeful the CM4 could be used as a Time Appliance / grandmaster clock so people could have an easier and less expensive option for highly accurate timing on their networks!
It's not only useful in the data center—a good clock source can be used in media production, industrial automation, 5G, heck, maybe even in coordinating turn signals someday!
Be sure to check out my YouTube video on the Time Card for more details and stay tuned—this isn't the last you'll see of the Pi and Time Card on the blog!
Comments
One can use the bounceback test to check if the clock's are not synchronized relative to the bb time. (you'll need to compile the latest iperf from master)
Below is an example:
root@raspberrypi:/usr/local/src/iperf2-code# iperf -c 192.168.1.33 -e -i 1 --trip-times --bounceback --bounceback-period 0
------------------------------------------------------------
Client connecting to 192.168.1.33, TCP port 5001 with pid 38489 (1/0 flows/load)
Bounceback test (req/reply size = 100 Byte/ 100 Byte) (server hold req=0 usecs & tcp_quickack)
TCP congestion control using cubic
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 85.0 KByte (default)
Event based writes (pending queue watermark at 16384 bytes)
------------------------------------------------------------
[ 1] local 192.168.1.32%eth0 port 42258 connected with 192.168.1.33 port 5001 (prefetch=16384) (bb w/quickack req/reply/hold=100/100/0) (trip-times) (sock=3) (icwnd/mss/irtt=14/1448/265) (ct=0.43 ms) on 2023-10-18 14:49:24.047 (PDT)
[ ID] Interval Transfer Bandwidth BB cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS(avg)
[ 1] 0.00-1.00 sec 998 KBytes 8.18 Mbits/sec 10223=0.093/0.078/1.057/0.027 ms 0 14K/62 us 10729 rps
[ 1] 1.00-2.00 sec 1.06 MBytes 8.93 Mbits/sec 11166=0.086/0.077/0.225/0.003 ms 0 14K/61 us 11631 rps
[ 1] 2.00-3.00 sec 1.07 MBytes 8.94 Mbits/sec 11172=0.086/0.077/0.434/0.004 ms 0 14K/60 us 11633 rps
[ 1] 3.00-4.00 sec 1.06 MBytes 8.87 Mbits/sec 11092=0.087/0.079/0.376/0.005 ms 0 14K/62 us 11547 rps
[ 1] 4.00-5.00 sec 979 KBytes 8.02 Mbits/sec 10025=0.096/0.090/0.442/0.004 ms 0 14K/61 us 10402 rps
[ 1] 5.00-6.00 sec 960 KBytes 7.86 Mbits/sec 9831=0.098/0.090/0.413/0.008 ms 0 14K/61 us 10213 rps
[ 1] 6.00-7.00 sec 984 KBytes 8.06 Mbits/sec 10080=0.096/0.090/0.150/0.002 ms 0 14K/61 us 10461 rps
[ 1] 7.00-8.00 sec 983 KBytes 8.06 Mbits/sec 10070=0.096/0.090/0.168/0.002 ms 0 14K/61 us 10452 rps
[ 1] 8.00-9.00 sec 984 KBytes 8.06 Mbits/sec 10074=0.096/0.092/0.149/0.002 ms 0 14K/61 us 10455 rps
[ 1] 9.00-10.00 sec 982 KBytes 8.04 Mbits/sec 10056=0.096/0.087/0.446/0.004 ms 0 14K/64 us 10434 rps
[ 1] 0.00-10.01 sec 9.90 MBytes 8.29 Mbits/sec 103791=0.093/0.077/1.057/0.011 ms 0 14K/1729 us 10795 rps
[ 1] 0.00-10.01 sec OWD (ms) Cnt=103791 TX=0.770/-0.357/4.012/1.683 RX=-0.678/-3.709/0.789/1.686 Asymmetry=2.299/0.001/7.702/2.857
[ 1] 0.00-10.01 sec OWD-TX-PDF: bin(w=100us):cnt(103791)=1:14920,2:263,3:275,4:271,5:273,6:272,7:272,8:272,9:271,10:272,11:270,12:275,13:273,14:272,15:269,16:272,17:272,18:272,19:272,20:273,21:271,22:269,23:273,24:271,25:272,26:273,27:273,28:270,29:272,30:271,31:272,32:271,33:273,34:270,35:271,36:272,37:269,38:20475,39:625,40:2,41:1 (5.00/95.00/99.7%=1/100000/100000,Outliers=0,obl/obu=57994/0)
[ 1] 0.00-10.01 sec OWD-RX-PDF: bin(w=100us):cnt(103791)=1:12175,2:3139,3:3132,4:3136,5:51552,6:5,8:2 (5.00/95.00/99.7%=1/100000/100000,Outliers=0,obl/obu=30650/0)
[ 1] 0.00-10.01 sec BB8-PDF: bin(w=100us):cnt(103791)=1:100388,2:3136,3:258,4:4,5:4,11:1 (5.00/95.00/99.7%=1/1/2,Outliers=0,obl/obu=0/0)
[ 1] 0.00-10.01 sec Clock sync error count = 92028
Below is an example where the clock are sync'd - there is no Clock sync error count message
root@raspberrypi:/usr/local/src/iperf2-code# iperf -c 192.168.1.33 -e -i 1 --trip-times --bounceback --bounceback-period 0
------------------------------------------------------------
Client connecting to 192.168.1.33, TCP port 5001 with pid 38492 (1/0 flows/load)
Bounceback test (req/reply size = 100 Byte/ 100 Byte) (server hold req=0 usecs & tcp_quickack)
TCP congestion control using cubic
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 85.0 KByte (default)
Event based writes (pending queue watermark at 16384 bytes)
------------------------------------------------------------
[ 1] local 192.168.1.32%eth0 port 46112 connected with 192.168.1.33 port 5001 (prefetch=16384) (bb w/quickack req/reply/hold=100/100/0) (trip-times) (sock=3) (icwnd/mss/irtt=14/1448/291) (ct=0.46 ms) on 2023-10-18 14:51:27.555 (PDT)
[ ID] Interval Transfer Bandwidth BB cnt=avg/min/max/stdev Rtry Cwnd/RTT RPS(avg)
[ 1] 0.00-1.00 sec 1003 KBytes 8.22 Mbits/sec 10270=0.093/0.079/1.218/0.025 ms 0 14K/61 us 10775 rps
[ 1] 1.00-2.00 sec 1.06 MBytes 8.93 Mbits/sec 11166=0.086/0.078/0.322/0.005 ms 0 14K/60 us 11628 rps
[ 1] 2.00-3.00 sec 1.07 MBytes 8.94 Mbits/sec 11175=0.086/0.078/0.263/0.003 ms 0 14K/61 us 11635 rps
[ 1] 3.00-4.00 sec 1.06 MBytes 8.93 Mbits/sec 11167=0.086/0.078/0.313/0.004 ms 0 14K/60 us 11630 rps
[ 1] 4.00-5.00 sec 998 KBytes 8.18 Mbits/sec 10224=0.094/0.080/0.410/0.006 ms 0 14K/62 us 10614 rps
[ 1] 5.00-6.00 sec 958 KBytes 7.85 Mbits/sec 9811=0.098/0.088/0.432/0.009 ms 0 14K/61 us 10187 rps
[ 1] 6.00-7.00 sec 982 KBytes 8.05 Mbits/sec 10060=0.096/0.090/0.306/0.003 ms 0 14K/61 us 10437 rps
[ 1] 7.00-8.00 sec 980 KBytes 8.03 Mbits/sec 10035=0.096/0.090/0.927/0.015 ms 0 14K/61 us 10409 rps
[ 1] 8.00-9.00 sec 981 KBytes 8.04 Mbits/sec 10048=0.096/0.090/0.763/0.010 ms 0 14K/61 us 10424 rps
[ 1] 9.00-10.00 sec 982 KBytes 8.04 Mbits/sec 10054=0.096/0.092/0.287/0.003 ms 0 14K/61 us 10432 rps
[ 1] 0.00-10.01 sec 9.92 MBytes 8.31 Mbits/sec 104011=0.092/0.078/1.218/0.012 ms 0 14K/1009 us 10816 rps
[ 1] 0.00-10.01 sec OWD (ms) Cnt=104011 TX=0.040/0.030/0.680/0.005 RX=0.053/0.045/0.887/0.008 Asymmetry=0.013/0.000/0.847/0.006
[ 1] 0.00-10.01 sec OWD-TX-PDF: bin(w=100us):cnt(104011)=1:103967,2:31,3:9,4:2,6:1,7:1 (5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0)
[ 1] 0.00-10.01 sec OWD-RX-PDF: bin(w=100us):cnt(104011)=1:103769,2:226,3:13,4:1,8:1,9:1 (5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0)
[ 1] 0.00-10.01 sec BB8-PDF: bin(w=100us):cnt(104011)=1:100646,2:3146,3:198,4:10,5:4,6:1,7:1,8:2,10:2,13:1 (5.00/95.00/99.7%=1/1/2,Outliers=0,obl/obu=0/0)