The Precision Time Protocol, PTP, is used for synchronizing time across the network. But what if the clocks I want to synchronize are all internal?
In previous posts, I have explained parts of PTP and how that is used to
synchronize devices in a network. PTP is often done with the ptp4l daemon from
the Linuxptp project, and it is overall a well-explained protocol across the
web. There are many resources. The project also provides a couple of other
tools, one of which is ts2phc, which has way less documentation about it
online. This post will focus on what it does and why it’s needed.
Why is it needed?
Some network switches will only have a single PHC that is shared across all
ports. In this case, no synchronization is needed. If instead the PHC and
timestamping are located in the PHY there is one PHC per port (or 2/4 ports for
dual/quad PHY). When running PTP across multiple PHYs their time needs to be
synchronized. For a Transparent Clock, they will never be adjusted by PTP, but
they must still have the same time. Running ts2phc on them will ensure that
they always stay within a few nanoseconds of each other. For a Boundary Clock,
the time of one PHY will constantly be adjusted by PTP (the port towards the PTP
Grandmaster), while the other ports act as masters for downstream clocks.
Internally the PHY towards the Grandmaster will act as a master for the other
PHY clocks to make sure that the time transmitted downstream is in sync with
upstream.
For Transparent Clocks, a single alignment pulse can sometimes be used to make all clocks start their time counting at the same time, and then never need to be synchronized again. But this depends on the PHYs and there is no official support for this in Linux and requires all PHYs to run on the same oscillator.
How synchronization works
Internal clocks do not have any network connections between them, so reusing the
PTP daemon isn’t an option. The naive approach would be to read out the current
time from each clock, compare them, and adjust the clocks accordingly. This is
what the Linuxptp application phc2sys does. But as the name suggests the
purpose of it is to synchronize the time of a Physical Hardware Clock (PHC) to
the system. Or vice versa if the system acts as the Grandmaster clock of the
network. As for synchronizing two PHCs this simply isn’t good enough. The
precision will be unacceptable compared to the precision you can reach with PTP;
a Boundary/Transparent clock that does this will probably introduce more
inaccuracy than if it just forwarded the PTP packet in hardware.
To the rescue comes the Linuxptp application ts2phc. This application needs
some specialized hardware design to work. The issue with phc2sys is that
comparing timestamps that are taken at different times inherently gives
inaccurate results. For perfect synchronization, the timestamps must be taken as
close together as possible. The solution for this is a signal line in the PCB.
Each PHC is set up by ts2phc to take a timestamp when a pulse is received on
that line. A pulse is then triggered and timestamps are read out from all PHC,
compared, and adjusted.
On the hardware, the different PHCs need to be connected on a single line, as shown below. One of the clocks will be configured to send out the pulse, while the other two are configured to timestamp the pulse. The line may be subject to some propagation delay on large boards. Online resources suggest 1 nanosecond per 15cm1. Which for normal PTP operation probably isn’t enough to worry about.
+-------+-------+
| | |
+----+ +----+ +----+
|PHC1| |PHC2| |PHC3|
+----+ +----+ +----+
Linux kernel API
The Linux kernel exposes two APIs for this purpose. EXTTS (EXTernal TimeStamp) configures the PHC to timestamp on a signal on a specific hardware pin. PEROUT (PERiodic OUTput) configures a PPS (Pulse-Per-Second) signal on one of the PHCs that will trigger every time the PHC is at a whole second time.
The PHC dedicated as master will be configured with PEROUT, while the rest are EXTTS. Because the PPS is always triggered at a whole second it can find the timestamp of the master PHC by reading the current time and rounding down, assuming we do it within a second (i.e. before the next signal). In theory, however, there is no requirement for the pulse to happen every second. It’s fully possible to have something external trigger a pulse to all PHC, including the master (the master also needs to be configured for EXTTS in this case). It’s also possible for the PHC sending PPS to be treated as slave and be adjusted. What is important is that there is a timestamp from every clock that we can compare.
Combining it with PTP
For every setup there needs to be one master PHC and the rest are slaves. The
slaves are compared to and adjusted to match the time of the master. ts2phc
expects one PHC to be configured as master. But there are cases when a dynamic
setup is required, e.g. when running a PTP Boundary Clock across the PHCs. The
master PHC must always be the one acting as PTP slave, the one in the direction
of the Grandmaster, as this is the one being adjusted from an external source.
This time must be distributed to all other PHCs for the time to continue being
accurate downstream. If the Grandmaster changes position and another PHC is now
being adjusted by PTP then ts2phc also requires changing master. To do this the
command line argument ts2phc -a can be used to listen to port state events
from ptp4l. Now ts2phc will always match its master to the PTP slave port.
It may also be useful to set the option ts2phc --step_threshold=0.0001 to
allow stepping the clocks. Otherwise, the clocks can end up stabilizing
internally before a Grandmaster is found, and will then take a long time to
converge.