Discussion:
[Linuxptp-devel] System clock synchronization
Miroslav Lichvar
2012-08-29 18:53:49 UTC
Permalink
Hi,

I'd like to ask some questions about system clock synchronization and
the future of linuxptp. I haven't found a todo list or roadmap, please
let me know if there is one.
Keller, Jacob E
2012-08-29 22:26:24 UTC
Permalink
-----Original Message-----
Sent: Wednesday, August 29, 2012 11:54 AM
Subject: [Linuxptp-devel] System clock synchronization
Hi,
I'd like to ask some questions about system clock synchronization and the
future of linuxptp. I haven't found a todo list or roadmap, please let me
know if there is one.
Richard Cochran
2012-08-30 06:13:22 UTC
Permalink
I would actually like to know how you got phc2sys to work. When I
tried to use it, it didn't seem to actually do what it advertised.
You can try it this way.

1. start ptp4l
2. wait until after the time jumps
3. start phc2sys

phc2sys -s /dev/ptp0 # without PHC PPS
phc2sys -s /dev/ptp0 -d /dev/pps0 # with PHC PPS
For the intel parts, it is supposed to be possible to get a smaller
pulse, but I have been having trouble with the PPS signals causing
side effects, so they probably won't be implemented.
Thats too bad. Didn't you have this working for ixgbe already?

I have been thinking about adding a PHC ioctl to do timecompare on
demand. This should allow a better sys/phc offset estimation for cards
without the PHC PPS feature.

Thanks,
Richard
Keller, Jacob E
2012-08-30 07:27:59 UTC
Permalink
-----Original Message-----
Sent: Wednesday, August 29, 2012 11:13 PM
To: Keller, Jacob E
Subject: Re: [Linuxptp-devel] System clock synchronization
I would actually like to know how you got phc2sys to work. When I
tried to use it, it didn't seem to actually do what it advertised.
You can try it this way.
1. start ptp4l
2. wait until after the time jumps
3. start phc2sys
phc2sys -s /dev/ptp0 # without PHC PPS
phc2sys -s /dev/ptp0 -d /dev/pps0 # with PHC PPS
Ok, last time I tried this I didn't seem to get the results I was wanting. I'll give it a go again
For the intel parts, it is supposed to be possible to get a smaller
pulse, but I have been having trouble with the PPS signals causing
side effects, so they probably won't be implemented.
Thats too bad. Didn't you have this working for ixgbe already?
I have been thinking about adding a PHC ioctl to do timecompare on demand.
This should allow a better sys/phc offset estimation for cards without the
PHC PPS feature.
It works for one of the ixgbe parts, but has some side-effects on performance that are unexpected and I haven't been able to root-cause it, so right now turning it on will cut network performance. I am having issues figuring out exactly why the performance issue occurs.

Other parts can't enable it because of a hardware limitation, since the pins exist but there is no way to translate that pin change into the software. Only one of the parts allows getting an interrupt when the pin changes. The other parts don't allow an interrupt when the pin triggers, so it would require special hardware.
Thanks,
Richard
Richard Cochran
2012-08-30 06:01:26 UTC
Permalink
Post by Miroslav Lichvar
Hi,
I'd like to ask some questions about system clock synchronization and
the future of linuxptp. I haven't found a todo list or roadmap, please
let me know if there is one.
There is no readmap, but you are asking the right questions. In
general, the issues you raise are important ones, and I think it will
require some thought and creativity in order to get this right.
Richard Cochran
2012-08-30 06:16:17 UTC
Permalink
- Will phc2sys be integrated into ptp4l? If no, how is phc2sys
supposed to be started to avoid syncing to an unsynced PHC?
I want to keep them separate if at all possible. One idea I had was to
define hooks for ptp4l to exec at important events.
Another idea would be for phc2sys to become a service. It could query
the PTP state using management messages.
BTW, the phc2sys progam/service will also have to query the
timePropertiesDS via management in order to find out the UTC offset,
too.

Thanks,
Richard
Miroslav Lichvar
2012-08-30 18:36:18 UTC
Permalink
Keller, Jacob E
2012-08-30 18:55:35 UTC
Permalink
-----Original Message-----
Sent: Wednesday, August 29, 2012 11:01 PM
To: Miroslav Lichvar
Subject: Re: [Linuxptp-devel] System clock synchronization
- Are the changes in PHC frequency done via clock_adjtime() applied
immediately or is there some unknown random delay like in the system
adjtimex()?
The changes are immediate.
Note that this may depend on the driver in question, because it would be possible (though wrong) to implement a PHC driver that doesn't apply adjtime instantly.
Richard Cochran
2012-08-31 06:32:35 UTC
Permalink
Post by Keller, Jacob E
Note that this may depend on the driver in question, because it would be possible (though wrong) to implement a PHC driver that doesn't apply adjtime instantly.
Oh yeah, I forgot, the PHYTER is sitting behind an mdio bus, and
writing the adjustment value to it will take a few dozen microseconds.

Thanks,
Richard
Richard Cochran
2012-08-31 06:27:14 UTC
Permalink
One point of my paper is that, considering user space scheduling
delays, this performance should be acceptable. The error associated
with just reading the clock dwarfs the PHC-sys error.
Reading of the TSC clock is just couple of tens of nanoseconds,
with no context switching. I agree it's not very useful to have such
precise clock in most applications, but I think it would be nice to be
able to say it's also accurate :).
It does not matter that reading the tsc via vdso is fast. There is a
fundamental problem with using time in applications. Any program can
be interrupted by an ISR (and then preempted by the scheduler) at any
time.

Consider what happens in the following example.

1 if (interesting_event) {
2 /* ??? */
3 getttimeofday():
4 }

If the program is interrupted at line 2, then the duration of the
interruption (ranging from dozens of microseconds to many
milliseconds) is an error in the time stamp.

Therfore, the overall real time behavior of the system has a direct
effect on application time accuracy. The single microsecond time error
due to the PHC/SYS servo is insignificant when compared with the
overall system latencies.
Ok, what if the PPS signal was used only to slowly (but accurately)
measure the error in the clock_gettime() readings? PHC allows us to
get as many readings and as often as we want. If someone wants very
accurate clock no matter what cost, phc2sys can make many PHC readings
(fixed by the measured offset) at high rate, do some filtering, and
quickly correct the system clock.
With the patches I just sent phc2sys -s /dev/ptp0 -R 100 -N 50 seems
to be giving pretty good results.
What do you think?
I will give it a try.

(BTW, when deriving PI constants, one of the inputs is the sample
rate. So, if you change that, then might need to change the weights as
well).
- Is anyone working or planning to work on a better servo?
Not me. You can of course adjust the PI weights to your taste. I would
not expect any magic silver bullet here.
Finding out good PI constants is the problem. They may even need to be
changed in runtime, e.g. during rapid thermal changes in the
oscillator. It would be nice if it could adapt to the stability of the
local clock and the jitter in the input.
The constants may be arrived at analytically, with a guarantee of
stability. The defaults of .7 and .3 have a fairly aggressive settling
time of 15 seconds. The constants from the paper are slower, at about
150 seconds.

I am not any kind of signal processing expert, and I like to stick
with the tried and true. Since the system clock only wanders slowly
with ambient temperature, I doubt very much that a dynamic sevro will
improve things very much.

But having the servo in a simple program (and not in kernel space)
like phc2sys makes it very easy to try out new ideas, so go ahead and
have fun with it.
One idea would be to try the hardpps in kernel space, but that
does require that the PHC provide PPS events.
IIRC the kernel PPS discipline is just a FLL with almost no sample
filtering. Is there any advantage to have it in the kernel instead of
using adjtimex()?
I am familiar will the original nanokernel implementation of
hardpps. It looks like the person who added the Linux variant removed
a bunch of filtering code. In any case, the original hardpps provides
both the frequency and phase offset to the main NTP servo.

The advantage would be to use the kernel's NTP servo. People do trust
it, even if no one really understands it.

Thanks,
Richard
Miroslav Lichvar
2012-09-05 19:25:36 UTC
Permalink
Post by Richard Cochran
Consider what happens in the following example.
1 if (interesting_event) {
2 /* ??? */
4 }
If the program is interrupted at line 2, then the duration of the
interruption (ranging from dozens of microseconds to many
milliseconds) is an error in the time stamp.
Therfore, the overall real time behavior of the system has a direct
effect on application time accuracy. The single microsecond time error
due to the PHC/SYS servo is insignificant when compared with the
overall system latencies.
I think I understand what you are saying here, but I'm still not
convinced there are no applications where only a minimal or average
latency would matter.

At least, it should be useful for the kernel itself. For example, it
would allow us to make a high quality PPS master synced to UTC or
measure the latency of the serial port I'm using for the GPS PPS and
compensate for it.
Post by Richard Cochran
The constants may be arrived at analytically, with a guarantee of
stability. The defaults of .7 and .3 have a fairly aggressive settling
time of 15 seconds. The constants from the paper are slower, at about
150 seconds.
I am not any kind of signal processing expert, and I like to stick
with the tried and true. Since the system clock only wanders slowly
with ambient temperature, I doubt very much that a dynamic sevro will
improve things very much.
I'm not an expert either, I just like to try new ideas and play with
simulations. I really like the simplicity of the PI controller.

In case you are interested in how it compares to some other
algorithms, here are results of simulations with the NTP
implementations ntpd and chrony using a SHM reference clock with poll
set to 8 seconds (the SHM is read every second) and the phc2sys
program reading a PHC once per second in two settings of the PI
constants. The SHM and PHC jitter was set to 1 microsecond with normal
distribution and the clock wander changed from 0.01 ppb/s to 100
ppb/s. The error in time and frequency of the system clock was
monitored.

Loading Image...
Loading Image...
Post by Richard Cochran
The advantage would be to use the kernel's NTP servo. People do trust
it, even if no one really understands it.
Some don't trust it. The kernel discipline is using 4x shorter PLL
time constant than the reference implementation, against the advice of
Dr. Mills. I think the problem is that it was designed many years ago,
when typical network jitters were orders of magnitude larger, while
the clock stability hasn't changed much or could be even worse due to
more power hungry devices inside current computers. The Allan
intercept is much shorter than it was before.
--
Miroslav Lichvar
Richard Cochran
2012-09-06 06:20:04 UTC
Permalink
Post by Miroslav Lichvar
Post by Richard Cochran
Therfore, the overall real time behavior of the system has a direct
effect on application time accuracy. The single microsecond time error
due to the PHC/SYS servo is insignificant when compared with the
overall system latencies.
I think I understand what you are saying here, but I'm still not
convinced there are no applications where only a minimal or average
latency would matter.
I am trying to say that the system latencies (like ISR latency which
ranges from single to dozens of microseconds and depends on system
activity) will always introduce an error. You can filter, estimate,
and finagle all you want, but without an independent method of
comparing the system clock (like HW pps), you can never really know
what you have achieved, since the way that you read the clock
re-introduces the same errors that you supposedly filtered out.

Also, it does no good to say, "on average, under idle load, the time
error is so and so." You are only fooling yourself or your customers
with such claims. Instead, you need to specify the worst case
error. The worst case *will* appear in normal usage.

For example, consider time stamping a series of "interesting events"
that occur at irregular intervals.

Q: What is the error in the time stamps?
A: Several milliseconds.

You can't tell which of the time stamps have the average error, and
which have the large error, so you must assume the larger error.

A similar argument applies when talking about application timers.

One of the main points of the paper was that the pps method provides
time that is "good enough" for most application. Other applications
with special needs can simply use clock_gettime on the PHC device.
Post by Miroslav Lichvar
At least, it should be useful for the kernel itself. For example, it
would allow us to make a high quality PPS master synced to UTC or
measure the latency of the serial port I'm using for the GPS PPS and
compensate for it.
With a lot effort, you can characterize the PPS jitter on your
particular serial port, with your particular motherboard, and your
particular kernel version, etc. That is all well and good, but it
requires some expert knowledge. It doesn't help us to offer a general
purpose solution.

I appreciate what you are doing with the phc2sys program, and I
welcome any improvements. Still, I am bit sceptical about achieving
better results than a few microsecond time error for the system clock.
Post by Miroslav Lichvar
In case you are interested in how it compares to some other
algorithms, here are results of simulations with the NTP
implementations ntpd and chrony using a SHM reference clock with poll
set to 8 seconds (the SHM is read every second) and the phc2sys
program reading a PHC once per second in two settings of the PI
constants. The SHM and PHC jitter was set to 1 microsecond with normal
distribution and the clock wander changed from 0.01 ppb/s to 100
ppb/s. The error in time and frequency of the system clock was
monitored.
http://mlichvar.fedorapeople.org/tmp/phc2sys_offset.png
http://mlichvar.fedorapeople.org/tmp/phc2sys_freq.png
Those are interesting looking curves. Can you explain what they show?

What is the SHM ref clock? Is it the NTP driver?
What is the X axis?
How did you measure the RMS error?
How do you set the jitter?
How do you change the wander?

I really don't get it :(
Post by Miroslav Lichvar
Post by Richard Cochran
The advantage would be to use the kernel's NTP servo. People do trust
it, even if no one really understands it.
Some don't trust it. The kernel discipline is using 4x shorter PLL
time constant than the reference implementation, against the advice of
Dr. Mills. I think the problem is that it was designed many years ago,
when typical network jitters were orders of magnitude larger, while
Yep. Although wireless and power modem networks are still pretty bad.
The machines are lot faster, too. This makes a difference for the pps
measurements.
Post by Miroslav Lichvar
the clock stability hasn't changed much or could be even worse due to
more power hungry devices inside current computers. The Allan
intercept is much shorter than it was before.
Thanks,
Richard
Miroslav Lichvar
2012-09-06 16:15:35 UTC
Permalink
This post might be inappropriate. Click to display it.
Richard Cochran
2012-09-06 18:49:48 UTC
Permalink
Post by Miroslav Lichvar
I was just trying to show you that the PI constants are important and
if not chosen well they can make both the time and frequency accuracy
significantly worse.
http://mlichvar.fedorapeople.org/clknetsim/
Okay, now I get it, thanks. Your simulator sounds cool, and I will
have to try it out. Now the graphs make sense. It sure looked like a
simulated clock, but I didn't know that such a thing existed, so
thought it must have been some kind of real life measurement.

Thanks,
Richard

Loading...