[Linuxptp-devel] limitation on frequency offset tracking

[Makarand Kulkarni]

How do we come up with an upper bound on how much "jump" that can be
handled with PTP ?

On the platform we have, the counters are 62 bit, but when it comes to
adjusting frequency or offset, the whole servo gets unlocked and never
converges again.

[Richard Cochran]

Post by Makarand Kulkarni
How do we come up with an upper bound on how much "jump" that can be
handled with PTP ?

Not sure what you are asking here. The only limits on the jumping the
clock is what can be represented by the 'struct timespec'.

Post by Makarand Kulkarni
On the platform we have, the counters are 62 bit, but when it comes to
adjusting frequency or offset, the whole servo gets unlocked and never
converges again.

Are you saying that there is problem with the PI controller in ptp4l?

If so, what is the problem?

Thanks,
Richard

[Makarand Kulkarni]

Post by Makarand Kulkarni
On the platform we have, the counters are 62 bit, but when it comes to
adjusting frequency or offset, the whole servo gets unlocked and never
converges again.
Are you saying that there is problem with the PI controller in ptp4l?

No, it's the issue I see with phc2sys (which I believe shares the same
PI controller. If I try to correct for error when source is CLOCK_REALTIMe
and target is ptp clock, the initial timing correction doesn't seem to take
care of moving the ptp clock.

[Richard Cochran]

Post by Makarand Kulkarni
No, it's the issue I see with phc2sys (which I believe shares the same
PI controller. If I try to correct for error when source is CLOCK_REALTIMe
and target is ptp clock, the initial timing correction doesn't seem to take
care of moving the ptp clock.

So you mean that phc2sys is not able to set the PHC time correctly?

Thanks,
Richard

[Makarand Kulkarni]

Not exactly.

The while(true) loop in phc goes this way (correct me if I'm wrong)
while (1) {
- sleep for (1sec/rate)
- read phc
- sample 'R' times a sec
- update clock
}

Now if your source (master) clock is REALTIME and destination (slave)
clock is ptp, then wou'd sample the
source "R" times. But since REALTIME clock doesn't have resolution as
good as ptp, we end up getting
same values being read between R*n and R*(n+1). So the delta between
the sysclk and ptp clock would go
bad in this interval

On a different note, is there something missing here:

main()
|
+- do_phc_loop(dst, src ..)
|
+- read_phc (src, dst)

Within read_phc(), the destination is referred to as sysclk and clkid. But
from the subsequent code it appears that sysclk is being treated as dst
clock
(dst1 and dst2 and used for reading in sysclk values)

Is this understanding correct ? or I'm missing something

Best Regards,

Makarand

Post by Richard Cochran

Post by Makarand Kulkarni
No, it's the issue I see with phc2sys (which I believe shares the same
PI controller. If I try to correct for error when source is CLOCK_REALTIMe
and target is ptp clock, the initial timing correction doesn't seem to take
care of moving the ptp clock.

So you mean that phc2sys is not able to set the PHC time correctly?
Thanks,
Richard

[Miroslav Lichvar]

Post by Makarand Kulkarni
Now if your source (master) clock is REALTIME and destination (slave)
clock is ptp, then wou'd sample the
source "R" times. But since REALTIME clock doesn't have resolution as
good as ptp, we end up getting
same values being read between R*n and R*(n+1). So the delta between
the sysclk and ptp clock would go
bad in this interval

Can you please post a phc2sys log showing the problem?

Post by Makarand Kulkarni
main()
|
+- do_phc_loop(dst, src ..)
|
+- read_phc (src, dst)
Within read_phc(), the destination is referred to as sysclk and clkid. But
from the subsequent code it appears that sysclk is being treated as dst
clock
(dst1 and dst2 and used for reading in sysclk values)
Is this understanding correct ? or I'm missing something

Yes, the variable naming is confusing. It's written when phc2sys
supported only synchronization of the system clock from PHC.

[Makarand Kulkarni]

I've attached the log here


About the read phc loop, I feel it is incorrect if I go by the logic
used there. The clock ids for sysclk and phc locked are reversed

Best Regards,


Makarand

Post by Miroslav Lichvar

Post by Makarand Kulkarni
Now if your source (master) clock is REALTIME and destination (slave)
clock is ptp, then wou'd sample the
source "R" times. But since REALTIME clock doesn't have resolution as
good as ptp, we end up getting
same values being read between R*n and R*(n+1). So the delta between
the sysclk and ptp clock would go
bad in this interval

Can you please post a phc2sys log showing the problem?

Post by Makarand Kulkarni
main()
|
+- do_phc_loop(dst, src ..)
|
+- read_phc (src, dst)
Within read_phc(), the destination is referred to as sysclk and clkid. But
from the subsequent code it appears that sysclk is being treated as dst
clock
(dst1 and dst2 and used for reading in sysclk values)
Is this understanding correct ? or I'm missing something

Yes, the variable naming is confusing. It's written when phc2sys
supported only synchronization of the system clock from PHC.

[Miroslav Lichvar]

Post by Makarand Kulkarni
I've attached the log here
# phc2sys -c /dev/ptp0 -s CLOCK_REALTIME -i eth0

Hm, are you sure about "-i eth0"? It will override the -s, so you
might be actually synchronizing one PTP clock to another and not from
the system clock.

[Makarand Kulkarni]

Post by Miroslav Lichvar

Post by Makarand Kulkarni
I've attached the log here
# phc2sys -c /dev/ptp0 -s CLOCK_REALTIME -i eth0

Hm, are you sure about "-i eth0"? It will override the -s, so you
might be actually synchronizing one PTP clock to another and not from
the system clock.

Yes, there is only 1 ptp device

[Miroslav Lichvar]

[Makarand Kulkarni]

Yes, that's the huge frequency offset, the slave clock is running
about eight times slower than the master clock. Something is terribly
wrong with one of the clocks, could be a driver or hw bug. The only
thing the servo could do to keep the clocks somewhat close together is
constantly stepping the slave clock (-S option).

But this difference will always be of this magnitude when you sync realtime
clock and ptp clock.

Do you see that there is an issue with what is source and destination clocks
in read_phc ?

-Makarand

[Richard Cochran]

Post by Makarand Kulkarni

Yes, that's the huge frequency offset, the slave clock is running
about eight times slower than the master clock. Something is terribly
wrong with one of the clocks, could be a driver or hw bug. The only
thing the servo could do to keep the clocks somewhat close together is
constantly stepping the slave clock (-S option).

But this difference will always be of this magnitude when you sync realtime
clock and ptp clock.

No, the frequency difference should be at most a few hundred parts per
million. You have an 8x difference? That is really broken.

Thanks,
Richard

[Makarand Kulkarni]

Hmmm.. Did you happen to look at the src and dst in read_phc ? I tend to
believe they are swapped

Best Regards,


Makarand

Post by Richard Cochran

Post by Makarand Kulkarni

Yes, that's the huge frequency offset, the slave clock is running
about eight times slower than the master clock. Something is terribly
wrong with one of the clocks, could be a driver or hw bug. The only
thing the servo could do to keep the clocks somewhat close together is
constantly stepping the slave clock (-S option).

But this difference will always be of this magnitude when you sync realtime
clock and ptp clock.

No, the frequency difference should be at most a few hundred parts per
million. You have an 8x difference? That is really broken.
Thanks,
Richard

[Richard Cochran]

Post by Makarand Kulkarni
Hmmm.. Did you happen to look at the src and dst in read_phc ? I
tend to believe they are swapped

We are trying to help you. Really, we are, but you keep ignoring what
we are telling you. Go back and read the replies again, please.

[ hint: "phc2sys -c /dev/ptp0 -s CLOCK_REALTIME -i eth0" is nonsense]

Richard

[Keller, Jacob E]

-----Original Message-----
Sent: Tuesday, September 24, 2013 10:07 AM
To: Makarand Kulkarni
Subject: Re: [Linuxptp-devel] limitation on frequency offset tracking

Post by Makarand Kulkarni
Hmmm.. Did you happen to look at the src and dst in read_phc ? I
tend to believe they are swapped

We are trying to help you. Really, we are, but you keep ignoring what
we are telling you. Go back and read the replies again, please.
[ hint: "phc2sys -c /dev/ptp0 -s CLOCK_REALTIME -i eth0" is nonsense]
Richard

Makarand,

Could you please update to the latest linuxPTP (v1.3)? I don't believe based on your arguments to the phc2sys program that you are using the latest phc2sys. The "-I" argument was removed in favor of a better system which prevents this confusion.

In the version of phc2sys that I believe you are using, you had to select clock devices separately from Ethernet devices. This means that -I and -c overwrite the same clock value in phc2sys's code.

What you want for your version of phc2sys is something like

"phc2sys -c /dev/ptp0 -s CLOCK_REALTIME"

This is because your version of phc2sys does not really support setting the slave clock by Ethernet device.

What you are doing right now is attempting to drive /dev/ptp0 by /dev/ptp0 if I am reading it correctly.

Please try again without the -I eth0 and/or update to the v1.3 version and try it there. Note that the options changed slightly so you will need to read the manual page and make sure you are using the right options.

Thanks,
Jake

[Makarand Kulkarni]

Richard, Jake: apologies, few things got mixed up.

I've moved to 1.3 now

I believe from the man pages: phc2sys -s CLOCK_REALTIME -c /dev/ptp0 -w
is correct ?
What I want to do is drive ptp0 with REALTIME clock


Best Regards,

Makarand

Post by Keller, Jacob E

-----Original Message-----
Sent: Tuesday, September 24, 2013 10:07 AM
To: Makarand Kulkarni
Subject: Re: [Linuxptp-devel] limitation on frequency offset tracking

Post by Makarand Kulkarni
Hmmm.. Did you happen to look at the src and dst in read_phc ? I
tend to believe they are swapped

We are trying to help you. Really, we are, but you keep ignoring what
we are telling you. Go back and read the replies again, please.
[ hint: "phc2sys -c /dev/ptp0 -s CLOCK_REALTIME -i eth0" is nonsense]
Richard

Makarand,
Could you please update to the latest linuxPTP (v1.3)? I don't believe based on your arguments to the phc2sys program that you are using the latest phc2sys. The "-I" argument was removed in favor of a better system which prevents this confusion.
In the version of phc2sys that I believe you are using, you had to select clock devices separately from Ethernet devices. This means that -I and -c overwrite the same clock value in phc2sys's code.
What you want for your version of phc2sys is something like
"phc2sys -c /dev/ptp0 -s CLOCK_REALTIME"
This is because your version of phc2sys does not really support setting the slave clock by Ethernet device.
What you are doing right now is attempting to drive /dev/ptp0 by /dev/ptp0 if I am reading it correctly.
Please try again without the -I eth0 and/or update to the v1.3 version and try it there. Note that the options changed slightly so you will need to read the manual page and make sure you are using the right options.
Thanks,
Jake

[Richard Cochran]

Post by Makarand Kulkarni
Richard, Jake: apologies, few things got mixed up.
I've moved to 1.3 now
I believe from the man pages: phc2sys -s CLOCK_REALTIME -c /dev/ptp0
-w is correct ?

The -w means:

-w wait for ptp4l

Post by Makarand Kulkarni
What I want to do is drive ptp0 with REALTIME clock

I assume that you want to configure a grand master clock, using the
Linux system time as the time source. In this case, you do not need
the -w flag because your ptp4l program should never change the PHC
time.

HTH,
Richard

[Makarand Kulkarni]

Post by Richard Cochran

Post by Makarand Kulkarni
What I want to do is drive ptp0 with REALTIME clock

I assume that you want to configure a grand master clock, using the
Linux system time as the time source. In this case, you do not need
the -w flag because your ptp4l program should never change the PHC
time.

Yes, this is what I'm upto, I'm running linux on one of the board that
has PTP supported NIC and it talks to the other similar card over ethernet

+-----------------+ +---------------- +
| |_ _| |
| master |_|-------------------|_| slave |
| | eth0 eth0 | |
+-----------------+ +-----------------+

What I'm trying to see is if I can sync up the master ptp with REALTIME
clock and then vary the real time clock to see how accurately the slave
follows it

But it seems that the frequency of realtime clock is far apart from the ptp
frequency and phc2sys never converges.

Can you please suggest some way to do this ? One way I can think of is to
scale down the ptp clock frequency, but is there something else that can be
done ?

Post by Richard Cochran
HTH,
Richard

[Keller, Jacob E]

Post by Makarand Kulkarni

Post by Richard Cochran

Post by Makarand Kulkarni
What I want to do is drive ptp0 with REALTIME clock

I assume that you want to configure a grand master clock, using the
Linux system time as the time source. In this case, you do not need
the -w flag because your ptp4l program should never change the PHC
time.

Yes, this is what I'm upto, I'm running linux on one of the board that
has PTP supported NIC and it talks to the other similar card over ethernet
+-----------------+ +---------------- +
| |_ _| |
| master |_|-------------------|_| slave |
| | eth0 eth0 | |
+-----------------+ +-----------------+
What I'm trying to see is if I can sync up the master ptp with REALTIME
clock and then vary the real time clock to see how accurately the slave
follows it
But it seems that the frequency of realtime clock is far apart from the ptp
frequency and phc2sys never converges.
Can you please suggest some way to do this ? One way I can think of is to
scale down the ptp clock frequency, but is there something else that can be
done ?

Post by Richard Cochran
HTH,
Richard

You could directly modify the PHC. There's an example program in
Documentation/ptp/testptp.c in the upstream kernel source.

I would be interested in knowing how your PHC is so far off the REALTIME
clock though, that seems like a bug. Your original phc2sys example
appeared to be trying to tune a ptp device to itself. Could you please
show the output of phc2sys again using the new version along with the
corrected command line arguments? (it is possible phc2sys has a bug when
trying to tune something to itself..? Even though that's bad behavior
anyways)

For what you are trying, I would suggest directly modifying the PHC
using something like testptp

[Richard Cochran]

Post by Makarand Kulkarni
But it seems that the frequency of realtime clock is far apart from the ptp
frequency and phc2sys never converges.

The Linux system clock and the PHC should be nominally within about
200 PPM of each other. If your custom PHC is off more than that (like
the 8x mentioned before), then you have a HW or driver bug, and no
amount of tinkering with phc2sys will help.

At the risk of stating the obvious, the rate of your PHC is *not* its
input clock rate. You must properly convert the raw clock ticks into
nanoseconds. For example, each tick of a 25 MHz clock is worth 40 ns.

Thanks,
Richard

[Makarand Kulkarni]

Resending since the original mail was blocked due to size limitation

Best Regards,


Makarand

Richard, Jake: few things inline, plus I've attached some logs. There
are some custom prints which I used to check the clock offsets like
this is at driver level counting how many real time ticks
happened till ptp device counted till 1e+9ns
measured length of do_phc_loop
here we see that phc offset is -ve, ptp clock slower than real time, but
the deltas are so large and adjustments get saturated so soon
ptp4l -p /dev/ptp0 -f /home/default.cfg -i eth0 &
phc2sys -c /dev/ptp0 -s CLOCK_REALTIME -O 35
-Makarand

Post by Richard Cochran

Post by Makarand Kulkarni
But it seems that the frequency of realtime clock is far apart from the ptp
frequency and phc2sys never converges.

The Linux system clock and the PHC should be nominally within about
200 PPM of each other.

I think I have around 800000 ppb (= 800 ppm, so not bad). I gather this
from the nsec
which are counted by PHC clock in 1sec realtime clock period

Post by Richard Cochran
If your custom PHC is off more than that (like
the 8x mentioned before), then you have a HW or driver bug, and no
amount of tinkering with phc2sys will help.

Sorry, it's 800ppm, not 8x

Post by Richard Cochran
At the risk of stating the obvious, the rate of your PHC is *not* its
input clock rate. You must properly convert the raw clock ticks into
nanoseconds. For example, each tick of a 25 MHz clock is worth 40 ns.

Yes, it's the same understanding here too

Post by Richard Cochran
Thanks,
Richard

[Richard Cochran]

this is at driver level counting how many real time ticks
happened till ptp device counted till 1e+9ns

Don't know what you mean by that, so I will just ignore those lines.

measured length of do_phc_loop
here we see that phc offset is -ve, ptp clock slower than real time, but
the deltas are so large and adjustments get saturated so soon

What is "-ve" ?

[ Omitting the other lines: ]

Jan 1 00:05:05 at501-1 user.info phc2sys: [288.266] phc offset -13299850229 s0 freq +0 delay 2666
Jan 1 00:05:06 at501-1 user.info phc2sys: [289.292] phc offset -13223016021 s1 freq [phone] delay 2667
Jan 1 00:05:07 at501-1 user.info phc2sys: [290.314] phc offset -26336727379 s2 freq -32767999 delay 2909
Jan 1 00:05:08 at501-1 user.info phc2sys: [291.335] phc offset -26290043771 s2 freq -32767999 delay 2666
Jan 1 00:05:09 at501-1 user.info phc2sys: [292.356] phc offset -26243361590 s2 freq -32767999 delay 2666
Jan 1 00:05:10 at501-1 user.info phc2sys: [293.377] phc offset -26196679579 s2 freq -32767999 delay 2666
Jan 1 00:05:11 at501-1 user.info phc2sys: [294.398] phc offset -26149997088 s2 freq -32767999 delay 2667
Jan 1 00:05:12 at501-1 user.info phc2sys: [295.409] phc offset -26103773082 s2 freq -32767999 delay 2788
Jan 1 00:05:13 at501-1 user.info phc2sys: [296.420] phc offset -26057548659 s2 freq -32767999 delay 2667

^^^^^^^ ^^^^^^^^^^^

Just look at this.

[Richard Cochran]

Either your system time is broken, or your PHC time is broken, or
both. The most likely cause is your new, custom driver.

If you don't believe me, just try running phc2sys with the calls to
clock_adjtime() disabled (commented out). By plotting the rate of
change of the offset, you can figure the frequency offset. If this is
more than about 200 PPM (that is, 0.02%) then you probably have a bug
in your driver, or maybe your board's clock frequencies are not what
you think they are...

HTH,
Richard

[Keller, Jacob E]

-----Original Message-----
Sent: Monday, September 30, 2013 12:06 PM
To: Makarand Kulkarni
Subject: Re: [Linuxptp-devel] limitation on frequency offset tracking

Either your system time is broken, or your PHC time is broken, or
both. The most likely cause is your new, custom driver.

If you don't believe me, just try running phc2sys with the calls to
clock_adjtime() disabled (commented out). By plotting the rate of
change of the offset, you can figure the frequency offset. If this is
more than about 200 PPM (that is, 0.02%) then you probably have a bug
in your driver, or maybe your board's clock frequencies are not what
you think they are...
HTH,
Richard

In addition, ensure that your adjust frequency ptp call is not continuously applying the ppb at each call to the current rate, but to the base rate. The adjfreq call is a ppb adjustment from the nominal clock frequency, not from the current clock frequency. I believe that could be the result of what you see here.

Regards,
Jake

[Makarand Kulkarni]

Post by Richard Cochran
If you don't believe me, just try running phc2sys with the calls to
clock_adjtime() disabled (commented out). By plotting the rate of
change of the offset, you can figure the frequency offset. If this is
more than about 200 PPM (that is, 0.02%) then you probably have a bug
in your driver, or maybe your board's clock frequencies are not what
you think they are...

I believe you :) the board's clock frequency is not what it says

Post by Richard Cochran
In addition, ensure that your adjust frequency ptp call is not continuously applying the ppb at each call to the current rate, but to the base rate. The adjfreq call is a ppb adjustment from the nominal clock frequency, not from the current clock frequency. I believe that could be the result of what you see here.

yes, I got it clarified with you guys earlier on a different mail thread

[Makarand Kulkarni]

I corrected for the frequency offset and tried to run phc2sys to sync my
ptp clock with real time clock. There's something unusual I noticed: the
corrections being passed to the driver were pushing it in wrong
direction, for instance, my ptp clock was ahead of the real time clock,
so going by the do_phc_loop in phc2sys:

offset = (dst1+interval/2) - src (I simplified by not splitting nsec and
sec)

since src is ptp here, the offset should be -ve and hence the ppb

At this point I see that the sign of ppb sent to driver in update_clock
is reversed, so it will push the driver to speed up the ptp clock
further. This causes the ppb and hence the max adjustments to saturate
after some cycles.

Is there something I'm missing in how phc2sys is working ? I don't doubt
the driver and the servo code since it's shared with ptp4l, but I'm not
sure if I get the above correctly

Best Regards,


Makarand

Post by Makarand Kulkarni

Post by Richard Cochran
If you don't believe me, just try running phc2sys with the calls to
clock_adjtime() disabled (commented out). By plotting the rate of
change of the offset, you can figure the frequency offset. If this is
more than about 200 PPM (that is, 0.02%) then you probably have a bug
in your driver, or maybe your board's clock frequencies are not what
you think they are...

I believe you :) the board's clock frequency is not what it says

Post by Richard Cochran
In addition, ensure that your adjust frequency ptp call is not continuously applying the ppb at each call to the current rate, but to the base rate. The adjfreq call is a ppb adjustment from the nominal clock frequency, not from the current clock frequency. I believe that could be the result of what you see here.

yes, I got it clarified with you guys earlier on a different mail thread
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134791&iu=/4140/ostg.clktrk
_______________________________________________
Linuxptp-devel mailing list
https://lists.sourceforge.net/lists/listinfo/linuxptp-devel

[Richard Cochran]

Post by Makarand Kulkarni
I corrected for the frequency offset and tried to run phc2sys to sync my
ptp clock with real time clock. There's something unusual I noticed: the
corrections being passed to the driver were pushing it in wrong
direction, for instance, my ptp clock was ahead of the real time clock,
offset = (dst1+interval/2) - src (I simplified by not splitting nsec and
sec)
since src is ptp here, the offset should be -ve and hence the ppb

No, src is CLOCK_REALTIME. At least, when you run the command,

phc2sys -c /dev/ptp0 -s CLOCK_REALTIME -O 35

then you will have it that way.

Post by Makarand Kulkarni
At this point I see that the sign of ppb sent to driver in update_clock
is reversed, so it will push the driver to speed up the ptp clock
further. This causes the ppb and hence the max adjustments to saturate
after some cycles.

In phc2sys.c:118 you have the following computation:

*offset = (tdst1.tv_sec - tsrc.tv_sec) * NS_PER_SEC
^^^^^^^^^ ^^^^^^^^^^^
/dev/ptp0 CLOCK_REALT.

If the offset is greater than zero, then ptp0 is ahead, and we slow it
down by reducing its frequency.

HTH,
Richard