Discussion:
[Linuxptp-devel] Stability tests linuxptp 1.5
Rich Schmidt
2015-02-26 19:59:01 UTC
Permalink
I have been running phase stability tests of linuxptp-1.5 using linreg
servo on an Intel i350-T2 (HP ProLiant DL120G6 RHEL 7.0 kernel 3.10)
synchronizing to an FEI-Zyfer Gsync Time and Frequency Processor. Below are
sigma-tau plots of the ptp4l phase offset between the HP server and the
Gsync PTP master, and phase offsets of CLOCK_REALTIME which is steered by
phc2sys to /dev/ptp0).

The igb driver is 5.2.5.

The sigma-tau plot for ptp4l (TDEV curve in violet) shows that we reach the
white noise flicker floor of 20 microseconds phase stability at about 8
hours. The environment is not temperature-controlled and there are
resulting frequency instabilities. But overall this looks great.

The ptp4l phase offset statistics:
min=-109 ns
max=104 ns
mean =-2.69 ns
standard deviation=35.0 ns
rate=1.32e-15 seconds/sec
drift=1.69e-16 per day
sample size 261,634 points
tau=2 sec.

The phc2sys phase offset statistics:
min=-365 ns
max=347 ns
mean=-8.31 ns
standard dev.=100.2 ns
rate=6.82e-06 drift=-1.80e-06
sample size 522,802 points
tau=1 sec.



​


My ptp4l.conf was:
network_transport UDPv4
logAnnounceInterval 1
logSyncInterval 1
logMinDelayReqInterval 2
logMinPdelayReqInterval 2
announceReceiptTimeout 4
clock_servo linreg
time_stamping hardware
slaveOnly 1
priority1 200
priority2 200
free_running 0

My compliments to the hard-working linuxptp team for this outstanding
performance!
Rich Schmidt
USNO
Gary E. Miller
2015-02-26 20:23:12 UTC
Permalink
Yo Rich!

On Thu, 26 Feb 2015 14:59:01 -0500
Post by Rich Schmidt
I have been running phase stability tests of linuxptp-1.5 using
linreg servo on an Intel i350-T2 (HP ProLiant DL120G6 RHEL 7.0 kernel
3.10) synchronizing to an FEI-Zyfer Gsync Time and Frequency
Processor.
Can you provide the full configuration and setup? Plus your measurement
procedure? My newbie results are many orders of magnitude worse.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97701
***@rellim.com Tel:+1(541)382-8588
Rich Schmidt
2015-02-26 21:29:00 UTC
Permalink
Gary,
The config file for ptp4l was just as provided in my last note. There
wasn't much else to setup. Note that I had no success with any of the Intel
igb drivers under RHEL 6.5. Everything works nicely under RHEL 7.0 kernel
3.10. (Or CenOS 7, same thing for free). I turned off tickless kernel
addng "nohz=off" to /boot/grub2/grub.cfg:
linux16 /vmlinuz-3.10.0-123.20.1.el7.x86_64
root=/dev/mapper/vg_ptp-lv_root ro audit=1 nomodeset crashkernel=auto
vconsole.keymap=us rd.lvm.lv=vg_ptp/lv_swap crashkernel=auto
vconsole.font=latarcyrheb-sun16 rd.lvm.lv=vg_ptp/lv_root rhgb quiet
LANG=en_US.UTF-8 nohz=off

I disabled chrony, as I am serving NTP. I have an FEI-Zyfer Gsync II Time
and Frequency Processor acting as PTP Grandmaster. I configured the IP
address of an i350 NIC to the same subnet as my Grandmaster. Using
"ethtool -T" I find that my PHC is number 0, which is /dev/ptp0.

So my ptp4l command is:
nohup nice --20 /usr/local/sbin/ptp4l -m -f /etc/ptp4l.conf -i eth2 -p
/dev/ptp0 > /var/log/ptp4l.log &
and to steer my system to the PHC I run:
nohup nice --20 /usr/local/sbin/phc2sys -q -m -c CLOCK_REALTIME -s
/dev/ptp0 -E linreg -O 0 -l 7 >/var/log/ptp/phc2sys.log &

Note that my Grandmaster is on UTC , not PTP timescale, so I specify -O 0
for the UTC offset for phc2sys.
Then I run as an NTP stratum 1 server using the local clock refclock in
/etc/ntp.conf:
server 127.127.1.0 prefer
server 192.5.41.40 noselect
fudge 127.127.1.0 stratum 0
fudge 127.127.1.0 refid PTP

Not everyone has an FEI-Zyfer Gsync for a grandmaster. At home I have two
Supermicro Atom servers, one is grandmaster, one is slave. The grandmaster
has a GPS receiver outputting 1PPS which is connected to the DCD pin on the
internal COM2 serial header. Here the NTP refclock_nmea driver
synchronizes the system clock of the grandmaster. Then 'phc2sys -s
CLOCK_REALTIME -c /dev/ptp0" steers the PHC to system time, while ptp4l
provides PTP on an Intel i210 NIC. On the slave Supermicro server a second
Intel i210 NIC synchronizes to the grandmaster as above. For complete
details google "Developing Low-Cost NTP Stratum 1 Servers with Linux PTP
and GPS".

Hope this helps,
Rich Schmidt
Post by Gary E. Miller
Yo Rich!
On Thu, 26 Feb 2015 14:59:01 -0500
Post by Rich Schmidt
I have been running phase stability tests of linuxptp-1.5 using
linreg servo on an Intel i350-T2 (HP ProLiant DL120G6 RHEL 7.0 kernel
3.10) synchronizing to an FEI-Zyfer Gsync Time and Frequency
Processor.
Can you provide the full configuration and setup? Plus your measurement
procedure? My newbie results are many orders of magnitude worse.
RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97701
Gary E. Miller
2015-02-26 21:43:10 UTC
Permalink
Yo Rich!

On Thu, 26 Feb 2015 16:29:00 -0500
Post by Rich Schmidt
The config file for ptp4l was just as provided in my last note. There
wasn't much else to setup.
Uh, sorry, already deleted...
Post by Rich Schmidt
I disabled chrony, as I am serving NTP.
Uh, say what? Chronyd is used to serve NTP, I hope you are not using ntpd
instead? I find chrony much superior in timeserving to chronyd.
Post by Rich Schmidt
I have an FEI-Zyfer Gsync II
Time and Frequency Processor acting as PTP Grandmaster. I configured
the IP address of an i350 NIC to the same subnet as my Grandmaster.
Using "ethtool -T" I find that my PHC is number 0, which is /dev/ptp0.
Could I please get the ethtool -T dump? I'm trying to figure out
what features do what...
Post by Rich Schmidt
nohup nice --20 /usr/local/sbin/ptp4l -m -f /etc/ptp4l.conf -i eth2 -p
/dev/ptp0 > /var/log/ptp4l.log &
nohup nice --20 /usr/local/sbin/phc2sys -q -m -c CLOCK_REALTIME -s
/dev/ptp0 -E linreg -O 0 -l 7 >/var/log/ptp/phc2sys.log &
Note that my Grandmaster is on UTC , not PTP timescale, so I specify
-O 0 for the UTC offset for phc2sys.
Then I run as an NTP stratum 1 server using the local clock refclock
server 127.127.1.0 prefer
server 192.5.41.40 noselect
fudge 127.127.1.0 stratum 0
fudge 127.127.1.0 refid PTP
Hmm, no secondary sources? How do you know you do not have the large
offset I see?
Post by Rich Schmidt
Not everyone has an FEI-Zyfer Gsync for a grandmaster. At home I have
two Supermicro Atom servers, one is grandmaster, one is slave. The
grandmaster has a GPS receiver outputting 1PPS which is connected to
the DCD pin on the internal COM2 serial header. Here the NTP
refclock_nmea driver synchronizes the system clock of the
grandmaster. Then 'phc2sys -s CLOCK_REALTIME -c /dev/ptp0" steers the
PHC to system time, while ptp4l provides PTP on an Intel i210 NIC.
And who is sending SHM 0? And how?
Post by Rich Schmidt
On the slave Supermicro server a second Intel i210 NIC synchronizes
to the grandmaster as above. For complete details google "Developing
Low-Cost NTP Stratum 1 Servers with Linux PTP and GPS".
This one?

http://www.academia.edu/10312557/DEVELOPING_LOW-COST_NTP_STRATUM_1_SERVERS_WITH_LINUX_PTP_AND_GPS

Sadly, after jumping through several hoops, it would not let me get
to the paper (eventual 404). Got a better URL?

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97701
***@rellim.com Tel:+1(541)382-8588
Rich Schmidt
2015-02-26 22:01:21 UTC
Permalink
Gary, this is for USNO Time Service. We are a high-end provider of NTP
service with over 20 years of satisfying customers! All of my work
involves improvements to tick and tock and their classified cousins in DoD
land.


​>​
Could I please get the ethtool -T dump? I'm trying to figure out
Post by Gary E. Miller
what features do what...
​Sent via email to Gary.​
Post by Gary E. Miller
Post by Rich Schmidt
server 127.127.1.0 prefer
server 192.5.41.40 noselect
fudge 127.127.1.0 stratum 0
fudge 127.127.1.0 refid PTP
​>​
Hmm, no secondary sources? How do you know you do not have the large
offset I see?
I don't know what large offset you are seeing.
But comparing at the internal NTP level we see no offset from the
comparison server tick.usno.navy.mil.
​

​​

​>​
Post by Gary E. Miller
And who is sending SHM 0? And how?
​Shared memory is not involved in this setup. Phc2sys takes care of
steering the system clock, and NTP accepts it using server 127.127.1.0.
​I compared the logged phase offsets from ptp4l and phc2sys using log level
6 or 7. In order to do a physical phase measurement I need to program the
software definable pins on the i210 header to output 1PPS, which can then
be compared with a time interval counter to the USNO Master Clock 1PPS
reference. I have yet to modify the igb driver to do that, stay tuned.
Rich Schmidt​
Post by Gary E. Miller
​
Richard Cochran
2015-02-27 10:44:31 UTC
Permalink
​I compared the logged phase offsets from ptp4l and phc2sys using log level
6 or 7. In order to do a physical phase measurement I need to program the
software definable pins on the i210 header to output 1PPS, which can then
be compared with a time interval counter to the USNO Master Clock 1PPS
reference. I have yet to modify the igb driver to do that, stay tuned.
FYI my patches to enable the i210 pins hit mainline as of v4.0-rc1.

720db4f igb: enable auxiliary PHC functions for the i210
00c6557 igb: enable internal PPS for the i210
8298c1e igb: serialize access to the time sync interrupt registers
61d7f75 igb: refactor time sync interrupt handling

Thanks,
Richard
Gary E. Miller
2015-03-02 23:27:33 UTC
Permalink
Yo Richard!

On Fri, 27 Feb 2015 11:44:31 +0100
Post by Richard Cochran
FYI my patches to enable the i210 pins hit mainline as of v4.0-rc1.
Any idea when they will hit the kernel mainline?

Probably before I can afford a phase analyzer. :-)

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97701
***@rellim.com Tel:+1(541)382-8588
Richard Cochran
2015-03-03 07:47:00 UTC
Permalink
Post by Gary E. Miller
Any idea when they will hit the kernel mainline?
Those patches are already in v4.0-rc1, and so they will definitely
appear in the upcoming v4.0.

Cheers,
Richard
Gary E. Miller
2015-03-03 08:40:36 UTC
Permalink
Yo Richard!

On Tue, 3 Mar 2015 08:47:00 +0100
Post by Richard Cochran
Post by Gary E. Miller
Any idea when they will hit the kernel mainline?
Those patches are already in v4.0-rc1, and so they will definitely
appear in the upcoming v4.0.
Cool, thanks.

I shoulda figured that out, v4.0 was not MY vote. :-)

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97701
***@rellim.com Tel:+1(541)382-8588

Gary E. Miller
2015-03-02 23:25:53 UTC
Permalink
Yo Rich!

On Thu, 26 Feb 2015 17:01:21 -0500
Post by Rich Schmidt
Gary, this is for USNO Time Service. We are a high-end provider of NTP
service with over 20 years of satisfying customers! All of my work
involves improvements to tick and tock and their classified cousins
in DoD land.
And I've used those servers with good results for almost that long. Kudos
on the USNO performance.

So, what am I doing wrong here?
Post by Rich Schmidt
Post by Gary E. Miller
Hmm, no secondary sources? How do you know you do not have the
large offset I see?
I don't know what large offset you are seeing.
But comparing at the internal NTP level we see no offset from the
comparison server tick.usno.navy.mil.
Well, at this moment, I am seeing low jitter, but -415 mSec offset to a
PPS receiver on the same host. A long trusted PPS receiver.
Post by Rich Schmidt
​I compared the logged phase offsets from ptp4l and phc2sys using log
level 6 or 7. In order to do a physical phase measurement I need to
program the software definable pins on the i210 header to output
1PPS, which can then be compared with a time interval counter to the
USNO Master Clock 1PPS reference. I have yet to modify the igb
driver to do that, stay tuned.
Very cool.

So, thinking about the differences of our setups...

I suspect, since you are in hard-core metrology, that these machines
are in climate controled rooms and never turned off? Maybe there is
a slow convergence and you are past that?

Have you also been logging your ntpd outputs?

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97701
***@rellim.com Tel:+1(541)382-8588
Rich Schmidt
2015-02-26 22:14:56 UTC
Permalink
I'm using the i210 with phc2sys on my Supermicro and the jitter as reported
by NTP is 954 nanoseconds, i.e., just sub-microsecond. I have gotten
similar results with the Supermicro's system board 82547L NICs (my paper at
the ION PTTI Boston, Dec. 2014, Developing Low Cost NTP Stratum 1 Servers
with Linux PTP an GPS), and have tested an 82547L on the HP ProLiant
DL120G6 with the FEI-Zyfer Gsync at USNO, and again similar performance. I
can say with confidence that all three, the i350, i210, and 82547L work
well with linuxptp. And with NTP v. 4.2.8p1.
Rich
Post by Gary E. Miller
Yo Rich!
On Thu, 26 Feb 2015 16:29:00 -0500
Post by Rich Schmidt
The config file for ptp4l was just as provided in my last note. There
wasn't much else to setup.
Uh, sorry, already deleted...
Post by Rich Schmidt
I disabled chrony, as I am serving NTP.
Uh, say what? Chronyd is used to serve NTP, I hope you are not using ntpd
instead? I find chrony much superior in timeserving to chronyd.
Post by Rich Schmidt
I have an FEI-Zyfer Gsync II
Time and Frequency Processor acting as PTP Grandmaster. I configured
the IP address of an i350 NIC to the same subnet as my Grandmaster.
Using "ethtool -T" I find that my PHC is number 0, which is /dev/ptp0.
Could I please get the ethtool -T dump? I'm trying to figure out
what features do what...
Post by Rich Schmidt
nohup nice --20 /usr/local/sbin/ptp4l -m -f /etc/ptp4l.conf -i eth2 -p
/dev/ptp0 > /var/log/ptp4l.log &
nohup nice --20 /usr/local/sbin/phc2sys -q -m -c CLOCK_REALTIME -s
/dev/ptp0 -E linreg -O 0 -l 7 >/var/log/ptp/phc2sys.log &
Note that my Grandmaster is on UTC , not PTP timescale, so I specify
-O 0 for the UTC offset for phc2sys.
Then I run as an NTP stratum 1 server using the local clock refclock
server 127.127.1.0 prefer
server 192.5.41.40 noselect
fudge 127.127.1.0 stratum 0
fudge 127.127.1.0 refid PTP
Hmm, no secondary sources? How do you know you do not have the large
offset I see?
Post by Rich Schmidt
Not everyone has an FEI-Zyfer Gsync for a grandmaster. At home I have
two Supermicro Atom servers, one is grandmaster, one is slave. The
grandmaster has a GPS receiver outputting 1PPS which is connected to
the DCD pin on the internal COM2 serial header. Here the NTP
refclock_nmea driver synchronizes the system clock of the
grandmaster. Then 'phc2sys -s CLOCK_REALTIME -c /dev/ptp0" steers the
PHC to system time, while ptp4l provides PTP on an Intel i210 NIC.
And who is sending SHM 0? And how?
Post by Rich Schmidt
On the slave Supermicro server a second Intel i210 NIC synchronizes
to the grandmaster as above. For complete details google "Developing
Low-Cost NTP Stratum 1 Servers with Linux PTP and GPS".
This one?
http://www.academia.edu/10312557/DEVELOPING_LOW-COST_NTP_STRATUM_1_SERVERS_WITH_LINUX_PTP_AND_GPS
Sadly, after jumping through several hoops, it would not let me get
to the paper (eventual 404). Got a better URL?
RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97701
Gary E. Miller
2015-03-02 23:19:10 UTC
Permalink
Yo Rich!

On Thu, 26 Feb 2015 17:14:56 -0500
Post by Rich Schmidt
I'm using the i210 with phc2sys on my Supermicro and the jitter as
reported by NTP is 954 nanoseconds, i.e., just sub-microsecond. I
have gotten similar results with the Supermicro's system board 82547L
NICs (my paper at the ION PTTI Boston, Dec. 2014, Developing Low Cost
NTP Stratum 1 Servers with Linux PTP an GPS), and have tested an
82547L on the HP ProLiant DL120G6 with the FEI-Zyfer Gsync at USNO,
and again similar performance. I can say with confidence that all
three, the i350, i210, and 82547L work well with linuxptp. And with
NTP v. 4.2.8p1. Rich
My jitter is also similarly good. It is my offset that is really
bad. Approximately +800 mSec to -400 mSec seen of the course of the last
4 days.

So, what is your offset?

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97701
***@rellim.com Tel:+1(541)382-8588
Keller, Jacob E
2015-02-26 22:58:03 UTC
Permalink
Hi,
Post by Gary E. Miller
Yo Rich!
On Thu, 26 Feb 2015 16:29:00 -0500
Post by Rich Schmidt
The config file for ptp4l was just as provided in my last note. There
wasn't much else to setup.
Uh, sorry, already deleted...
Post by Rich Schmidt
I disabled chrony, as I am serving NTP.
Uh, say what? Chronyd is used to serve NTP, I hope you are not using ntpd
instead? I find chrony much superior in timeserving to chronyd.
Post by Rich Schmidt
I have an FEI-Zyfer Gsync II
Time and Frequency Processor acting as PTP Grandmaster. I configured
the IP address of an i350 NIC to the same subnet as my Grandmaster.
Using "ethtool -T" I find that my PHC is number 0, which is /dev/ptp0.
Could I please get the ethtool -T dump? I'm trying to figure out
what features do what...
Post by Rich Schmidt
nohup nice --20 /usr/local/sbin/ptp4l -m -f /etc/ptp4l.conf -i eth2 -p
/dev/ptp0 > /var/log/ptp4l.log &
nohup nice --20 /usr/local/sbin/phc2sys -q -m -c CLOCK_REALTIME -s
/dev/ptp0 -E linreg -O 0 -l 7 >/var/log/ptp/phc2sys.log &
Note that my Grandmaster is on UTC , not PTP timescale, so I specify
-O 0 for the UTC offset for phc2sys.
Then I run as an NTP stratum 1 server using the local clock refclock
server 127.127.1.0 prefer
server 192.5.41.40 noselect
fudge 127.127.1.0 stratum 0
fudge 127.127.1.0 refid PTP
Hmm, no secondary sources? How do you know you do not have the large
offset I see?
I am wondering if maybe it is because your provided PTP source wasn't in
PTP timescale, so you might try the -O 0 option?

Regards,
Jake
Richard Cochran
2015-02-27 10:12:54 UTC
Permalink
Post by Keller, Jacob E
Post by Gary E. Miller
Hmm, no secondary sources? How do you know you do not have the large
offset I see?
I am wondering if maybe it is because your provided PTP source wasn't in
PTP timescale, so you might try the -O 0 option?
I think Gary was seeing an 800 ms offset (not 35 seconds).

Thanks,
Richard
Gary E. Miller
2015-03-02 23:17:21 UTC
Permalink
Yo Jacob!

On Thu, 26 Feb 2015 22:58:03 +0000
Post by Keller, Jacob E
Post by Gary E. Miller
Hmm, no secondary sources? How do you know you do not have the
large offset I see?
I am wondering if maybe it is because your provided PTP source wasn't
in PTP timescale, so you might try the -O 0 option?
-O is in whole seconds, my issue is 100s of mSec.

Chrony refuses to sync to my timestamp hardware port, so it just
freesuns. Short term, very little jitter, long term wander of just under
one Second. I have seen from about +800 mSec to about -400 mSec.

And I know my source is good, other timestamp software clients getting
6 uSec jitter and sub mSec offset.

RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97701
***@rellim.com Tel:+1(541)382-8588
Loading...