[Linuxptp-devel] need help (newbie)

[Makarand Kulkarni]

I'm trying to port linuxptp to work with cadence GMAC. Test setup
consists of 2 boards connected through switch, one as master and other
as slave. The exchanges and timestamps go fine, but I do not see the
convergence in terms of offsets and path delays. If I plot the master
offset vs time, what I get is a sine wave with amplitude ranging in few
seconds

Is it normal to get this variation ? I'm using v1.2 of linuxptp

[Richard Cochran]

Post by Makarand Kulkarni
I'm trying to port linuxptp to work with cadence GMAC. Test setup

What is "cadence GMAC"?

What do you mean by "port"?

Post by Makarand Kulkarni
consists of 2 boards connected through switch, one as master and other
as slave. The exchanges and timestamps go fine, but I do not see the
convergence in terms of offsets and path delays. If I plot the master
offset vs time, what I get is a sine wave with amplitude ranging in few
seconds
Is it normal to get this variation ? I'm using v1.2 of linuxptp

No, that is not normal. I guess that the time stamps are faulty.

Also, why not use v1.3?

Thanks,
Richard

[Makarand Kulkarni]

Hi Richard,

Cadence GMAC is the MAC controller with 1588 logic emdedded. The
timestamps are generated by the HW and they (atleast) appear to be
consistent. Infact the HW detects the PTP packets and generates time stamps

I can switch to v1.3 and give it a try. Meanwhile if there is something
that rings bell looking at this plot of master offset, it would be great



Thanks,
Makarand

Post by Richard Cochran

Post by Makarand Kulkarni
I'm trying to port linuxptp to work with cadence GMAC. Test setup

What is "cadence GMAC"?
What do you mean by "port"?

Post by Makarand Kulkarni
consists of 2 boards connected through switch, one as master and other
as slave. The exchanges and timestamps go fine, but I do not see the
convergence in terms of offsets and path delays. If I plot the master
offset vs time, what I get is a sine wave with amplitude ranging in few
seconds
Is it normal to get this variation ? I'm using v1.2 of linuxptp

No, that is not normal. I guess that the time stamps are faulty.
Also, why not use v1.3?
Thanks,
Richard

[Richard Cochran]

Post by Makarand Kulkarni
Cadence GMAC is the MAC controller with 1588 logic emdedded. The
timestamps are generated by the HW and they (atleast) appear to be
consistent. Infact the HW detects the PTP packets and generates time stamps

But what driver are you using? I never heard of that one.

Post by Makarand Kulkarni
I can switch to v1.3 and give it a try. Meanwhile if there is
something that rings bell looking at this plot of master offset, it
would be great

Probably the servo is alternating between +max_adj and -max_adj.
This is a sign of bad time stamps or really broken clocks.

Thanks,
Richard

[Makarand Kulkarni]

Post by Richard Cochran

Post by Makarand Kulkarni
Cadence GMAC is the MAC controller with 1588 logic emdedded. The
timestamps are generated by the HW and they (atleast) appear to be
consistent. Infact the HW detects the PTP packets and generates time stamps

But what driver are you using? I never heard of that one.

It's the macb/gem device (drivers/net/ethernet). I added parts over
it to support timestamping as described in
ptp.txt in documentation section

Would this be the right mailer to put some diffs specific to a
particular device ? I joined couple of days back,
so not sure

Post by Richard Cochran

Post by Makarand Kulkarni
I can switch to v1.3 and give it a try. Meanwhile if there is
something that rings bell looking at this plot of master offset, it
would be great

Probably the servo is alternating between +max_adj and -max_adj.
This is a sign of bad time stamps or really broken clocks.

When you say bad time stamps, what exactly would you characterize
as ? I looked for couple of things:
1) timestamps are continuously increasing
2) delta between consecutive ones are fairly constant
3) delta between time reported by sender and time received is
fairly stable

Post by Richard Cochran
Thanks,
Richard

[Richard Cochran]

Post by Makarand Kulkarni
It's the macb/gem device (drivers/net/ethernet). I added parts
over it to support timestamping as described in
ptp.txt in documentation section

So you added support for software transmit time stamps?

(This means adding a single line of code.)

Post by Makarand Kulkarni
Would this be the right mailer to put some diffs specific to a
particular device ? I joined couple of days back,
so not sure

For Linux network drivers, the netdev list is the right place, but we
can also discuss this here. I really suspect that you have a driver
issue related to your custom modifications.

Post by Makarand Kulkarni
When you say bad time stamps, what exactly would you
1) timestamps are continuously increasing
2) delta between consecutive ones are fairly constant
3) delta between time reported by sender and time received is
fairly stable

The first question is, are you using software time stamping?

Thanks,
Richard

[Makarand Kulkarni]

I'm using HW timestamps. The device works this way: it has
a) free running clock which can be programmed to give you desired
frequency by taking adjustments in freq and time
b) logic that determines if any of the PTP packet is received or transmitted
c) latch the timestamp and generate interrupt when event b) occurs

Here's some piece of code that handles the timestamping: It's little
more than just timestamping the packet since it also handles invoking
some APIs to manipulate sk_buff

if (status & MACB_PTP_TX_INT_FLAGS) {
if (!bp->txstamp_skb) {
netdev_err(bp->dev,"TX timestamp for NULL packet ?");
} else {
memset(&shhwtstamps, 0,sizeof(shhwtstamps));
shhwtstamps.hwtstamp = ns_to_ktime(nsec);
netdev_vdbg(bp->dev,"macb_ptp_tstamp_complete txstamp_skb =
0x%x ts = %llu\n",
(int)bp->txstamp_skb, shhwtstamps.hwtstamp.tv64);

/* Writing 0 to IER bit has no effect, if you want to
* really disable, write to IDR with bit 1
*/
skb_complete_tx_timestamp(bp->txstamp_skb, &shhwtstamps);
}
macb_writel(bp, IER, MACB_PTP_TX_INT_FLAGS);
} else if (status & MACB_PTP_RX_INT_FLAGS) {
if (bp->rxstamp_skb == NULL) {
netdev_err(bp->dev,"NULL descriptor stored for rx handler\n");
macb_writel(bp, IER, MACB_PTP_RX_INT_FLAGS);
return;
}
pshhwtstamps = skb_hwtstamps(bp->rxstamp_skb);
/* Note that pshhwtstamps doesn't point to the same variable now */
memset(pshhwtstamps, 0, sizeof(*pshhwtstamps));
pshhwtstamps->hwtstamp = ns_to_ktime(nsec);
netdev_vdbg(bp->dev,"macb_ptp_tstamp_complete rxstamp_skb = 0x%x
ts = %llu\n",
(int)bp->rxstamp_skb, pshhwtstamps->hwtstamp.tv64);

if(!nsec) netdev_err(bp->dev,"RX buffer has 0 ts!!");
netif_rx(bp->rxstamp_skb); <===================================
this is where I give the packet to net stack
macb_writel(bp, IER, MACB_PTP_RX_INT_FLAGS);
} else {
netdev_err(bp->dev,"ERR ! received interrupt for no reason\n");
}

Other than these, there are couple of functions to store the tx and rx
buffers which get diverted on the path different from normal UDP

Please let me know if you ant other information
-Makarand

Post by Richard Cochran

Post by Makarand Kulkarni
It's the macb/gem device (drivers/net/ethernet). I added parts
over it to support timestamping as described in
ptp.txt in documentation section

So you added support for software transmit time stamps?
(This means adding a single line of code.)

Post by Makarand Kulkarni
Would this be the right mailer to put some diffs specific to a
particular device ? I joined couple of days back,
so not sure

For Linux network drivers, the netdev list is the right place, but we
can also discuss this here. I really suspect that you have a driver
issue related to your custom modifications.

Post by Makarand Kulkarni
When you say bad time stamps, what exactly would you
1) timestamps are continuously increasing
2) delta between consecutive ones are fairly constant
3) delta between time reported by sender and time received is
fairly stable

The first question is, are you using software time stamping?
Thanks,
Richard

[Richard Cochran]

Post by Makarand Kulkarni
I'm using HW timestamps. The device works this way: it has

Okay, then. Just out of curiousity, is the HW time stamping available
on all Cadence MACB/GEM devices?

Without looking at your driver, I can only give a few hints...

Post by Makarand Kulkarni
a) free running clock which can be programmed to give you desired
frequency by taking adjustments in freq and time

Have you verified that the programmed frequency is correct?

Post by Makarand Kulkarni
b) logic that determines if any of the PTP packet is received or transmitted
c) latch the timestamp and generate interrupt when event b) occurs

[ So there is a potential that the time stamps get dropped or
associated with the wrong packets. ]

In order to make sure the driver and hardware are working properly,
you can perform a few simple tests.

* clock sanity check

testptp -s
testptp -g; date
#
# are the times the same?
#
testptp -g; sleep 1; testptp -g
#
# has one second transpired?
#
testptp -g; sleep 10; testptp -g
#
# has ten seconds transpired?
#
testptp -f 10000
#
# That shifts the frequency by 10 PPM.
# Now repeat the above tests.
#
testptp -f -10000
#
# Ditto for -10 PPM.
#

* time stamp check

Use the test program

linux/Documentation/networking/timestamping.c

to make sure that the Rx and Tx time stamps come at the expected
intervals.

Good luck,
Richard

[Keller, Jacob E]

It could also be an issue with your PTP functions (adjust frequency and
adjust time) being incorrect. I know several Intel drivers had overflow
issues when doing frequency adjustment which could cause the cycle
between min and max adjustment.

The easiest way to determine this is to do controlled testing using the
test_phc program Richard described.

Without seeing the driver source there is little we can do, and even
then not much because we don't know the hardware.

Thanks,
Jake

Post by Richard Cochran

Post by Makarand Kulkarni
I'm using HW timestamps. The device works this way: it has

Okay, then. Just out of curiousity, is the HW time stamping available
on all Cadence MACB/GEM devices?
Without looking at your driver, I can only give a few hints...

Post by Makarand Kulkarni
a) free running clock which can be programmed to give you desired
frequency by taking adjustments in freq and time

Have you verified that the programmed frequency is correct?

Post by Makarand Kulkarni
b) logic that determines if any of the PTP packet is received or transmitted
c) latch the timestamp and generate interrupt when event b) occurs

[ So there is a potential that the time stamps get dropped or
associated with the wrong packets. ]
In order to make sure the driver and hardware are working properly,
you can perform a few simple tests.
* clock sanity check
testptp -s
testptp -g; date
#
# are the times the same?
#
testptp -g; sleep 1; testptp -g
#
# has one second transpired?
#
testptp -g; sleep 10; testptp -g
#
# has ten seconds transpired?
#
testptp -f 10000
#
# That shifts the frequency by 10 PPM.
# Now repeat the above tests.
#
testptp -f -10000
#
# Ditto for -10 PPM.
#
* time stamp check
Use the test program
linux/Documentation/networking/timestamping.c
to make sure that the Rx and Tx time stamps come at the expected
intervals.
Good luck,
Richard
------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
_______________________________________________
Linuxptp-devel mailing list
https://lists.sourceforge.net/lists/listinfo/linuxptp-d

[Makarand Kulkarni]

Jake,

Is there any Intel device which detects the PTP packets and timestamps
on it's own ? That would serve as a good reference for me. Most of the
ones I've seen merely provide a timestamp unit which has adjustable
frequency and time. I think this is one more level of sophistication
where HW detects and timestamps the PTP packets

Thanks,
Makarand

Post by Keller, Jacob E
It could also be an issue with your PTP functions (adjust frequency and
adjust time) being incorrect. I know several Intel drivers had overflow
issues when doing frequency adjustment which could cause the cycle
between min and max adjustment.
The easiest way to determine this is to do controlled testing using the
test_phc program Richard described.
Without seeing the driver source there is little we can do, and even
then not much because we don't know the hardware.
Thanks,
Jake

Post by Richard Cochran

Post by Makarand Kulkarni
I'm using HW timestamps. The device works this way: it has

Okay, then. Just out of curiousity, is the HW time stamping available
on all Cadence MACB/GEM devices?
Without looking at your driver, I can only give a few hints...

Post by Makarand Kulkarni
a) free running clock which can be programmed to give you desired
frequency by taking adjustments in freq and time

Have you verified that the programmed frequency is correct?

Post by Makarand Kulkarni
b) logic that determines if any of the PTP packet is received or transmitted
c) latch the timestamp and generate interrupt when event b) occurs

[ So there is a potential that the time stamps get dropped or
associated with the wrong packets. ]
In order to make sure the driver and hardware are working properly,
you can perform a few simple tests.
* clock sanity check
testptp -s
testptp -g; date
#
# are the times the same?
#
testptp -g; sleep 1; testptp -g
#
# has one second transpired?
#
testptp -g; sleep 10; testptp -g
#
# has ten seconds transpired?
#
testptp -f 10000
#
# That shifts the frequency by 10 PPM.
# Now repeat the above tests.
#
testptp -f -10000
#
# Ditto for -10 PPM.
#
* time stamp check
Use the test program
linux/Documentation/networking/timestamping.c
to make sure that the Rx and Tx time stamps come at the expected
intervals.
Good luck,
Richard
------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
_______________________________________________
Linuxptp-devel mailing list
https://lists.sourceforge.net/lists/listinfo/linuxptp-devel

[Keller, Jacob E]

All of our hardware does this (receive side). For TX, we have to request
that a packet be timestamped in its descriptor, then hardware will
timestamp and put the timestamp into registers.

For the Rx side, we tell the device what types of PTP packets and it
does timestamping on those. Some devices also support timestamping every
packet.

Have you tested your adjust frequency function to ensure that the max
frequency adjustment is correct and will not overflow? The way that
linuxptp resolves time offsets is to start correcting the frequency by
maxing the adjustment to either negative or positive.

Without the ability for hardware to detect and timestamp packets there
would be very little value in hardware timestamping clock.

How did you calculate the maximum adjustment value your driver/hardware
supports on its internal clock?

Thanks,
Jake

Post by Makarand Kulkarni
Jake,
Is there any Intel device which detects the PTP packets and timestamps
on it's own ? That would serve as a good reference for me. Most of the
ones I've seen merely provide a timestamp unit which has adjustable
frequency and time. I think this is one more level of sophistication
where HW detects and timestamps the PTP packets
Thanks,
Makarand

Post by Keller, Jacob E
It could also be an issue with your PTP functions (adjust frequency and
adjust time) being incorrect. I know several Intel drivers had overflow
issues when doing frequency adjustment which could cause the cycle
between min and max adjustment.
The easiest way to determine this is to do controlled testing using the
test_phc program Richard described.
Without seeing the driver source there is little we can do, and even
then not much because we don't know the hardware.
Thanks,
Jake

Post by Richard Cochran

Post by Makarand Kulkarni
I'm using HW timestamps. The device works this way: it has

Okay, then. Just out of curiousity, is the HW time stamping available
on all Cadence MACB/GEM devices?
Without looking at your driver, I can only give a few hints...

Post by Makarand Kulkarni
a) free running clock which can be programmed to give you desired
frequency by taking adjustments in freq and time

Have you verified that the programmed frequency is correct?

Post by Makarand Kulkarni
b) logic that determines if any of the PTP packet is received or transmitted
c) latch the timestamp and generate interrupt when event b) occurs

[ So there is a potential that the time stamps get dropped or
associated with the wrong packets. ]
In order to make sure the driver and hardware are working properly,
you can perform a few simple tests.
* clock sanity check
testptp -s
testptp -g; date
#
# are the times the same?
#
testptp -g; sleep 1; testptp -g
#
# has one second transpired?
#
testptp -g; sleep 10; testptp -g
#
# has ten seconds transpired?
#
testptp -f 10000
#
# That shifts the frequency by 10 PPM.
# Now repeat the above tests.
#
testptp -f -10000
#
# Ditto for -10 PPM.
#
* time stamp check
Use the test program
linux/Documentation/networking/timestamping.c
to make sure that the Rx and Tx time stamps come at the expected
intervals.
Good luck,
Richard
------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
_______________________________________________
Linuxptp-devel mailing list
https://lists.sourceforge.net/lists/listinfo/linux

[Makarand Kulkarni]

Post by Keller, Jacob E
All of our hardware does this (receive side). For TX, we have to request
that a packet be timestamped in its descriptor, then hardware will
timestamp and put the timestamp into registers.
For the Rx side, we tell the device what types of PTP packets and it
does timestamping on those. Some devices also support timestamping every
packet.
Have you tested your adjust frequency function to ensure that the max
frequency adjustment is correct and will not overflow? The way that
linuxptp resolves time offsets is to start correcting the frequency by
maxing the adjustment to either negative or positive.

Ah! this is something I need to check. I'm curious as to why does the code
go for correcting frequency when it might just adjust the time offsets ?

For instance, consider the case I have: there are 2 boards with same clock,
one as master and other as slave. Since both came up at different times,
their clocks are offset by some constant value. But running linuxptp I see
that that the frequency adjustments swinging on both ends.

Shouldn't it just adjust the clocks first and then look for drift over
certain period of time ?

Post by Keller, Jacob E
Without the ability for hardware to detect and timestamp packets there
would be very little value in hardware timestamping clock.
How did you calculate the maximum adjustment value your driver/hardware
supports on its internal clock?
Thanks,
Jake

Post by Makarand Kulkarni
Jake,
Is there any Intel device which detects the PTP packets and timestamps
on it's own ? That would serve as a good reference for me. Most of the
ones I've seen merely provide a timestamp unit which has adjustable
frequency and time. I think this is one more level of sophistication
where HW detects and timestamps the PTP packets
Thanks,
Makarand

Post by Keller, Jacob E
It could also be an issue with your PTP functions (adjust frequency and
adjust time) being incorrect. I know several Intel drivers had overflow
issues when doing frequency adjustment which could cause the cycle
between min and max adjustment.
The easiest way to determine this is to do controlled testing using the
test_phc program Richard described.
Without seeing the driver source there is little we can do, and even
then not much because we don't know the hardware.
Thanks,
Jake

Post by Richard Cochran

Post by Makarand Kulkarni
I'm using HW timestamps. The device works this way: it has

Okay, then. Just out of curiousity, is the HW time stamping available
on all Cadence MACB/GEM devices?
Without looking at your driver, I can only give a few hints...

Post by Makarand Kulkarni
a) free running clock which can be programmed to give you desired
frequency by taking adjustments in freq and time

Have you verified that the programmed frequency is correct?

Post by Makarand Kulkarni
b) logic that determines if any of the PTP packet is received or transmitted
c) latch the timestamp and generate interrupt when event b) occurs

[ So there is a potential that the time stamps get dropped or
associated with the wrong packets. ]
In order to make sure the driver and hardware are working properly,
you can perform a few simple tests.
* clock sanity check
testptp -s
testptp -g; date
#
# are the times the same?
#
testptp -g; sleep 1; testptp -g
#
# has one second transpired?
#
testptp -g; sleep 10; testptp -g
#
# has ten seconds transpired?
#
testptp -f 10000
#
# That shifts the frequency by 10 PPM.
# Now repeat the above tests.
#
testptp -f -10000
#
# Ditto for -10 PPM.
#
* time stamp check
Use the test program
linux/Documentation/networking/timestamping.c
to make sure that the Rx and Tx time stamps come at the expected
intervals.
Good luck,
Richard
------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
_______________________________________________
Linuxptp-devel mailing list
https://lists.sourceforge.net/lists/listinfo/linuxptp-devel

[Richard Cochran]

Post by Makarand Kulkarni
Shouldn't it just adjust the clocks first and then look for drift over
certain period of time ?

Yes, and that is exactly what the code does.

Thanks,
Richard

[Makarand Kulkarni]

Hi Richard,

Post by Makarand Kulkarni
a) free running clock which can be programmed to give you desired
frequency by taking adjustments in freq and time
Have you verified that the programmed frequency is correct?

Is there some method/tool which can be used ? Or I'll have to measure
time over several cycles to verify the frequency

Post by Makarand Kulkarni
b) logic that determines if any of the PTP packet is received or transmitted
c) latch the timestamp and generate interrupt when event b) occurs
[ So there is a potential that the time stamps get dropped or
associated with the wrong packets. ]

I don't think this would be a possibility with the current linuxptp
implementation. On Tx side, we do an immediate read
on UDP socket, being a single threaded application, this won't cause
another message read to happen. While matching the
REQ/RSP pairs, we are looking at sequence IDs

On the Rx side, the network code moves the packet out of normal
processing path and an interrupt is issued which reads
the timestamp from different set of registers than the Tx and
re-enqueues the packet. So this too can not have a timestamp loss

I've a question here though: is netif_rx the right function to put the
packet back into netif ?

Post by Makarand Kulkarni
In order to make sure the driver and hardware are working properly,
you can perform a few simple tests.

I ran all these tests, they work fine.

Post by Makarand Kulkarni
* clock sanity check
testptp -s
testptp -g; date
#
# are the times the same?
#
testptp -g; sleep 1; testptp -g
#
# has one second transpired?
#
testptp -g; sleep 10; testptp -g
#
# has ten seconds transpired?
#
testptp -f 10000
#
# That shifts the frequency by 10 PPM.
# Now repeat the above tests.
#
testptp -f -10000
#
# Ditto for -10 PPM.
#
* time stamp check
Use the test program
linux/Documentation/networking/timestamping.c
to make sure that the Rx and Tx time stamps come at the expected
intervals.
Good luck,
Richard

[Richard Cochran]

Post by Makarand Kulkarni
Hi Richard,

Post by Makarand Kulkarni
a) free running clock which can be programmed to give you desired
frequency by taking adjustments in freq and time
Have you verified that the programmed frequency is correct?

Is there some method/tool which can be used ? Or I'll have to
measure time over several cycles to verify the frequency

If your driver passes the simple "eye ball" tests using testptp that I
mentioned, then you can use the PTP_SYS_OFFSET ioctl to estimate the
drift between CLOCK_MONOTONIC and your PHC. Here is how:

1. testptp -f 0
2. measure the drift over, say 60 seconds
3. testptp -f x
4. measure the drift again, it should be off by "x"
5. repeat for various values of "x"
(don't forget the extremes of 0, +max_adj, and -max_adj

Post by Makarand Kulkarni
On the Rx side, the network code moves the packet out of normal
processing path and an interrupt is issued which reads
the timestamp from different set of registers than the Tx and
re-enqueues the packet. So this too can not have a timestamp loss

What happens if two event messages occur, one immediately after the
other, before you read the time stamp? Is there a time stamp queue in
hardware?

Is your matching code robustly handling if a time stamp or a packet
gets dropped?

Post by Makarand Kulkarni
I've a question here though: is netif_rx the right function to put
the packet back into netif ?

That depends on the context. NAPI drivers should use netif_receive_skb
but remember that:

* This function may only be called from softirq context and interrupts
* should be enabled.

Thanks,
Richard

[Makarand Kulkarni]

Richard,
Yes there is some issue with programming the frequency, few comments inline
-Makarand

Post by Richard Cochran

Post by Makarand Kulkarni
Hi Richard,

Post by Makarand Kulkarni
a) free running clock which can be programmed to give you desired
frequency by taking adjustments in freq and time
Have you verified that the programmed frequency is correct?

Is there some method/tool which can be used ? Or I'll have to
measure time over several cycles to verify the frequency

If your driver passes the simple "eye ball" tests using testptp that I
mentioned, then you can use the PTP_SYS_OFFSET ioctl to estimate the
1. testptp -f 0
2. measure the drift over, say 60 seconds
3. testptp -f x
4. measure the drift again, it should be off by "x"
5. repeat for various values of "x"
(don't forget the extremes of 0, +max_adj, and -max_adj

This was quite helpful. I added an option to testptp to read the
counters and report delta given a time interval
On an untouched system it reports cycles correctly. However after I
apply the corrections things don't seem to
go fine. For example, I increased the freq by 10KHz and again measured
number of cycles per sec. It doesn't seem to reflect the expected change

For clarification, I measured time with nanosleep which serves as
reference to me.

Let me dig at the frequency adjustment a little more

Post by Richard Cochran

Post by Makarand Kulkarni
On the Rx side, the network code moves the packet out of normal
processing path and an interrupt is issued which reads
the timestamp from different set of registers than the Tx and
re-enqueues the packet. So this too can not have a timestamp loss

What happens if two event messages occur, one immediately after the
other, before you read the time stamp? Is there a time stamp queue in
hardware?
Is your matching code robustly handling if a time stamp or a packet
gets dropped?

At this moment: no. There is no queue in HW, it needs to be implemented in
driver. That's on TODO list, won't impact if there are not several
masters and topology is peer-peer

Post by Richard Cochran

Post by Makarand Kulkarni
I've a question here though: is netif_rx the right function to put
the packet back into netif ?
That depends on the context. NAPI drivers should use netif_receive_skb
* This function may only be called from softirq context and interrupts
* should be enabled.

Thanks,
Richard

[Richard Cochran]

Post by Makarand Kulkarni
This was quite helpful. I added an option to testptp to read the
counters and report delta given a time interval
On an untouched system it reports cycles correctly. However after I
apply the corrections things don't seem to
go fine. For example, I increased the freq by 10KHz and again
measured number of cycles per sec. It doesn't seem to reflect the
expected change

Just to clarify, the '-f' for the testptp program does not set the
frequency. It adjusts the frequency by so many ppb. Thus running

testptp -f 10000

will increase the frequency by 10 PPM.

Post by Makarand Kulkarni

Post by Richard Cochran
Is your matching code robustly handling if a time stamp or a packet
gets dropped?

At this moment: no. There is no queue in HW, it needs to be implemented in
driver. That's on TODO list, won't impact if there are not several
masters and topology is peer-peer

Well, if there is no HW queue, then you can have missing Rx time
stamps. For example, P2P mode, if any two of {Sync, Pdelay_Req,
Pdelay_Resp} arrive at nearly the same time, then the second time
stamp could go missing.

HTH,
Richard

[Makarand Kulkarni]

Post by Richard Cochran

Post by Makarand Kulkarni
This was quite helpful. I added an option to testptp to read the
counters and report delta given a time interval
On an untouched system it reports cycles correctly. However after I
apply the corrections things don't seem to
go fine. For example, I increased the freq by 10KHz and again
measured number of cycles per sec. It doesn't seem to reflect the
expected change

Just to clarify, the '-f' for the testptp program does not set the
frequency. It adjusts the frequency by so many ppb. Thus running
testptp -f 10000
will increase the frequency by 10 PPM.

Understood, only then it goes through adjfreq call

Post by Richard Cochran

Post by Makarand Kulkarni

Post by Richard Cochran
Is your matching code robustly handling if a time stamp or a packet
gets dropped?

At this moment: no. There is no queue in HW, it needs to be implemented in
driver. That's on TODO list, won't impact if there are not several
masters and topology is peer-peer

Well, if there is no HW queue, then you can have missing Rx time
stamps. For example, P2P mode, if any two of {Sync, Pdelay_Req,
Pdelay_Resp} arrive at nearly the same time, then the second time
stamp could go missing.

Yes, I'm trying E2E now. I see this happening even when you have several
masters

Post by Richard Cochran
HTH,
Richard

[Makarand Kulkarni]

On a little different line, is there a guideline on choosing
DEFAULT_ADDEND ?

I would park it somewhere midway between the frequency range.

-Makarand

Post by Richard Cochran

Post by Makarand Kulkarni
This was quite helpful. I added an option to testptp to read the
counters and report delta given a time interval
On an untouched system it reports cycles correctly. However after I
apply the corrections things don't seem to
go fine. For example, I increased the freq by 10KHz and again
measured number of cycles per sec. It doesn't seem to reflect the
expected change

Just to clarify, the '-f' for the testptp program does not set the
frequency. It adjusts the frequency by so many ppb. Thus running
testptp -f 10000
will increase the frequency by 10 PPM.

Post by Makarand Kulkarni

Post by Richard Cochran
Is your matching code robustly handling if a time stamp or a packet
gets dropped?

At this moment: no. There is no queue in HW, it needs to be implemented in
driver. That's on TODO list, won't impact if there are not several
masters and topology is peer-peer

Well, if there is no HW queue, then you can have missing Rx time
stamps. For example, P2P mode, if any two of {Sync, Pdelay_Req,
Pdelay_Resp} arrive at nearly the same time, then the second time
stamp could go missing.
HTH,
Richard

[Richard Cochran]

Post by Makarand Kulkarni
On a little different line, is there a guideline on choosing
DEFAULT_ADDEND ?

If your device is anything like the gianfar or the ixp, then what you
want is to choose a nominal frequency value so that the ratio of the
input clock frequency to the nominal frequency is greater than 1 but
as small as possible. For example, for the gianfar:

TimerOsc = system clock MHz
tclk_period = desired clock period nanoseconds
NominalFreq = 1000 / tclk_period MHz
FreqDivRatio = TimerOsc / NominalFreq (must be greater that 1.0)
tmr_add = ceil(2^32 / FreqDivRatio)

Also, check the data sheet, users manual, or app note for your part.

HTH,
Richard

[Makarand Kulkarni]

I guess the last part (tmr_add = ceil(2^32 / FreqDivRatio) comes from
the fact that operating under system clock, the device counts till 2^32
per sec, right ?

I did make corrections in frequency adjustments and the testptp -f works okaish (since I'm losing some precision while I program the new addend). But it still doesn't help me understand my initial observation - why does frequency adjustment values swing between +/- max adjustments. Plus these changes are abrupt, meaning at one point, the adjustment goes exactly on opposite side. I'd expect the "amplitude" to die down slowly as the adjustment approaches zero

an 1 00:01:27 at501-2 user.info ptp4l: [73.105] master offset 822150571 s2 freq [phone] path delay 998638020
Jan 1 00:01:28 at501-2 user.debug ptp4l: [73.890] port 1 : sent message DELAY_REQ seq = 30 at 65949240305 (t3)
Jan 1 00:01:28 at501-2 user.debug ptp4l: [73.891] port 1 : recv message DELAY_RESP seq = 29 at 65616476264 (t4)
Jan 1 00:01:28 at501-2 user.debug ptp4l: [73.892] path delay 1043934812 1154211864
Jan 1 00:01:28 at501-2 user.[ 74.050000] diff = 11018036 cns = 2 acns = 308 nit = 346
debug ptp4l: [74.116] port 1 : recv message FUP seq = 36 at 64830824203 (t1)
Jan 1 00:01:28 at501-2 user.debug ptp4l: [74.116] port 1 : recv message SYNC seq = 36 at 66037574110 (t2)
Jan 1 00:01:28 at501-2 user.debug ptp4l: [74.117] clock_synchronize : path delay = 1043934812
Jan 1 00:01:28 at501-2 user.info ptp4l: [74.118] master offset 162815095 s2 freq [phone] path delay 1043934812 <-------- note +ve freq adjustment
Jan 1 00:01:29 at501-2 user.[ 75.060000] diff = 11379075 cns = 2 acns = 332 nit = 334
debug ptp4l: [75.127] port 1 : recv message FUP seq = 37 at 65841810556 (t1)
Jan 1 00:01:29 at501-2 user.debug ptp4l: [75.127] port 1 : recv message SYNC seq = 37 at 66506045587 (t2)
Jan 1 00:01:29 at501-2 user.debug ptp4l: [75.128] clock_synchronize : path delay = 1043934812
Jan 1 00:01:29 at501-2 user.info ptp4l: [75.129] master offset -379699781 s2 freq -32767999 path delay 1043934812 <-------- now -ve freq adjustment

Post by Richard Cochran

Post by Makarand Kulkarni
On a little different line, is there a guideline on choosing
DEFAULT_ADDEND ?

If your device is anything like the gianfar or the ixp, then what you
want is to choose a nominal frequency value so that the ratio of the
input clock frequency to the nominal frequency is greater than 1 but
TimerOsc = system clock MHz
tclk_period = desired clock period nanoseconds
NominalFreq = 1000 / tclk_period MHz
FreqDivRatio = TimerOsc / NominalFreq (must be greater that 1.0)
tmr_add = ceil(2^32 / FreqDivRatio)
Also, check the data sheet, users manual, or app note for your part.
HTH,
Richard

[Richard Cochran]

Post by Makarand Kulkarni
I guess the last part (tmr_add = ceil(2^32 / FreqDivRatio) comes
from the fact that operating under system clock, the device counts
till 2^32 per sec, right ?

No, it is because the addend and accumulator registers are 32 bits
wide.

Post by Makarand Kulkarni
I did make corrections in frequency adjustments and the testptp -f works okaish (since I'm losing some precision while I program the new addend). But it still doesn't help me understand my initial observation - why does frequency adjustment values swing between +/- max adjustments. Plus these changes are abrupt, meaning at one point, the adjustment goes exactly on opposite side. I'd expect the "amplitude" to die down slowly as the adjustment approaches zero

I can only say again that this behavior means either:

1. the oscillator is off by more than the max adjustment, or
2. the time stamps are bogus and misleading, or
3. the frequency adjustment does not work.

HTH,
Richard

[Keller, Jacob E]

-----Original Message-----
Sent: Tuesday, September 03, 2013 11:58 AM
To: Makarand Kulkarni
Subject: Re: [Linuxptp-devel] need help (newbie)

Post by Makarand Kulkarni
I guess the last part (tmr_add = ceil(2^32 / FreqDivRatio) comes
from the fact that operating under system clock, the device counts
till 2^32 per sec, right ?

No, it is because the addend and accumulator registers are 32 bits
wide.

Post by Makarand Kulkarni
I did make corrections in frequency adjustments and the testptp -f

works okaish (since I'm losing some precision while I program the new
addend). But it still doesn't help me understand my initial observation -
why does frequency adjustment values swing between +/- max
adjustments. Plus these changes are abrupt, meaning at one point, the
adjustment goes exactly on opposite side. I'd expect the "amplitude" to
die down slowly as the adjustment approaches zero
1. the oscillator is off by more than the max adjustment, or
2. the time stamps are bogus and misleading, or
3. the frequency adjustment does not work.
HTH,
Richard

Can you show us how you calculated the maximum allowed ppb frequency adjustment? I believe the possibility is because your frequency adjustment max value is wrong, and you are actually overflowing. This happened in one of the Intel parts, because the max allowed frequency was too high, so our frequency adjustment calculation actually froze the clock, which caused PTP4l to go crazy as it would apply maximum adjustment one direction, and suddenly the clock would stop so next time it was now off by a lot more in the other direction.

Thanks,
Jake

[Makarand Kulkarni]

Jacob,

I don't have an "freq adjustment" register, I've to just reprogram the
freq. So I'm not sure if there's a cap on adjustment except that
implied by the size of data structure. Let me explain what I'm doing,
maybe you can point out something here

My oscillator runs at 132MHz (osc_freq), so the adjustments which I get
(ppb) are over this freq

adj = curr_freq

adj *= ppb

delta = adj / 1000000000 (since the adjustment is in part per billion)

new_freq = osc_freq + delta (osc freq is signed)

At this point, I've to program this freq in such a way that:

cns * nit + acns proper divisor of 1000000000 where

cns = nanosec count
acns = alternative nanosec count
nit = number of increments before we use acns.

all of above are 8 bit quantities.

The idea is to fill these values in such a way that the number of cycles
you count equates with the freq you deserve till 1billion ns is reached
This last part makes me lose some precision, but this is the constraint
for me



One more question on the existing use of DEFAULT_ADDEND: doesn't the
adjustment come over and above the existing frequency programmed ? I see in
the code that every time adjustment is taken over DEFAULT_ADDEND, which
seemed little wierd to me




Best Regards,


Makarand

Post by Keller, Jacob E

-----Original Message-----
Sent: Tuesday, September 03, 2013 11:58 AM
To: Makarand Kulkarni
Subject: Re: [Linuxptp-devel] need help (newbie)

Post by Makarand Kulkarni
I guess the last part (tmr_add = ceil(2^32 / FreqDivRatio) comes
from the fact that operating under system clock, the device counts
till 2^32 per sec, right ?

No, it is because the addend and accumulator registers are 32 bits
wide.

Post by Makarand Kulkarni
I did make corrections in frequency adjustments and the testptp -f

works okaish (since I'm losing some precision while I program the new
addend). But it still doesn't help me understand my initial observation -
why does frequency adjustment values swing between +/- max
adjustments. Plus these changes are abrupt, meaning at one point, the
adjustment goes exactly on opposite side. I'd expect the "amplitude" to
die down slowly as the adjustment approaches zero
1. the oscillator is off by more than the max adjustment, or
2. the time stamps are bogus and misleading, or
3. the frequency adjustment does not work.
HTH,
Richard

Can you show us how you calculated the maximum allowed ppb frequency adjustment? I believe the possibility is because your frequency adjustment max value is wrong, and you are actually overflowing. This happened in one of the Intel parts, because the max allowed frequency was too high, so our frequency adjustment calculation actually froze the clock, which caused PTP4l to go crazy as it would apply maximum adjustment one direction, and suddenly the clock would stop so next time it was now off by a lot more in the other direction.
Thanks,
Jake

[Makarand Kulkarni]

Jake,

I've sent the scheme on earlier email, but someone help me understand this:
if ppb is just a ratio which indicates deviation in a billion, when we
adjust frequency, we should remember what we adjusted to last time.
But in existing ptp driver or pch or intel device, I see that we always
start from DEFAULT_ADDEND

addend = DEFAULT_ADDEND;
adj = addend;
adj *= ppb;
diff = div_u64(adj, 1000000000ULL);

addend = neg_adj ? addend - diff : addend + diff;

If you've already maxed the adjustment, then this will always be clamped
at max value. Is this correct ?

Best Regards,


Makarand

Post by Keller, Jacob E

-----Original Message-----
Sent: Tuesday, September 03, 2013 11:58 AM
To: Makarand Kulkarni
Subject: Re: [Linuxptp-devel] need help (newbie)

Post by Makarand Kulkarni
I guess the last part (tmr_add = ceil(2^32 / FreqDivRatio) comes
from the fact that operating under system clock, the device counts
till 2^32 per sec, right ?

No, it is because the addend and accumulator registers are 32 bits
wide.

Post by Makarand Kulkarni
I did make corrections in frequency adjustments and the testptp -f

works okaish (since I'm losing some precision while I program the new
addend). But it still doesn't help me understand my initial observation -
why does frequency adjustment values swing between +/- max
adjustments. Plus these changes are abrupt, meaning at one point, the
adjustment goes exactly on opposite side. I'd expect the "amplitude" to
die down slowly as the adjustment approaches zero
1. the oscillator is off by more than the max adjustment, or
2. the time stamps are bogus and misleading, or
3. the frequency adjustment does not work.
HTH,
Richard

Can you show us how you calculated the maximum allowed ppb frequency adjustment? I believe the possibility is because your frequency adjustment max value is wrong, and you are actually overflowing. This happened in one of the Intel parts, because the max allowed frequency was too high, so our frequency adjustment calculation actually froze the clock, which caused PTP4l to go crazy as it would apply maximum adjustment one direction, and suddenly the clock would stop so next time it was now off by a lot more in the other direction.
Thanks,
Jake

[Keller, Jacob E]

Post by Makarand Kulkarni
Jake,
if ppb is just a ratio which indicates deviation in a billion, when we
adjust frequency, we should remember what we adjusted to last time.
But in existing ptp driver or pch or intel device, I see that we always
start from DEFAULT_ADDEND
addend = DEFAULT_ADDEND;
adj = addend;
adj *= ppb;
diff = div_u64(adj, 1000000000ULL);
addend = neg_adj ? addend - diff : addend + diff;
If you've already maxed the adjustment, then this will always be clamped
at max value. Is this correct ?
Best Regards,

Makarand,

Correct. You need to start at the "base" value, because the ppb given
via the ptp ioctl is actually a measure from the base, not a measure
from the current. The reason for that is so that passing "0" should
reset you to the default addend.

Also it makes the max value make more sense. If you keep applying the
value from the "current" you have no max frequency adjustment from
nominal frequency. The max adjustment should be the largest (positive or
negative, whichever is smaller!) value that you can adjust by without
overflowing the counters.

I believe that is why your code is not working. It is because if you
adjust from the current value, all the calculations done by ptp4l to
determine the offset are not correct. Hence why you get the crazy graph
of offset that you do.

This occurs because ptp4l uses the P/I servo to accumulate offset
calculation and prevent outlying data from swinging the clock too fast.
However, it continually applies the same value every time it detects the
change. You are then re-applying the value over and over.

Your hardware is very similar to how the Intel parts work. I would
suggest re-coding it using the assumption that the ppb will be measured
from the nominal frequency..

Does this make sense?

Regards,
Jake

Post by Makarand Kulkarni
Makarand

Post by Keller, Jacob E

-----Original Message-----
Sent: Tuesday, September 03, 2013 11:58 AM
To: Makarand Kulkarni
Subject: Re: [Linuxptp-devel] need help (newbie)

Post by Makarand Kulkarni
I guess the last part (tmr_add = ceil(2^32 / FreqDivRatio) comes
from the fact that operating under system clock, the device counts
till 2^32 per sec, right ?

No, it is because the addend and accumulator registers are 32 bits
wide.

Post by Makarand Kulkarni
I did make corrections in frequency adjustments and the testptp -f

works okaish (since I'm losing some precision while I program the new
addend). But it still doesn't help me understand my initial observation -
why does frequency adjustment values swing between +/- max
adjustments. Plus these changes are abrupt, meaning at one point, the
adjustment goes exactly on opposite side. I'd expect the "amplitude" to
die down slowly as the adjustment approaches zero
1. the oscillator is off by more than the max adjustment, or
2. the time stamps are bogus and misleading, or
3. the frequency adjustment does not work.
HTH,
Richard

Can you show us how you calculated the maximum allowed ppb frequency adjustment? I believe the possibility is because your frequency adjustment max value is wrong, and you are actually overflowing. This happened in one of the Intel parts, because the max allowed frequency was too high, so our frequency adjustment calculation actually froze the clock, which caused PTP4l to go crazy as it would apply maximum adjustment one direction, and suddenly the clock would stop so next time it was now off by a lot more in the other direction.

[Makarand Kulkarni]

Hmmm.. if PI controller has memory of past adjustments, yes it makes
sense for driver to take ppb with respect to DEFAULT_ADDEND, it also
helps me appreciate why we need default addend.
Let me make the changes and try

Meanwhile, I could verify the SW timestamping, so it's surely a bug with
driver

Thank you so much

Best Regards,


Makarand

Post by Keller, Jacob E

Post by Makarand Kulkarni
Jake,
if ppb is just a ratio which indicates deviation in a billion, when we
adjust frequency, we should remember what we adjusted to last time.
But in existing ptp driver or pch or intel device, I see that we always
start from DEFAULT_ADDEND
addend = DEFAULT_ADDEND;
adj = addend;
adj *= ppb;
diff = div_u64(adj, 1000000000ULL);
addend = neg_adj ? addend - diff : addend + diff;
If you've already maxed the adjustment, then this will always be clamped
at max value. Is this correct ?
Best Regards,

Makarand,
Correct. You need to start at the "base" value, because the ppb given
via the ptp ioctl is actually a measure from the base, not a measure
from the current. The reason for that is so that passing "0" should
reset you to the default addend.
Also it makes the max value make more sense. If you keep applying the
value from the "current" you have no max frequency adjustment from
nominal frequency. The max adjustment should be the largest (positive or
negative, whichever is smaller!) value that you can adjust by without
overflowing the counters.
I believe that is why your code is not working. It is because if you
adjust from the current value, all the calculations done by ptp4l to
determine the offset are not correct. Hence why you get the crazy graph
of offset that you do.
This occurs because ptp4l uses the P/I servo to accumulate offset
calculation and prevent outlying data from swinging the clock too fast.
However, it continually applies the same value every time it detects the
change. You are then re-applying the value over and over.
Your hardware is very similar to how the Intel parts work. I would
suggest re-coding it using the assumption that the ppb will be measured
from the nominal frequency..
Does this make sense?
Regards,
Jake

Post by Makarand Kulkarni
Makarand

Post by Keller, Jacob E

-----Original Message-----
Sent: Tuesday, September 03, 2013 11:58 AM
To: Makarand Kulkarni
Subject: Re: [Linuxptp-devel] need help (newbie)

Post by Makarand Kulkarni
I guess the last part (tmr_add = ceil(2^32 / FreqDivRatio) comes
from the fact that operating under system clock, the device counts
till 2^32 per sec, right ?

No, it is because the addend and accumulator registers are 32 bits
wide.

Post by Makarand Kulkarni
I did make corrections in frequency adjustments and the testptp -f

works okaish (since I'm losing some precision while I program the new
addend). But it still doesn't help me understand my initial observation -
why does frequency adjustment values swing between +/- max
adjustments. Plus these changes are abrupt, meaning at one point, the
adjustment goes exactly on opposite side. I'd expect the "amplitude" to
die down slowly as the adjustment approaches zero
1. the oscillator is off by more than the max adjustment, or
2. the time stamps are bogus and misleading, or
3. the frequency adjustment does not work.
HTH,
Richard

Can you show us how you calculated the maximum allowed ppb frequency adjustment? I believe the possibility is because your frequency adjustment max value is wrong, and you are actually overflowing. This happened in one of the Intel parts, because the max allowed frequency was too high, so our frequency adjustment calculation actually froze the clock, which caused PTP4l to go crazy as it would apply maximum adjustment one direction, and suddenly the clock would stop so next time it was now off by a lot more in the other direction.
Thanks,
Jake

[Richard Cochran]

Post by Makarand Kulkarni
I'm trying to port linuxptp to work with cadence GMAC. Test setup
consists of 2 boards connected through switch, one as master and other
as slave.

Until you have your (probable) driver issues worked out, I would limit
testing the new code on the slave only, with a known good master.

After you get the driver working, I would encourage you to post it to
the netdev list.

Thanks,
Richard