Discussion:
[Linuxptp-devel] [PATCH 0/2] Make software mode work without driver support
Andy Lutomirski
2013-07-27 00:37:43 UTC
Permalink
The kernel supports software rx timestamping on all hardware, but most
drivers don't support any kind of kernel tx timestamping. Add a userspace
fallback.

As currently implemented, the fallback is automatic if -S is specified.

I've only tested it partially because my PTP setup is somewhat screwed
up right now.

Andy Lutomirski (2):
Fix check for timestamping modes
Fall back to userspace timestamping if no SOF_TIMESTAMPING_TX_SOFTWARE

port.c | 8 ++++++++
ptp4l.c | 5 ++---
sk.c | 12 +++++++++++-
transport.h | 1 +
4 files changed, 22 insertions(+), 4 deletions(-)
--
1.8.3.1
Andy Lutomirski
2013-07-27 00:37:45 UTC
Permalink
This allows a completely software implementation if driver support is not
available.

Signed-off-by: Andy Lutomirski <***@amacapital.net>
---
port.c | 8 ++++++++
ptp4l.c | 3 +--
sk.c | 12 +++++++++++-
transport.h | 1 +
4 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/port.c b/port.c
index f03aa05..a4d8efa 100644
--- a/port.c
+++ b/port.c
@@ -22,6 +22,7 @@
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
+#include <linux/net_tstamp.h>

#include "bmc.h"
#include "clock.h"
@@ -2170,6 +2171,13 @@ struct port *port_open(int phc_index,
return NULL;
}

+ if (timestamping == TS_SOFTWARE &&
+ !(interface->ts_info.so_timestamping &
+ SOF_TIMESTAMPING_TX_SOFTWARE)) {
+ pr_warning("port %d: falling back to userspace tx timestamping");
+ timestamping = TS_SOFTWARE_USERTX;
+ }
+
p->pod = interface->pod;
p->name = interface->name;
p->clock = clock;
diff --git a/ptp4l.c b/ptp4l.c
index a83e3fe..4b0c7a7 100644
--- a/ptp4l.c
+++ b/ptp4l.c
@@ -315,8 +315,7 @@ int main(int argc, char *argv[])

switch (*timestamping) {
case TS_SOFTWARE:
- required_modes |= SOF_TIMESTAMPING_TX_SOFTWARE |
- SOF_TIMESTAMPING_RX_SOFTWARE |
+ required_modes |= SOF_TIMESTAMPING_RX_SOFTWARE |
SOF_TIMESTAMPING_SOFTWARE;
break;
case TS_LEGACY_HW:
diff --git a/sk.c b/sk.c
index 0437ad8..d87fdc0 100644
--- a/sk.c
+++ b/sk.c
@@ -214,6 +214,11 @@ int sk_receive(int fd, void *buf, int buflen,
msg.msg_control = control;
msg.msg_controllen = sizeof(control);

+ if (flags == MSG_ERRQUEUE && hwts->type == TS_SOFTWARE_USERTX) {
+ clock_gettime(CLOCK_REALTIME, &hwts->ts);
+ return 1;
+ }
+
if (flags == MSG_ERRQUEUE) {
struct pollfd pfd = { fd, 0, 0 };
res = poll(&pfd, 1, sk_tx_timeout);
@@ -252,6 +257,7 @@ int sk_receive(int fd, void *buf, int buflen,

switch (hwts->type) {
case TS_SOFTWARE:
+ case TS_SOFTWARE_USERTX:
hwts->ts = ts[0];
break;
case TS_HARDWARE:
@@ -276,6 +282,10 @@ int sk_timestamping_init(int fd, char *device, enum timestamp_type type,
SOF_TIMESTAMPING_RX_SOFTWARE |
SOF_TIMESTAMPING_SOFTWARE;
break;
+ case TS_SOFTWARE_USERTX:
+ flags = SOF_TIMESTAMPING_RX_SOFTWARE |
+ SOF_TIMESTAMPING_SOFTWARE;
+ break;
case TS_HARDWARE:
case TS_ONESTEP:
flags = SOF_TIMESTAMPING_TX_HARDWARE |
@@ -291,7 +301,7 @@ int sk_timestamping_init(int fd, char *device, enum timestamp_type type,
return -1;
}

- if (type != TS_SOFTWARE) {
+ if (type != TS_SOFTWARE && type != TS_SOFTWARE_USERTX) {
filter1 = HWTSTAMP_FILTER_PTP_V2_EVENT;
one_step = type == TS_ONESTEP ? 1 : 0;
switch (transport) {
diff --git a/transport.h b/transport.h
index 46af456..68c3c4d 100644
--- a/transport.h
+++ b/transport.h
@@ -49,6 +49,7 @@ enum transport_event {

enum timestamp_type {
TS_SOFTWARE,
+ TS_SOFTWARE_USERTX,
TS_HARDWARE,
TS_LEGACY_HW,
TS_ONESTEP,
--
1.8.3.1
Keller, Jacob E
2013-07-29 19:03:09 UTC
Permalink
Hi Andy,
Post by Andy Lutomirski
This allows a completely software implementation if driver support is not
available.
---
port.c | 8 ++++++++
ptp4l.c | 3 +--
sk.c | 12 +++++++++++-
transport.h | 1 +
4 files changed, 21 insertions(+), 3 deletions(-)
diff --git a/port.c b/port.c
index f03aa05..a4d8efa 100644
--- a/port.c
+++ b/port.c
@@ -22,6 +22,7 @@
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
+#include <linux/net_tstamp.h>
#include "bmc.h"
#include "clock.h"
@@ -2170,6 +2171,13 @@ struct port *port_open(int phc_index,
return NULL;
}
+ if (timestamping == TS_SOFTWARE &&
+ !(interface->ts_info.so_timestamping &
+ SOF_TIMESTAMPING_TX_SOFTWARE)) {
+ pr_warning("port %d: falling back to userspace tx timestamping");
+ timestamping = TS_SOFTWARE_USERTX;
+ }
+
p->pod = interface->pod;
p->name = interface->name;
p->clock = clock;
diff --git a/ptp4l.c b/ptp4l.c
index a83e3fe..4b0c7a7 100644
--- a/ptp4l.c
+++ b/ptp4l.c
@@ -315,8 +315,7 @@ int main(int argc, char *argv[])
switch (*timestamping) {
- required_modes |= SOF_TIMESTAMPING_TX_SOFTWARE |
- SOF_TIMESTAMPING_RX_SOFTWARE |
+ required_modes |= SOF_TIMESTAMPING_RX_SOFTWARE |
SOF_TIMESTAMPING_SOFTWARE;
break;
diff --git a/sk.c b/sk.c
index 0437ad8..d87fdc0 100644
--- a/sk.c
+++ b/sk.c
@@ -214,6 +214,11 @@ int sk_receive(int fd, void *buf, int buflen,
msg.msg_control = control;
msg.msg_controllen = sizeof(control);
+ if (flags == MSG_ERRQUEUE && hwts->type == TS_SOFTWARE_USERTX) {
+ clock_gettime(CLOCK_REALTIME, &hwts->ts);
+ return 1;
+ }
+
if (flags == MSG_ERRQUEUE) {
struct pollfd pfd = { fd, 0, 0 };
res = poll(&pfd, 1, sk_tx_timeout);
@@ -252,6 +257,7 @@ int sk_receive(int fd, void *buf, int buflen,
switch (hwts->type) {
hwts->ts = ts[0];
break;
@@ -276,6 +282,10 @@ int sk_timestamping_init(int fd, char *device, enum timestamp_type type,
SOF_TIMESTAMPING_RX_SOFTWARE |
SOF_TIMESTAMPING_SOFTWARE;
break;
+ flags = SOF_TIMESTAMPING_RX_SOFTWARE |
+ SOF_TIMESTAMPING_SOFTWARE;
+ break;
flags = SOF_TIMESTAMPING_TX_HARDWARE |
@@ -291,7 +301,7 @@ int sk_timestamping_init(int fd, char *device, enum timestamp_type type,
return -1;
}
- if (type != TS_SOFTWARE) {
+ if (type != TS_SOFTWARE && type != TS_SOFTWARE_USERTX) {
filter1 = HWTSTAMP_FILTER_PTP_V2_EVENT;
one_step = type == TS_ONESTEP ? 1 : 0;
switch (transport) {
diff --git a/transport.h b/transport.h
index 46af456..68c3c4d 100644
--- a/transport.h
+++ b/transport.h
@@ -49,6 +49,7 @@ enum transport_event {
enum timestamp_type {
TS_SOFTWARE,
+ TS_SOFTWARE_USERTX,
TS_HARDWARE,
TS_LEGACY_HW,
TS_ONESTEP,
Are we sure that everyone would want to use fully user-space software tx
timestamping? That would have really horrid latency issues due to when
software gets the packets. In addition, it would be a different level of
latency from the rx side, so the delays would be un-even.

I understand that sometimes there is no tx-software timestamping, but I
am not sure how well this would even perform. If you have some examples
of how well userspace timestamping for TX performs could you post them
here? Maybe also re-work this patch to include a configuration option
which disables it in the case where you actually don't want
userspace-only tx timestamping?

I do understand the frustration for testing purposes when you just want
to check your part but do not have a working partner. And also just
wanting to play around with ptp4l. However, I do not really see a lot of
value in the use of userspace timestamping due to how mu
Richard Cochran
2013-07-29 19:12:18 UTC
Permalink
Post by Keller, Jacob E
I understand that sometimes there is no tx-software timestamping, but I
am not sure how well this would even perform. If you have some examples
of how well userspace timestamping for TX performs could you post them
here?
IIRC, I once tried this out, briefly, but the results were not too
good. This should definitely be tested.
Post by Keller, Jacob E
Maybe also re-work this patch to include a configuration option
which disables it in the case where you actually don't want
userspace-only tx timestamping?
Or better yet, an option to *enable* this fallback behavior, with the
default being off.

Thanks,
Richard
Andy Lutomirski
2013-07-29 19:07:10 UTC
Permalink
On Mon, Jul 29, 2013 at 12:03 PM, Keller, Jacob E
Post by Keller, Jacob E
Hi Andy,
Post by Andy Lutomirski
This allows a completely software implementation if driver support is not
available.
[...]
Are we sure that everyone would want to use fully user-space software tx
timestamping? That would have really horrid latency issues due to when
software gets the packets. In addition, it would be a different level of
latency from the rx side, so the delays would be un-even.
I understand that sometimes there is no tx-software timestamping, but I
am not sure how well this would even perform. If you have some examples
of how well userspace timestamping for TX performs could you post them
here? Maybe also re-work this patch to include a configuration option
which disables it in the case where you actually don't want
userspace-only tx timestamping?
This is actually intended for production. I have some servers with
non-PTP Solarflare SKUs. The driver is apparently fixable to have
kernel-side software support, but in the mean time, some support is
better than none.

The main goal here is to switch from NTP to PTP, which should give a
considerable improvement in timing quality.
Post by Keller, Jacob E
I do understand the frustration for testing purposes when you just want
to check your part but do not have a working partner. And also just
wanting to play around with ptp4l. However, I do not really see a lot of
value in the use of userspace timestamping due to how much latency is
introduced.
Are there good measurements for me to do?

(All measurements will have to wait until my PTP source gets fixed,
unfortunately.)

--Andy
Keller, Jacob E
2013-07-29 21:06:55 UTC
Permalink
Post by Andy Lutomirski
On Mon, Jul 29, 2013 at 12:03 PM, Keller, Jacob E
Post by Keller, Jacob E
Hi Andy,
Post by Andy Lutomirski
This allows a completely software implementation if driver support is not
available.
[...]
Are we sure that everyone would want to use fully user-space software tx
timestamping? That would have really horrid latency issues due to when
software gets the packets. In addition, it would be a different level of
latency from the rx side, so the delays would be un-even.
I understand that sometimes there is no tx-software timestamping, but I
am not sure how well this would even perform. If you have some examples
of how well userspace timestamping for TX performs could you post them
here? Maybe also re-work this patch to include a configuration option
which disables it in the case where you actually don't want
userspace-only tx timestamping?
This is actually intended for production. I have some servers with
non-PTP Solarflare SKUs. The driver is apparently fixable to have
kernel-side software support, but in the mean time, some support is
better than none.
The main goal here is to switch from NTP to PTP, which should give a
considerable improvement in timing quality.
Post by Keller, Jacob E
I do understand the frustration for testing purposes when you just want
to check your part but do not have a working partner. And also just
wanting to play around with ptp4l. However, I do not really see a lot of
value in the use of userspace timestamping due to how much latency is
introduced.
Are there good measurements for me to do?
(All measurements will have to wait until my PTP source gets fixed,
unfortunately.)
--Andy
Run ptp4l and graph/plot the path delay and offset from master over time.

And yes, Richard is correct, have this feature off by default with
ability to enable it. (that would just be a differenc
Miroslav Lichvar
2013-07-30 08:08:35 UTC
Permalink
Post by Keller, Jacob E
Post by Andy Lutomirski
This is actually intended for production. I have some servers with
non-PTP Solarflare SKUs. The driver is apparently fixable to have
kernel-side software support, but in the mean time, some support is
better than none.
The main goal here is to switch from NTP to PTP, which should give a
considerable improvement in timing quality.
With userspace timestamping PTP might not be able to improve the
quality as it depends on accuracy and stability of the path delay
measurements. NTP knows delay for each measurement and can easily
ignore outliers. I'm interested in your results.
Post by Keller, Jacob E
Post by Andy Lutomirski
Are there good measurements for me to do?
(All measurements will have to wait until my PTP source gets fixed,
unfortunately.)
--Andy
Run ptp4l and graph/plot the path delay and offset from master over time.
That can be used to estimate the jitter, but it won't show how much
asymmetry is there or how well is the clock controller. Ideally, the
accuracy is measured by comparing PPS signals from the master and
slave with a scope. If the slave has a parallel port, the
pps_gen_parport module can be used as genertor.
--
Miroslav Lichvar
Fugang Duan
2013-11-22 10:37:16 UTC
Permalink
Hi, all,

I build the ptp4l for the first time, but it build fail as below log.
I follow linuxptp doc step to build. (6.1 Linux kernel and 6.2 PTP stack)

linuxtest/ptpd/code$ make clean
rm -f bmc.o clock.o clockadj.o clockcheck.o config.o fault.o filter.o fsm.o mave.o mmedian.o msg.o phc.o pi.o port.o print.o ptp4l.o raw.o servo.o sk.o stats.o tlv.o tmtab.o transport.o udp.o udp6.o uds.o util.o version.o hwstamp_ctl.o phc2sys.o pmc.o pmc_common.o sysoff.o bmc.d clock.d clockadj.d clockcheck.d config.d fault.d filter.d fsm.d mave.d mmedian.d msg.d phc.d pi.d port.d print.d ptp4l.d raw.d servo.d sk.d stats.d tlv.d tmtab.d transport.d udp.d udp6.d uds.d util.d version.d hwstamp_ctl.d phc2sys.d pmc.d pmc_common.d sysoff.d
***@shlinux2:~/linuxtest/ptpd/code$ make
DEPEND sysoff.c
DEPEND pmc_common.c
DEPEND pmc.c
DEPEND phc2sys.c
In file included from /u/home/b38611/fsl-linux/toolchain/arm-toolchain-yacto/sysroots/i686-pokysdk-linux/usr/bin/cortexa9hf-vfp-neon-poky-linux-gnueabi/../../lib/cortexa9hf-vfp-neon-poky-linux-gnueabi/gcc/arm-poky-linux-gnueabi/4.7.2/include-fixed/syslimits.h:7:0,
from /u/home/b38611/fsl-linux/toolchain/arm-toolchain-yacto/sysroots/i686-pokysdk-linux/usr/bin/cortexa9hf-vfp-neon-poky-linux-gnueabi/../../lib/cortexa9hf-vfp-neon-poky-linux-gnueabi/gcc/arm-poky-linux-gnueabi/4.7.2/include-fixed/limits.h:34,
from phc2sys.c:24:
/u/home/b38611/fsl-linux/toolchain/arm-toolchain-yacto/sysroots/i686-pokysdk-linux/usr/bin/cortexa9hf-vfp-neon-poky-linux-gnueabi/../../lib/cortexa9hf-vfp-neon-poky-linux-gnueabi/gcc/arm-poky-linux-gnueabi/4.7.2/include-fixed/limits.h:169:61: error: no include path in which to search for limits.h
DEPEND hwstamp_ctl.c
DEPEND version.c
DEPEND util.c
DEPEND uds.c
DEPEND udp6.c
DEPEND udp.c
DEPEND transport.c
DEPEND tmtab.c
DEPEND tlv.c
DEPEND stats.c
DEPEND sk.c
DEPEND servo.c
DEPEND raw.c
DEPEND ptp4l.c
DEPEND print.c
DEPEND port.c
DEPEND pi.c
DEPEND phc.c
DEPEND msg.c
DEPEND mmedian.c
DEPEND mave.c
DEPEND fsm.c
DEPEND filter.c
DEPEND fault.c
DEPEND config.c
In file included from /u/home/b38611/fsl-linux/toolchain/arm-toolchain-yacto/sysroots/i686-pokysdk-linux/usr/bin/cortexa9hf-vfp-neon-poky-linux-gnueabi/../../lib/cortexa9hf-vfp-neon-poky-linux-gnueabi/gcc/arm-poky-linux-gnueabi/4.7.2/include-fixed/syslimits.h:7:0,
from /u/home/b38611/fsl-linux/toolchain/arm-toolchain-yacto/sysroots/i686-pokysdk-linux/usr/bin/cortexa9hf-vfp-neon-poky-linux-gnueabi/../../lib/cortexa9hf-vfp-neon-poky-linux-gnueabi/gcc/arm-poky-linux-gnueabi/4.7.2/include-fixed/limits.h:34,
from config.c:21:
/u/home/b38611/fsl-linux/toolchain/arm-toolchain-yacto/sysroots/i686-pokysdk-linux/usr/bin/cortexa9hf-vfp-neon-poky-linux-gnueabi/../../lib/cortexa9hf-vfp-neon-poky-linux-gnueabi/gcc/arm-poky-linux-gnueabi/4.7.2/include-fixed/limits.h:169:61: error: no include path in which to search for limits.h
DEPEND clockcheck.c
DEPEND clockadj.c
DEPEND clock.c
DEPEND bmc.c
arm-poky-linux-gnueabi-gcc -Wall -DVER=1.3-00032-g85a1bcf -I/home/b38611/linuxtest/ptpd/code/ptp_debian/usr/include -DHAVE_ONESTEP_SYNC -c -o ptp4l.o ptp4l.c
ptp4l.c:20:20: fatal error: signal.h: No such file or directory
compilation terminated.
make: *** [ptp4l.o] Error 1


And then I search the "KBUILD_OUTPUT" directory, it can find the signal.h.
linuxtest/ptpd/code$ find . -name "signal.h"
./ptp_debian/usr/include/asm-generic/signal.h
./ptp_debian/usr/include/asm/signal.h
./ptp_debian/usr/include/linux/signal.h


Thanks,
Andy

Andy Lutomirski
2013-07-27 00:37:44 UTC
Permalink
The code was checking the wrong interface's capabilities.

Signed-off-by: Andy Lutomirski <***@amacapital.net>
---
ptp4l.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ptp4l.c b/ptp4l.c
index 1d4a3e0..a83e3fe 100644
--- a/ptp4l.c
+++ b/ptp4l.c
@@ -335,7 +335,7 @@ int main(int argc, char *argv[])
/* check whether timestamping mode is supported. */
for (i = 0; i < cfg_settings.nports; i++) {
if (iface[i].ts_info.valid &&
- ((iface[0].ts_info.so_timestamping & required_modes) != required_modes)) {
+ ((iface[i].ts_info.so_timestamping & required_modes) != required_modes)) {
fprintf(stderr, "interface '%s' does not support "
"requested timestamping mode.\n",
iface[i].name);
--
1.8.3.1
Richard Cochran
2013-07-28 12:44:48 UTC
Permalink
Post by Andy Lutomirski
The code was checking the wrong interface's capabilities.
Applied.

Thanks,
Richard
Richard Cochran
2013-07-28 12:50:41 UTC
Permalink
Post by Andy Lutomirski
The kernel supports software rx timestamping on all hardware, but most
drivers don't support any kind of kernel tx timestamping. Add a userspace
fallback.
As currently implemented, the fallback is automatic if -S is specified.
I feel a bit ambivalent about this. One one hand, having a mode that
always works is an attractive idea, provided that the performance is
adequate. On the other hand, I really want people to either add the
one missing line into their MAC driver or ask me to do it for them.
I worry that we will never get good Tx time stamping coverage in the
kernel.

Thoughts?

Thanks,
Richard
Andy Lutomirski
2013-07-28 19:59:11 UTC
Permalink
On Sun, Jul 28, 2013 at 5:50 AM, Richard Cochran
Post by Richard Cochran
Post by Andy Lutomirski
The kernel supports software rx timestamping on all hardware, but most
drivers don't support any kind of kernel tx timestamping. Add a userspace
fallback.
As currently implemented, the fallback is automatic if -S is specified.
I feel a bit ambivalent about this. One one hand, having a mode that
always works is an attractive idea, provided that the performance is
adequate. On the other hand, I really want people to either add the
one missing line into their MAC driver or ask me to do it for them.
I worry that we will never get good Tx time stamping coverage in the
kernel.
I didn't know that this was an easy thing for the driver to do, and
the error messages from current linuxptp didn't enlighten me. I
wonder if a better solution would be to change the warning to
something like "falling back to userspace timing because your network
driver lacks software timing support -- contact your driver's
maintainer".

Now that I know, I'll see if I can write a patch for sfc.

FWIW, there may be a bigger issue with software mode (kernel and
userspace) in general: rx wakeup latency. A couple weeks ago I did
some ping experiments, and I discovered that pings (using
SO_TIMESTAMPNS) were delayed by ~50µs if the core handling the irq was
idle. This may be enough of an asymmetry to matter.

--Andy
Richard Cochran
2013-07-29 05:23:29 UTC
Permalink
Post by Andy Lutomirski
I didn't know that this was an easy thing for the driver to do, and
the error messages from current linuxptp didn't enlighten me. I
wonder if a better solution would be to change the warning to
something like "falling back to userspace timing because your network
driver lacks software timing support -- contact your driver's
maintainer".
Makes sense to me.
Post by Andy Lutomirski
Now that I know, I'll see if I can write a patch for sfc.
Solarflare?

I think they recently added HW time stamping support.
Post by Andy Lutomirski
FWIW, there may be a bigger issue with software mode (kernel and
userspace) in general: rx wakeup latency. A couple weeks ago I did
some ping experiments, and I discovered that pings (using
SO_TIMESTAMPNS) were delayed by ~50µs if the core handling the irq was
idle. This may be enough of an asymmetry to matter.
One way to monitor this (in aggregate) is to watch the "path delay"
column in the ptp4l output. Wake up latency will be reflected in that
value.

Thanks,
Richard
Andy Lutomirski
2013-07-29 06:37:36 UTC
Permalink
On Sun, Jul 28, 2013 at 10:23 PM, Richard Cochran
Post by Richard Cochran
Post by Andy Lutomirski
I didn't know that this was an easy thing for the driver to do, and
the error messages from current linuxptp didn't enlighten me. I
wonder if a better solution would be to change the warning to
something like "falling back to userspace timing because your network
driver lacks software timing support -- contact your driver's
maintainer".
Makes sense to me.
I'll submit an updated patch. I'll even try to test it for real once
my PTP provider fixes their routing table or whatever is currently
broken.
Post by Richard Cochran
Post by Andy Lutomirski
Now that I know, I'll see if I can write a patch for sfc.
Solarflare?
I think they recently added HW time stamping support.
It only works on one model card, and it's not the model I have. I
don't actually need sub-microsecond precision, but a couple of
microseconds would be nice.
Post by Richard Cochran
Post by Andy Lutomirski
FWIW, there may be a bigger issue with software mode (kernel and
userspace) in general: rx wakeup latency. A couple weeks ago I did
some ping experiments, and I discovered that pings (using
SO_TIMESTAMPNS) were delayed by ~50盜 if the core handling the irq was
idle. This may be enough of an asymmetry to matter.
One way to monitor this (in aggregate) is to watch the "path delay"
column in the ptp4l output. Wake up latency will be reflected in that
value.
If there's 50µs of wakeup latency, does anything prevent it from being
interpreted as 25µs of path delay (as opposed to 50µs, but only on the
rx side)?


--Andy
Richard Cochran
2013-07-29 11:06:09 UTC
Permalink
Post by Andy Lutomirski
It only works on one model card, and it's not the model I have. I
don't actually need sub-microsecond precision, but a couple of
microseconds would be nice.
With SW time stamping, you can expect 25 to 100 microseconds,
depending on the MAC (buffering, chaining, etc) and on system load.
Post by Andy Lutomirski
If there's 50µs of wakeup latency, does anything prevent it from being
interpreted as 25µs of path delay (as opposed to 50µs, but only on the
rx side)?
(t4 - t1) - (t3 - t2)
Path delay = --------------------
2

t1 local tx
t2 remote rx
t3 remote tx
t4 local rx

So the wake latency will only affect t4 (and presumably, only
sometimes). We also keep a moving average of length 10.

HTH,
Richard
Keller, Jacob E
2013-07-29 19:07:58 UTC
Permalink
Post by Andy Lutomirski
On Sun, Jul 28, 2013 at 5:50 AM, Richard Cochran
Post by Richard Cochran
Post by Andy Lutomirski
The kernel supports software rx timestamping on all hardware, but most
drivers don't support any kind of kernel tx timestamping. Add a userspace
fallback.
As currently implemented, the fallback is automatic if -S is specified.
I feel a bit ambivalent about this. One one hand, having a mode that
always works is an attractive idea, provided that the performance is
adequate. On the other hand, I really want people to either add the
one missing line into their MAC driver or ask me to do it for them.
I worry that we will never get good Tx time stamping coverage in the
kernel.
I didn't know that this was an easy thing for the driver to do, and
the error messages from current linuxptp didn't enlighten me. I
wonder if a better solution would be to change the warning to
something like "falling back to userspace timing because your network
driver lacks software timing support -- contact your driver's
maintainer".
Now that I know, I'll see if I can write a patch for sfc.
FWIW, there may be a bigger issue with software mode (kernel and
userspace) in general: rx wakeup latency. A couple weeks ago I did
some ping experiments, and I discovered that pings (using
SO_TIMESTAMPNS) were delayed by ~50µs if the core handling the irq was
idle. This may be enough of an asymmetry to matter.
--Andy
There is actually a similar issue with some ethernet hardware which has
EEE enabled. I know at least in the Intel igb driver we had to disable
EEE if ptp was enabled, otherwise sometimes you could start having
delays if there wasn't enough traffic to keep the part awake.

Mostly just adding this as an FYI for those who might be interested.

- Jak
Andy Lutomirski
2013-07-29 19:12:18 UTC
Permalink
On Mon, Jul 29, 2013 at 12:07 PM, Keller, Jacob E
Post by Keller, Jacob E
Post by Andy Lutomirski
On Sun, Jul 28, 2013 at 5:50 AM, Richard Cochran
Post by Richard Cochran
Post by Andy Lutomirski
The kernel supports software rx timestamping on all hardware, but most
drivers don't support any kind of kernel tx timestamping. Add a userspace
fallback.
As currently implemented, the fallback is automatic if -S is specified.
I feel a bit ambivalent about this. One one hand, having a mode that
always works is an attractive idea, provided that the performance is
adequate. On the other hand, I really want people to either add the
one missing line into their MAC driver or ask me to do it for them.
I worry that we will never get good Tx time stamping coverage in the
kernel.
I didn't know that this was an easy thing for the driver to do, and
the error messages from current linuxptp didn't enlighten me. I
wonder if a better solution would be to change the warning to
something like "falling back to userspace timing because your network
driver lacks software timing support -- contact your driver's
maintainer".
Now that I know, I'll see if I can write a patch for sfc.
FWIW, there may be a bigger issue with software mode (kernel and
userspace) in general: rx wakeup latency. A couple weeks ago I did
some ping experiments, and I discovered that pings (using
SO_TIMESTAMPNS) were delayed by ~50µs if the core handling the irq was
idle. This may be enough of an asymmetry to matter.
--Andy
There is actually a similar issue with some ethernet hardware which has
EEE enabled. I know at least in the Intel igb driver we had to disable
EEE if ptp was enabled, otherwise sometimes you could start having
delays if there wasn't enough traffic to keep the part awake.
If we wanted to be really fancy, there could be some kind of support
for waking up the Ethernet hardware and the CPU that will get the rx
irq a bit early. (This would probably be a PITA.)

--Andy
Keller, Jacob E
2013-07-29 21:04:54 UTC
Permalink
Post by Andy Lutomirski
On Mon, Jul 29, 2013 at 12:07 PM, Keller, Jacob E
Post by Keller, Jacob E
Post by Andy Lutomirski
On Sun, Jul 28, 2013 at 5:50 AM, Richard Cochran
Post by Richard Cochran
Post by Andy Lutomirski
The kernel supports software rx timestamping on all hardware, but most
drivers don't support any kind of kernel tx timestamping. Add a userspace
fallback.
As currently implemented, the fallback is automatic if -S is specified.
I feel a bit ambivalent about this. One one hand, having a mode that
always works is an attractive idea, provided that the performance is
adequate. On the other hand, I really want people to either add the
one missing line into their MAC driver or ask me to do it for them.
I worry that we will never get good Tx time stamping coverage in the
kernel.
I didn't know that this was an easy thing for the driver to do, and
the error messages from current linuxptp didn't enlighten me. I
wonder if a better solution would be to change the warning to
something like "falling back to userspace timing because your network
driver lacks software timing support -- contact your driver's
maintainer".
Now that I know, I'll see if I can write a patch for sfc.
FWIW, there may be a bigger issue with software mode (kernel and
userspace) in general: rx wakeup latency. A couple weeks ago I did
some ping experiments, and I discovered that pings (using
SO_TIMESTAMPNS) were delayed by ~50µs if the core handling the irq was
idle. This may be enough of an asymmetry to matter.
--Andy
There is actually a similar issue with some ethernet hardware which has
EEE enabled. I know at least in the Intel igb driver we had to disable
EEE if ptp was enabled, otherwise sometimes you could start having
delays if there wasn't enough traffic to keep the part awake.
If we wanted to be really fancy, there could be some kind of support
for waking up the Ethernet hardware and the CPU that will get the rx
irq a bit early. (This would probably be a PITA.)
--Andy
I don't think it would be worth the effort, plus I am not sure how it
would be done in a way to get accepted?
Loading...