Re: gpsd time service: PPS-synchronised SHM samples rare, both sources gradually less and less frequent

[Marek Szuba]

On 2020-09-17 18:00, gpsd-users-request@nongnu.org wrote:

> GLONASS has the worst timing of the big-4.

I didn't know that. Good to learn something new!


> Galileo has the worst outage record.

Not really surprising given that they are the youngest of the Big 4 and
aren't even at full operational capacity yet. I'm mostly interested in
it for testing purposes anyway, my list of active satellites on the
stationary receiver consists on average of 2/3 GPS and 1/3 GLONASS
regardless of whether the third enabled constellation is BeiDou,
Galileo, or neither.

> But most will never notice any difference.

Good!

> That is telling you about a known bug in the pps-gpio driver.  It
> only passes on edge, so clients can not auto-detect the proper
> PPS edge.
> 
> If you pick the wrong edge, you are off by the pulse width (often 100 ms),
> not by a whole second. 

Got it. Not sure where I've got the 1 second from actually, seeing as I
did see somewhere that PPS pulses from the NEO-M8N are indeed 100 ms
wide by default.


>> So I did pass capture_clear to pps-gpio.
> 
> WHich may, or may not be the correct edge.

I am afraid you have misunderstood me. The pps-gpio option which changes
asserts from the rising to the falling edge is assert-falling-edge and I
haven't touched that.

>> and after enabling capture_clear I began seeing non-zero values for
>> clear as well.
> 
> How did you do that?

This is what the capture_clear option of the pps-gpio DT overlay does!
Mind you, it doesn't always seem to work correctly - if for instance I
enable this on my Raspberry Pi 3 B+, clear values look okay at first but
stop updating after a while (i.e. when I run ppstest, they are still
non-zero but the same in every line).

> The "precision" is a joke.  It is just a constant someone pulled out of
> air air.  Itgnore it.

OK.

>> That's the weird thing, last time I checked ppstest looked okay (i.e.
>> a burst of quite a number of lines, haven't counted how many, every
>> second) both at the beginning of a run and when ntpd had long stopped
>> receiving samples from gpsd.
> 
> 
> Well, that would be a harware or kernel problem.  Hard to say since it
> is gone.

I wish it were that simple... Unfortunately what I meant by "ppstest
looked okay" was that ppstest produced the expected output regardless of
how gpsd was doing. Specifically, I have checked this:
 - right after starting gpsd, when ntpshmmon showed NTP0 samples being
produced roughly every second but NTP1 hardly ever showed up;
 - much later, when ntpshmmon showed ZERO activity; and
 - having shut down gpsd with the receiver still powered and maintaining
the fix (confirmed by looking directly at NMEA messages arriving at the
serial port).
In each case 'ppstest /dev/pps0' showed expected activity.


Anyway, I have decided to retry the whole tests under a more controlled
environment:

1. Receiver - factory reset, enable Galileo, save configuration;

2. System - Raspberry Pi 3 B+ with a freshly installed copy of Raspbian,
pps-gpio enabled (dtoverlay=pps-gpio, _without_ capture_clear for the
reasons mentioned above) and Bluetooth disabled (dtoverlay=disable-bt)
so that the GPIO serial port uses real UART;

3. gpsd - 3.21, compiled from source on the target system with the
following options: timeservice=yes nmea0183=yes ublox=yes python=yes
shm_export=yes socket_export=yes systemd=yes gpsd_user=gpsd
gpsd_group=dialout. When launched by systemd, the exact command line is
"/usr/local/sbin/gpsd -n /dev/ttyAMA0 /dev/pps0"

4. chrony (I stopped using ntp on production systems so long ago that I
feel I have got more control over the process if I stick with chrony for
these tests) with two SHM refclocks defined word by word exactly as
suggested in the "Feeding chrony from GPSD" section of the time-service
HOWTO, and the logs "refclocks" and "statistics" enabled.


I then cold-restarted the GNSS receiver, rebooted the Pi for good
measure, waited for the time-pulse LED on the receiver to start blinking
so that it has already established a fix, and launched chronyd and gpsd.


INITIAL OUTPUT

1. cgps: shows a 3D fix with 13/30 satellites, reported UTC time is
accurate. The initial set of messages (i.e. before "TPV" start showing up is

{"class":"VERSION","release":"3.21","rev":"3.21","proto_major":3,"proto_minor":14}
{"class":"DEVICES","devices":[{"class":"DEVICE","path":"/dev/ttyAMA0","driver":"u-blox","subtype":"SW
EXT CORE 3.01 (
16559b),HW 00080000","subtype1":"ROM BASE 2.01 (75331),FWVER=ADR
4.00,PROTVER=19.00,MOD=NEO-M8L-0,FIS=0xEF4015 (10011
1),GPS;GLO;GAL;BDS,SBAS;IMES;QZSS","activated":"2020-09-17T20:36:42.859Z","flags":1,"native":1,"bps":9600,"parity":"N","stopbits":1,"cycle":1.00,"mincycle":0.02},{"class":"DEVICE","path":"/dev/pps0","driver":"PPS","activated":"2020-09
-17T20:33:25.161Z"}]}
{"class":"WATCH","enable":true,"json":true,"nmea":false,"raw":0,"scaled":false,"timing":false,"split24":false,"pps":f
alse}

OK, so it turns out that in spite of what it says on the box my receiver
is actually a NEO-M8L rather than M8N. Funny, I could have sworn that it
said M8N here too before the factory reset... I don't think this will be
a problem though, 'ubxtool -P 19.00 -p MODEL' reports the dynamic model
as 0 (which IIRC is "portable", i.e. not automotive) and if I understand
the documentation correctly M8L without automotive sensors connected to
it will act like a M8N.

2. ppstest /dev/pps0: produces bursts of about 2,000 messages every second.

3. ntpshmmon: left it running for about 5 minutes. During that time NTP0
samples showed up about once per second but *no* NTP1 (or any other
non-NTP0) samples arrived. I later confirmed the latter with 'chronyc
sources', where the 'PPS' refclock showed "LastRx -".


I then logged off and just let the whole set-up running.


AFTER AROUND 2.5 HOURS

1. 'chronyc sources' shows no data AT ALL has been received from the
'PPS' refclock. Moreover, the last time the 'GPS' clock sent anything
was just under two hours earlier. The refclocks log confirms this - no
PPS lines, and GPS lines ceased abruptly about 40 minutes after start-up.

2. ntpshmmon shows no activity whatsoever, which is unsurprising given
the above

3. cgps: shows "NO FIX", no TPV messages arrive.



4. 'ppstest /dev/pps0' continues to work fine.

5. The receiver's time-pulse LED continues to blink and connecting
directly to the serial port (at 9600 bps) shows incoming NMEA messages.


I then restarted both gpsd and (just in case) chronyd, repeated the
steps from "initial setup" to confirm that everything appears to be in
order again.


AFTER FURTHER 8 HOURS

Exactly the same as the previous time except for how long 'GPS' refclock
worked - this time it was only just over 20 minutes.

Also, this time I had a closer look at the initial messages produced by
cgps. Those were:

{"class":"VERSION","release":"3.21","rev":"3.21","proto_major":3,"proto_minor":14}
{"class":"DEVICES","devices":[{"class":"DEVICE","path":"/dev/ttyAMA0","driver":"u-blox","subtype":"Unknown","activated":"2020-09-17T23:56:09.855Z","flags":1,"native":1,"bps":4800,"parity":"N","stopbits":1,"cycle":1.00,"mincycle":0.02},{"
class":"DEVICE","path":"/dev/pps0","driver":"PPS","activated":"2020-09-17T23:55:59.649Z"}]}
{"class":"WATCH","enable":true,"json":true,"nmea":false,"raw":0,"scaled":false,"timing":false,"split24":false,"pps":false}

Notice how 'bps' now reports 4,800 instead of the expected 9,600? Which
gave me an idea for something to try - instead of restarting anything, I
simply ran

stty -F /dev/ttyAMA0 9600

on another console. Et voila, cgps immediately began reporting having a
3D fix and NTP0 samples reappeared in ntpshmmon output.


AFTER FURTHER 4 HOURS

Same. This time NTP0 samples ceased after just under 40 minutes. I have
since just let the set-up untouched to see if anything changes on its
own, no luck so far. And all this time, 'ppstest /dev/pps0' continues to
work.

-- 
MS