M+ : issues with RTCM correction stream

Hi,
I’m having a few different issues with an M+ (fw 2.22.5) and a public RTCM 3.1 stream ( [http://142.41.245.88:2103]).

The final goal is to stream corrections over an SX1278 LoRa link.

Issue 1 : as a sanity test, I connect the M+ to a PC via USB, then “nc 142.41.245.88 2103 > /dev/ttyACM0” and configure the M+ accordingly. This mostly works, but on the Status page, I notice often the “Age of differential” increases to 3-4 seconds, even though the server is sending out 1Hz updates. It’s as if some corrections are dropped sometimes ? Is this expected ?

Issue 2: I’d like to use my modems at 4800bps, but reachview only allows >= 9600. Not a big problem, I can live without this.

issue 3 : this time connecting my lora modems. Sanity checked by configuring the M+ to send NMEA position updates: that works.
Also sanity checked by doing a loopback test at the M+ end : also works but not tested at full throughput.

In the other direction (simply piping that public RTCM stream to the modem), I can see RX activity at the M+ end, but the corrections probably get corrupted or truncated by my modems. That’s my fault, using 3rd-party modems, but I’m having difficulties troubleshooting this. ReachView doesn’t have any way of showing raw serial traffic.

I tried to SSH into the reach and after a bit of research, I discovered I can do basically nothing without root access. I also just learned root access is no longer allowed, which is very frustrating.

Any suggestions to troubleshoot this would be appreciated.

@ emlid , I understand your reasons for disabling root, but in this case it would’ve been helpful (I probably would’ve found the problem already and not be asking here).
Or at least providing a root-enabled dev version that comes with a big disclaimer about “unsupported, experimental, etc etc.”…

I hate the thought of having to jailbreak a linux-based device that I paid for, and that used to have root access.

Hi Chris,

Could you please share the raw data, solution and RTCM3 correction logs? That will help us to troubleshoot the issue with the rising age of differential.

Could you please also specify why do you need the 3rd party LoRa modems? As you’ve mentioned, it is a bit tricky for us to troubleshoot the equipment which we’ve never tested. Plus, various LoRa modules have different configuration protocols, that is why it could be possible why there are no readings observed.

I’m afraid we can’t provide the access to the root since it will be way more complicated to troubleshoot any issues after the changes into the firmware.

Hi Artem, thanks for the reply.

Yes, I can log the data you mentioned and will try to post here later today.

why 3rd party LoRa : because I wanted a 433MHz 1W link, and I find >200$ is a bit expensive for a pair of radios, even if they are nicely integrated and packaged.

no root access : yes, I understand your point and I disagree. But I don’t want to argue about that here, I think we have one interesting problem (age of differential) we can work on !
For testing I will use direct USB or serial connections to eliminate the radios from the equation.

Here is an example log. At least 2-3 times the “age of differential” increased to 6 and even 8 seconds .

base_202007040020_RTCM3.log (15.8 KB)

The setup was : PC => USB => Reach, no modem. Running on PC:
“nc 142.41.245.88 2103 > /dev/ttyACM0”

(note, there was some garbage data at the very beginning from a dumb test I ran just before)

I would prefer not posting the positions and raw obs here but can email them if required.

Hi Chris,

Could you please PM me the position and raw logs? You can do that by clicking my avatar and selecting “Message” button.

It would be also great if you’d provided me with hardware setup photos.

Sure, sending PM.
Hardware setup is simple : only USB and antenna connections.

Hi Chris,

I can confirm there’s an intermittent connection between base and rover because of the radio link. This can be seen in RTCM3 log. This is why the age of differential is rising. At this stage, I’d recommend configuring both radios with the PC and confirm the link is working stably.

I’m not sure what you mean ? as I said, those logs were with no radios connected, just a straight USB connection from the PC to the M+, and piping the RTCM3 log over USB.
The radio on that photo is completely disconnected and not even powered on.

Hi Chris,

Excuse me for missing that you don’t use the LoRa radio in this setup. However, there’s still an intermittent in corrections stream. Due to this, the age of differential may rise. For instance, here is the part of the RTCM3 correction stream with constant reception:


I was not able to visualize your log, which means that information could be corrupted:

If opening the RTCM3 log you’ve sent me in a text editor, some notable interruptions are noticeable too. That might happen if there is a low quality of the stream from the NTRIP caster or the connection via the USB cable is not stable. To locate the issue I’d recommend checking the USB connection first. You can use another USB cable or configure the correction input over TCP. With that, we will be able to see whether it is a connectivity issue.

UPD: I’ve edited my initial post with a little bit more screenshots.

Hi Artem,
thanks for checking the logs.
I just quickly tried 2 things :

  • correction input straight from NTRIP caster : no dropouts, age-of-diff always stays below 1.2sec.
  • tried a different USB cable : same issue. looks like as soon as I’m piping to /dev/ttyACM0, something happens to the stream ? some buffering issue maybe ? Not sure how to reproduce this.
    Have you tried connecting to the same NTRIP caster on one of your test devices ? the baseline would be ridiculous, probably halfway around the world, but it might be possible to log the corrections anyway and see if you can reproduce the problem ?corruption…

Unfortunately I’m running out of time and won’t be able to do more tests for the next few weeks. I’ll still follow here and will report back when I work on this again !

Chris

Hi Chris,

Yes, I’ve tried connecting to this NTRIP caster, but it didn’t work somehow. I suppose it does not support VRS, as the baseline is definitely halfway around the world as you’ve mentioned.

We would really want to troubleshoot this issue. However, the log you’ve shared is relatively short, and the satellite vision is not really smooth as you conduct your tests inside the room, as far as I understand.

For more extensive troubleshooting, could you please record a 30 minute RTCM3 correction log with an open sky view providing good satellite signal, so that we could have an in-depth look at what is happening there? Could you please also specify, which RTCM3 messages does the M+ receive? For that, you can share a screenshot with the correction input tab.

The raw data and solution logs with the Full System Report attached will be of help too.

Please, send all the data to my PM as there might be sensitive information included.

does not support VRS

Correct. I just use the stream as-is ( < 50km baseline ). Would it still be relevant to look at the corrections log without any Reach observations ? Just in case I saved 30 minutes of rtcm3 corrections to the attached raw file.
rtcm3_2103_20200714.bin (577.9 KB)

I forgot to check which messages are decoded, and I don’t have access to my hardware until the end of july. At that time I will do a better 30-min log + obs as you asked, and report back.

Thanks !

Ok, I think I found the problem, I’m 90% sure it has to do with linux’s medieval serial port handling. I’ve worked with serial ports a lot, I should’ve known better.

I troubleshooted this with the help of a hex editor and a basic RTCM3 parser (GitHub - jcmb/RTCM3: System for Parseing RTCM3.x packets) . I noticed a lot of packets were “undecoded” . I ran the tool with this command:

python2 ./RTCM3_Decode.py -L 4 -U -D < base_202007040020_RTCM3_trim.log 2>&1 | less

In a hex editor, I compared the two first messages of type 1004 (frames #0 and #2 in the log), chunk by chunk. I manually compared the similar chunks, since consecutive messages of the same type will be very similar, and added a space when the data was unsynchronized . Here’s the result :

format : 
[frame 0 chunk]
[frame 2 chunk]

D3 00 A5 3E C0 00 7B ED 47 E2 A0 49 77 1A 0D 0A 03 C2 7F D0 F1 C0 30 C0 DA
D3 00 A5 3E C0 00 7B ED 57 82 A0 49 77 15 90    04 09 7F D0 F1 C0 33 40 E3

27 FE B3 A6 76 21 9F F7 18 AA A7 14 02 61 F8 7F 1F E0 8D 71 F4 A3 01 63 07
0B FE B3 A6 7E 8F 3F F4 C0 AA A7 16 02 31 F8 33 1F E0 0D 72 23 53 01 46 9F

F4 7B CC 0C    E0 3E AE FF 20 C0 B4 A1 08 01 04 3F A6 51 00 A5 81 DF E7 FA
F4 7B CC 0D 0A E0 3B 22 FF 1C C0 B2 99 54 02 75 FF A6 51 80 B9 02 0E 2F FA

A5 13 8E C2 E0 50 5D FD 1E E3 02 A4 0D 10 FF C8 36 60 CB 0C FF 5D 43 0D 0A 
B5 13 83 90 60 4D 75 FD 1E E3 02 9C 0C B4 7F C7 B6 60 9D B8 FF 7C A3 2A   

30 80 2E 40 0D 0A AD FE 34 F6 DC 30 68 1B 9E FF 42 C7 00 C9 03 C0 9F F9 65 
30 A0 2E 60 0E    99 FE 24 F6 DD 28 80 1B E8 FF 42 C8 00 D1 03 C9 EF F9 65

2E ED 54 C0 01 AB AA 8C 68 0B    A0 00 7F F5 9F 14 60 A7 10 01 F9 1F D4 65
2E D1 04 7F F2 FB AA 8C 70 0D 0A AF FE B4 F5 9E 14 61 91 9A 02 2D 9F D4 64

40 3D C0 50 47 FC 6A A8 54 E8 B0 04 2F FE 99 65 81 42 01 47 9F E0 40 23 30 8D
C0 3C 40 56 DF FC 62 A8 4F 9A D0 07 F7 FE 99 65 81 6C 01 C0 DF E0 00 FF A0 BE

And, it really looks like some spurious 0x0A (LF) were added. And I know very well that the default serial port options have all kinds of silly text-based rules that mangle raw data exactly like this.

I won’t be able to try for a while, but I think the solution will to be to run a “setserial /dev/ttyACM0 …” command to set the port to raw, unprocessed data.

I’m sorry I didn’t think of checking this before - I was using a terminal program for testing (that configures the port properly) and I assumed the device would stay in the correct mode.

I’m confident this will solve at least part of the problem.

Hi Chris,

Thank you for the idea of possible solution. We’ll try to reproduce the issue, but it would be great if you could share your progress once you will come back to the gear.

Hi Chris,

We’ve tried to reproduce the issue on Ubuntu 16.04, but we were not able to experience the same behaviour. For example, here is the 24h log via the USB pipe:


It is noticeable that there are no random interruptions in the stream except for the changes in the observable satellites.

Could you please test the same setup on another PC once you will be able to get access to the hardware? That could help us to localise the problem.

Interesting ! Yes, l will definitely try on a different computer when I come back (7-8 days).
On your test machine, what is the output of
stty -F /dev/ttyACM0 -a
?

Hi Chris,

Here’s the output:

  speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>;
eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; discard = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff
-iuclc -ixany -imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt
echoctl echoke -flusho -extproc

Thanks Artem.
On that last screenshot you posted (base log of 2020-07-25), what broadcaster was that ?

My /dev/ttyACM0 has the same stty output as the one you posted. Are you also using netcat (nc) to stream the RTCM data to the USB pipe ?

I’ve just got back and can still reproduce the issue on 2 different machines (both on Arch linux), but running

  stty -F /dev/ttyACM0 raw

before piping the stream solves it (age-of-diff remains below 1.2sec at all times instead of frequent peaks >3 sec.)

(unrelated side-note : should I be able to connect to the Reach’s webserver via the USB-CDC_ether connection ? It would be convenient for testing, but I haven’t been able to make this work )

Hi Chris,

Thanks for your patience!

I will answer your questions below:

  • On that last screenshot you posted (base log of 2020-07-25), what broadcaster was that ?

We were using the same caster as you did - 142.41.245.88:2103.

  • Are you also using netcat (nc) to stream the RTCM data to the USB pipe?

Yes, we used the nc for the data streaming over the USB port.

Using the stty -F /dev/ttyACM0 raw enables the processing of the raw data, e.g. there is no additional processing on the PC itself. It may affect the piping process overall, but it should not work in a way that the stream is only affected at a certain stage, e.g. initialization.

  • Should I be able to connect to the Reach’s webserver via the USB-CDC_ether connection?

Regarding the using USB connection to access the web interface of Reach, I’d recommend checking this thread: Can't connect to new RS2 - #3 by bide

This topic was automatically closed 100 days after the last reply. New replies are no longer allowed.