Realy bad / unhealthy GPS in ardupilot

Ardupilot : V4.0.3
Image : the latest (did an update last week)
Problem : The GPS is detected in Ardupuilot, the position is right (approx 2-5m of random slow deviation) while on ground with 10 sats. But there is a “unhealthy GPS” message in ardupilot and, while flying, LOITER and POSHOLD go everywhere.

Hi, I’ve already made posts on the ardupilot forum about my build and another issue.

This last issue was in part due to the GPS and was mainly due to the Extended Kalman Filter. The GPS vertical velocity was wrong, so I disactivated this measurement in the EKF to make AltHold work. I’ve been struggling with the GPS for quite a while without noticing it was the GPS.

Now the GPS is still wrong and I can’t use it in any of Ardupilot modes. It seems to come from some Navio2, ardupilot and u-center’s settings.

Previous thread

I’m aware of this one. But no clear solution came out of it. Plus, I don’t have the “baudrate” messages in mission planner.
It is said that it was a bug from Ardu 4.0.3 as stated in the changelog, but nothing is written about GPSs in said changelog. Also, it has been 4 months since and there is no new update of Ardupilot for Navio if all navio’s GPS stopped working the update would be here.

On ground behavior

While disarmed, with any number of satellites, mission planner displays “unhealth GPS signal”. The GPS is “enabled” and “present” but keeps switching from “Ok” to “bad” in mission planner when opening the CtrL+F menu. The position of the copter on the map is not perfect, it slowly drifts in a 3-5m radius.

Setup

I’m pretty sure it does not come from interferences. When powering the board with an external battery the error message is also here. Plus this is my setup:

the GPS (and its cable) is far from the powerlines, above everything and the esc is really far (beside the white tape at the far right)

Analyzing logs

I’ve tried Loiter and PosHold, with 2D gps velocity and without. Neither of those work.
Here is a video of a “good” flight (at least it’s not going in a straight line) : https://youtu.be/mwm1F3GWNhQ

here are several logs with different settings : https://drive.google.com/drive/folders/1BbkUHXdomawUMbq8Sp3r7iXRVKOQ1Day?usp=sharing
number 1 is the video.

Number 0 is with default settings. So let us look at this one. The main problem is that
NKF1/VN (speed north) and the slope of NKF1/PN (position north) are vastly different. Something is throwing the EKF (kalman filter) off and I strongly think it’s the GPS (I have very low vibrations BTW).

The Loiter algorithm works by controlling NKF1/VN (speed north) depending on the desired NKF1/PN (position north) value. If the copter is 1m to the north, a -1m/s is sent to the north speed controller. If speed and position are not linked the control does not work and that’s what’s happening here.

(TPX is the desired position and TVX desired speed).

This discrepancy between speed and position happens in all 3 dimensions when using GPS.

There is a huge amount of innovation in the kalman filter’s (EKF) position and speed:

In the end this either come from a bad EKF or a bad GPS. Because we’ve an error on the GPS and disabling it on the height already solved a problem I think it comes from the GPS.

Refresh Rate

A previous discussion that I cannot find, found that the rate of the GPS did not correspond to what Ardupilot wants. Ardupilot wants at least 5Hz. But it happens that the GPS is too slow as on this log:

This was because too many constellations were used at once. Navio’s GPS can achieve 5Hz with only one or two, but if you enable everything it becomes 1Hz.

Now I’m only using GPS (no GLONASS, GALILEO, SBAS …) and it looks like 5Hz, though it’s hard to tell:

sometime the rate is slower, here between 03 and 04 there is only 3 samples, and 8 samples in the next second.

here it’s 3 sample/s, i.e. 3Hz:

The sample rate could be the reason behind the unhealthy GPS. But I’m not sure.

This could explain why this keeps changing in the CtrL+F menu, when the rate is too slow it switches to “bad”:
image

image

Also I don’t know how to properly setup u-center. I’ve disabled Ardupilot Auto Config for the gps, so I can manually set it up in u-center at 5Hz, but I’m not 100% sure of how to use u-center. The logs 1-2-3-4 where with my settings inside u-center, or I think so. Log 0 was with Ardupilot auto config. Either way it doesn’t solve the problem

Conclusion

I don’t know where exactly is the problem (it could be the rate but I’m not sure), I have no idea of what to try next to find the issue and fix it.

I would gladly receive some help. If you want me to do some test flights with video, logs, and other settings I’ll gladly do so.

Edit : clarity and typos

Oops, this discussion should be in #navio:flightstack .

I also just found this thread that reported on the “maybe a rate problem” for the GPS : Navio2 with second GPS module

It’s at the end of it. I don’t know how he manage to quickly extract the rates from Ardupilot

Alright, I managed to fix the rate, but the gps is still very bad.

Rate fix:

Ardupilot will do the config so auto_config=1. GPS_GNSS_MODE should not use more than 1 constellation (otherwise you cannot reach 10Hz), in my case it is set to GALILEO so 4. GPS_RATE_MS=100 ms so you’ll run the GPS at 10Hz. I haven’t tested with SBAS yet

It is strange because the rate is awful at 5Hz (GPS_RATE_MS=200) but much better at 10Hz (GPS_RATE_MS=100)

10Hz:

5Hz:

It continues to not work tough

The speed and position continue to not match:

At this point I don’t know where it comes from. You’ll find today’s log here.

Cause is not easy to spot.

I had a look at old logs. Some are fine, other are disturbing.

On my test configuration, at home with a roof window, Rpi4, latest Emlid distribution with Arducopter 4.0.6, two M8N GPS config, GPS+GNSS+SBAS at 5Hz, GPA is correct.

and this older one (arducopter 3.6.5) had some variation but with good navigation result

Latest graph is for information only. There are two M9N units (GPS+GNSS+GALILEO+SBAS at 5 Hz) on another flight computer

RMK: M9N at 10 Hz limit the number of used satellites at 16. No limit at 5 Hz…

Indeed those GPA are great !

I absolutely don’t know what’s up with mine. Is the Emlid distribution with Arducopter 4.0.6 available somewhere ?

Because the latest image is ardu 4.0.3 on the website. Or did I missed something on how to update arducopter ?

Now I’m curious to see if this was indeed a 4.0.3 issue (as in one of the first post) or if it’s more of a me problem.

You have to update arducopter via ssh:
Commands are in bold
#comments#

pi@navio:~ $ cd /opt/ardupilot/navio2/arducopter-4.0/bin/
#to access the arducopter folder#

pi@navio:/opt/ardupilot/navio2/arducopter-4.0/bin $ sudo mv arducopter arducopter403old
#rename to have a backup if needed and free the name for the new arducopter file#

pi@navio:/opt/ardupilot/navio2/arducopter-4.0/bin $ sudo wget http://firmware.ardupilot.org/Copter/stable/navio2/arducopter
#to download the latest stable -4.0.7 now#

–2021-02-23 17:06:41-- http://firmware.ardupilot.org/Copter/stable/navio2/arducopter
Resolving firmware.ardupilot.org (firmware.ardupilot.org)… 159.65.186.182
Connecting to firmware.ardupilot.org (firmware.ardupilot.org)|159.65.186.182|:80… connected.
HTTP request sent, awaiting response… 301 Moved Permanently
Location: https://firmware.ardupilot.org/Copter/stable/navio2/arducopter [following]
–2021-02-23 17:06:41-- https://firmware.ardupilot.org/Copter/stable/navio2/arducopter
Connecting to firmware.ardupilot.org (firmware.ardupilot.org)|159.65.186.182|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 1989616 (1.9M)
Saving to: ‘arducopter’

arducopter 100%[===================>] 1.90M 984KB/s in 2.0s

2021-02-23 17:06:44 (984 KB/s) - ‘arducopter’ saved [1989616/1989616]

pi@navio:/opt/ardupilot/navio2/arducopter-4.0/bin $ sudo chmod +x arducopter
#to have arducopter executable#

pi@navio:/opt/ardupilot/navio2/arducopter-4.0/bin $ sudo cp arducopter arducopter407
#copy with another name so you have a backup if needed later#

pi@navio:/opt/ardupilot/navio2/arducopter-4.0/bin $ sudo reboot
#to relaunch with latest version#

Thanks I’ll try and report back.

Thanks a lot for the clear explanations, I did the update.

Here are some tests for the rates:

Works:
10Hz GALILEO
10Hz GALILEO SBAS
10 Hz GPS SBAS

Doesn’t work:

5Hz GALILEO
5Hz GALILEO SBAS
5Hz GPS SBAS
5Hz GPS SBAS GLONASS

So the weird thing with the rate is still here in 4.0.7 with the “unhealthy GPS signal” if the rate is at 5Hz.
So updating didn’t change this.

With 10Hz
The “EKF” is now red when disarmed (with GPS fix outside), I don’t recall having it red in 4.0.3
image
The sad thing is that my copter blew up … so I can’t conduct any flight test to check if the speed and position continue to not match. From the red EKF and similar drifting on the map as before I think it is still the case.
I tried running around with the board like a crazy person to check the gps in a log but ardupilot disarms after 5s so it’s useless.

I have a workaround for that. Power your Raspberry with an external phone power bank, Disable arming check (set to 0), Arm with RC and set power level to middle, Disarm at the end of the simulation.

I’m experiencing bad data flash log reading and download with Mission Planner. Qgroundcontrol Dataflash handling is fine… Another mystery.

There is problem somewhere with your GPS configuration. I will try some different configuration tomorrow and share settings and result

Here is a test today (GPS+GLONASS+SBAS at 5Hz on both Navio2 and External M8N)

My config and why I choose some settings

Capture d’écran 2021-02-25 à 11.19.06

I configure my internal Navio2 GPS with uCenter and network connection. I do not want arducopter change something (GPS_SAVE_CFG 0)

Capture d’écran 2021-02-25 à 09.41.48

I stop arducopter before connecting with uCenter

GPS+GLONASS+SBAS

Capture d’écran 2021-02-25 à 11.20.23

Pedestrian, valid to 30m/s, 108 Km/h or 70 mph. This mode offer best accuracy.

Capture d’écran 2021-02-25 à 11.20.48

Protocol UBX+NMEA (NMEA only is not working with Navio2, UBX only is fine but Satellite data is not displayed in uCenter)

Capture d’écran 2021-02-25 à 11.21.35

SBAS is set to EGNOS (Europe) for my use.

Capture d’écran 2021-02-25 à 11.22.14

Rate is set at 5Hz

Capture d’écran 2021-02-25 à 11.22.37

Do not forget to Send config (after each modification and at the end to be sure everything is configured).

Last log after config and reboot

1 Like

Thanks for your help, I’ll be commenting every step as I go, just to be sure:

Ardupilot config set to not touch anything on the GPS

Reboot, stop arducopter and I’m in u-center (usualy I used Mavlink Serial Passthrough in Ardupilot to connect u-center, but now I’m doing it like you said by turning off Ardupilot, it’s much less buggy)

GNSS is already just like yours, I didn’t change a thing

Nav5/Dynamic model was on 8 (airborn) as usualy advised by ardupilot. I switched it to 3 (pedestrian), I won’t be going fast anyway.

This was my PRT (seems empty):
image

Changed it to SPI:
image

SBAS was enabled, but PRN Codes were set to “Other”. Switched it to EGNOS because I’m also in Europe:
image

RATE was at 100ms (10Hz), switched it to 200:
image

Sent the config at the bottom right on every step, in the CFG panel at the end of everything, then double checked everything.
image

Reboot and let’s see if it works…

Sadly it’s the same as before, GPA.Delta is going everywhere with “Unhealth GPS Signal” (9 sats) and red EKF:


(12 sats on my roof now)

I’m going back to u-center to check if the changes are still here (I also changed GPS_RATE_MS to 200 and GPS_GNSS_MODE to GPS+SBAS+GLONASS in Ardupilot, didn’t do anything).

For some reason this changed, I’m changing it to SPI, the rest is ok:
image

Saving the config, reboot and tying again… Nope:

Going back to u-center … This bloody Port doesn’t want to change!
image

In message I tried enabling SPI (only once):
image

rebooting… and it changed back to the wrong thing
image

I’m disabling Ardupilot on launch to check if he is not changing this parameter on launch.
Nope, PRT keeps changing back to 1-UART whatever I do:
image

Before you ask, for sure I’m clicking on send at the bottom left while in “PRT” and then a second time in “CFG”.

I believe “poll” downloads the configuration from the GPS, when I “poll” after changing PRT/Target to “4 - SPI”, nothing happens, SPI is there just like I want it.
BUT after some time, if I close the “Configure” window, and re-open it, PRT/Target falls back on “1 - UART”, same if I restart u-center.
While in “1-UART” I tried changing protocol in/out, but this also falls back on ‘none’.

I don’t know what’s happening here and why PRT/Target doesn’t want to change.

Hello,

To clear any difference between configuration, I just redid mine.

UART1 is not used by Navio but when setting other GPS, other port adopt UART speed…

What I just did:

In CFG screen, I resetted config to default, Select item then click send (then select save current config item for later)

Capture d’écran 2021-02-25 à 16.33.20

Redo the other steps (GNSS, SBAS, Rates, …, MSG SPI On, Ports)

Set UART1 at 115200,

Save and reboot.

EKF status change when EKF2 IMU0 is using GPS (20" waiting time, could be more)

I did another log.

My Y6 testbed

2 Likes

I’ve reset u-center, and re-did the steps (GNSS, SBAS, Rates, …, MSG SPI On, Ports). Well, now Ardupilot doesn’t seem to understand what’s happening on the GPS port. I’ve set Port to UART1 at 115200, not SPI ('cause it always falls back to UART)

Set GPS_AUTO_CONFIG to 1

Capture d’écran 2021-02-25 à 17.41.02

Do not ask why, I do not know… but 1 is the good setting.

SPI window: default screen is UART1, SPI is set…

YEESSS !

I don’t know either… but now it’s stable

EKF is red (15 sats) but, I’ll do some running around my garden in a few day when I’ll be back home. I’ve good hope.

Another source of problem. You need a tight fit of the GPS plug.

Or expect such result:

Yep, that’s a good advice I’ve seen another thread where this was the cause. Anyway, now everything works fine, thank you very much !

I’ve done a quick run with your technic

The rate is perfect:

And the measurements look good:


They are small discrepancies between East speed and Longitude, but it’s almost nothing. It most likely comes from my shity setup that had a bad Yaw estimate so the EKF was confused. The altitude that doesn’t need the yaw is very good.

Merci de votre aide, fellow French engineer (one of your screen-capture had a tidbit of French in it).

1 Like

Ce sera un plaisir de passer dans ta vallée vosgienne lors d’une visite à Strasbourg.

1 Like

Je n’y serais bientôt plus, mais le coin est très sympa ; )