Reach and Pixhawk problem: momentary communication failure makes the aircraft fly-away

Hey guys, how is it going?

I have been testing Reach in RTK mode in my quadcopter.

My base setup:
Base Reach connected through Ethernet over USB to a notebook with Mission Planner feeding corrections through a HK radio connected by COM port.

My rover setup:
Pixhawk connected to Reach in the GPS serial port. Reach was configured as main GPS, and I am not using a second one. Antenna is 25cm higher then other circuitry and with a copper ground plane.

The problem:
At random moments, the communication between Pixhawk and Reach (that’s a guess) seems to fail and this happens:
-NSat goes to 0;
-HDop goes to 99,99;
-LAT and LONG goes to 0.

So the aircraft flies away. After a short while, the GPS behaves like nothing happened, and I didn’t even restart the Pixhawk, just kept flying.

It happened before, when I used used Reach as a Single GPS.

The flight test was a series of take offs and landings to test accuracy, and you can see the error happening between the 10th and 11th landing.

I reviewed Reach’s log and it doesn’t present the error. That is what makes me think it’s a comm problem, but I’m not experienced in troubleshooting this things.

Can anyone help me with that?

Logs for Pixhawk and Reach:
Pixhawk log
Base ref
Rover ref

@Enio_Freitas We are looking into this, but need some time for analysis. Thank you for a detailed report!

Nice, if I can help in anything just let me know. Thank you for the effort!

BTW, just did a precision landing with this same configuration. The error reported here seems to be random and did not occur at the time. Got fairly nice results measuring landing vs take-off position difference:

Mean diference: 25cm
Standard deviation: 5cm

It will probably not help the error analysis, but at least for sake of future reference:

I did testings using dual GPS, with Reach as 1st GPS and 3DR as 2nd GPS. The error above repeated itself. At points similar to the blue arrows in the picture below, where NSat droped for a short time, the Pixhawk accuratelly changed for the 3DR GPS, and fly-away did not happen as before (although sudden position changes did, as expected).

If it helps, I can send the data from the new tests where the error repeated itself.

We have spent some time trying to reproduce the error without luck. In the log GPS status goes all the way to 0, which means that there was no data coming at all from the Reach, not just a loss of fix. Maybe the problem actually is electrical? Maybe you can try to systemize what flow of events leads to the error?

Hi Igor,

I did change the port used in the Pixhawk (from GPS to Serial 4/5), but the error does not vanish. It does not seem to be an eelctrical connection problem, but I’m changing the connectors for new ones anyway.

Also, as the Reach module is powered through the serial conn itself, if it had a bad connection it would probably take a long time to restart and get a lock, like when reach is first powered up. It would not explain these quick errors.

There is nothing specifically leading to the error, it seems to be random. No flight mode change, no maneuvers… There are even flights when the error doesn’t happen. I did not correlate the error with anything yet.

One thing I did not do was swaping both base and rover units. I’ll do that and put new conns, and reply here asap.

Hi!

What version of ReachView are you using?

It’s v0.4.9

I did new tests:

-Swapped rover/base Reach units;
-Put new connectors.

Did a simple loiter test. Same errors… At least with the second GPS it did not fly away, it simply corrected it’s position agressively.

I am using Ublox config file as GPS_GLONASS_5Hz, I guess there’s no problem with this, right?

I had the same issue. When lose telemetry and correction link, the plane just jumped 5 meters out of the flying line.

@arnaldoasneto
Is it during a switch between GPS 1 and 2? You can see which GPS is in use by the parametes “U” in Pixhawks’s GPS telemetry logs.

Is there any way we could work on the diagnosys of this problem? We could perhaps do something to monitor the serial comm between Reach and Pixhawk, any suggestion?

I had the exact same problem…fought it for nearly a week every day before giving up and pulling Reach off hoping that with time an update would fix it. I seemed to have some luck with getting rid of SBAS, but then it wasn’t very accurate. I’m really glad to see the data captured and time taking to make the post. I hope something is figured out soon as I would really like to get it going again!

hey,
post your logs (from navio) to help resolve that problem!

All my testing was done about 2 months back. I’ll need to get it setup and try again. I’ll try to do so in the next few days. I would tell you though what’s described above is EXACTLY the same behavior I had. The only difference in setup is I had the 3dr GPS as Primary and Reach as secondary. I did though try making REACH the only GPS, but that didn’t fix it.

it’s very hard to find the reason without logs; behaviour like “fly away” can have a lot of reasons; what’s your reach firmware? try to find those logs and post them…

I can share my experiences with using reach on trekker.

When loosing connection to base , solution goes to float then to single.

And sometimes it goes to no solution output.
Going to single or nothing makes my steering software to go crazy… and trekker drives away…

It did use a higher value (300 sec) for losing correction like read in rtklibexplorers blog.

If someone could provide raw data and solution log where this issue can be seen that would really help us with debug.

Hi! @igor.vereninov , is the data I sent inicially not complete enough? Tell me how can I help, and I’ll try getting new data (probably won’t be able until feb/2017 but I will do my best!)

@ngjones, it would probably help a lot if you could log everything up and send as well, so we have more than one case to study!

Thanks!

Hi @Enio_Freitas, we did actually spend quite a bit of time on this issue and performed test flights to try and reproduce it. Everything performed nicely after we disconnected corrections with gradual precision reduction.

We will keep performing this sort of test every time we go our flying and if there is any update write here.

Sorry about the trouble. Maybe you can give our new beta a shot in Feb and let us know if the problem is still there for you?