ErrYaw increases after yaw spins leads to high speed runaway

Navio / Edge Flight stack
1

Good morning,
I’m hoping someone can help me along in some debugging of a crash we had the other day. There are two things I don’t understand. First, after a couple of 360deg yaw spins, the IMU and Mags get significantly out of sync and ErrYaw increases quickly. This led to a EKF error. That is not great but manageable. Second, after this occurred, the copter took off at high speeds and become unresponsive and uncontrollable. It crashed about 200 meters away while moving extremely fast.

This seems like a similar problem to:

but in that case the errors gradually correct themselves, whereas here it led to some extremely unpredictable behavior.

Setup:
DJI S1000, RPi3/Navio2, using built in magnetometers (both active), built in GPS
Wifi was active and connected to GCS, but all commands were issued only with the RC
Intwifi was disabled and generic usb wifi antenna was used instead
Barometer covered with some opaque opencell foam
20170323_emlid disk image and arducopter 3.4
Light winds: 1-2 m/s (from about 280-290 deg WNW)
Temp: 20C

Logfile:
2017-06-21 14-20-57.bin (1.4 MB)

Situation:
First, ignore the crash listed at line 9000. I was an idiot and accidentally put my knee on the end of the landing foot while installing the battery and sunk it into the soft ground. It stuck during takeoff and the copter did a graceful 90 deg tip over. Two dirty blades.

In short, I was doing a quick control demo using RC only. Took off to about 10 meters. Stayed in Loiter/simple flight mode the entire time. Little forward, little back, little side to side. Performed 90 deg yaw turn (line 31,000). Did two 360 yaw spins (line 33,000). Almost immediately after that, copter started rolling and traveling away from me while picking up speed. I appeared to have minor control (I could still somewhat affect the pitch so there is a curve to it’s path) but could not change the roll or bring it back to level flight (still in loiter mode). After a few seconds, I started quickly decreasing throttle, no effect.

Map:

20170621_crash_map.png741×504 528 KB

Obviously there is a pretty big disconnect between the GPS measurement and recorded position.

Examining the log shows that after the 90 yaw, the errors increase moderately and then immediately after the two yaw spins, the ErrYaw started to increase quickly.

ErrYaw.jpg1920×504 53.8 KB

ErrRP.jpg1920×504 57.7 KB

Vibrations look fine:

Vibe.jpg1920×504 66.7 KB

The two compasses match well so I’m guessing that the ErrYaw is due to the IMU getting ‘dizzy’, separation doesn’t really occur until after the two yaw spins.

Mag_heading_df.jpg1920×626 73.5 KB

NKF4_SM.jpg1920×626 60.6 KB

Finally, I’m at a loss as to why it decided to leave the area at high speed or started ignoring incoming RC commands (most importantly the throttle). More than anything else, I would love some help in that area.
Thanks.

3

Hello bgantner.
This is a very good report, thanks.
I think that it was a GPS glitch. There is no 100% reliable protection against these which means the pilot should always be ready to take-over manual control.
I recommend you to always have a possibility to turn on stabilize mode. Usually I tune fifth channel on a copter and three-position stick on RC. One mode is always stabilize mode.
You should try PosHold mode, I think that it will be better.

It was failsafe for EKF-check error.

4

Thanks for the help! I had the channel five set to switch to stabilize mode, I just didn’t react fast enough and didn’t realize that once the EKF triggered, it would ignore throttle (thanks for pointing that out). Couple of follow up questions if you don’t mind.

  1. Is your GPS glitch opinion based on something in the flight log or just on the actions/results described? I’m still learning the subtle details of ardupilot and would love to know what you use to come to this conclusion. I look at the number of satellites and HDOP in the data and there is no indication of how/when something like this would occur. Is that normal? Is it really just that random?

1a) I’ll also add that to my eye and from memory, the GPS track looks correct on the map overlay of the path it actually flew. It’s the Pos that seems out of place. Thus, even if the GPS sent 1-2 bad readings, it seems to have quickly returned to accurate measurements. This implies to me that something else was going on, maybe triggered by the GPS but that didn’t fix itself when the GPS returned to normal. Yes/No?

  1. Is the increasing ErrYaw and the GPS glitch/runaway overlap just random chance? Or is the disconnect of the IMU and Mags make it more likely that a faulty GPS measurement can cause havoc?

  2. I have a Reach on the way and will add it into the system shortly. Will the second GPS measurement make this more or less likely to occur? That is, I’ll be doubling the number of systems and thus doubling the odds of a faulty measurement coming in. However, it’ll be basically 100% likely that ONE of the two will be accurate. In short, I’m asking what happens in a similar situation if the regular GPS and Reach GPS wildly disagree for a moment?

Thanks again!

5

My answer was not quite correct. GPS signal was good. Your situation looks like a GPS Failsafe.
If you want more information, better ask these kind of questions on the ArduPilot forum.
When a copter has two GNSS then the copter uses GNSS which is “bigger fix” value (true for Reach, so when RTK fix is acquired, the Reach is chosen). ArduCopter 3.5 (the release will be pretty soon) will drastically change everything for the better as it does exactly what’s expected. Namely all data is filtered and the best data will be used.

6

Thanks for the help. I’ll follow your advice and move to the ardupilot forum.