EKF issues causing crash/flyaway

(Erroneous Mavlink Messages Cause Crash/Flyaway - Discussions - diydrones).

All,

Had a flight yesterday with virtually no changes from the previous day (with successful flight) on the same flying field here in San Diego. Notably this time I started receiving erroneous data on my OSD (mavlink messages); it was clearly causing some control issues. I’m wondering if anyone has experienced this? I’d really like to not hit the ground going 17mph again :).
Most of the reason for the crash is my fault in that I was flying LOS and didn’t look at the FPV monitor and notice the bad data coming down until watching the video later. I could definitely tell something was wrong while maneuvering around. Here’s the video (having a DVR is an AMAZING troubleshooting tool). I’ve also included the log file, showing that at one point I try to activate an RTL, which essentially triggers a flyaway situation, at which point I switch back to STAB and try to recover. Then erroneous data comes down (clearly visible 10000ft/min VS!! and crazy horizon line) and its too hard to recover, so I set it down. Lucky I was flying in a big empty field!
I’ve also got the logs. I did an analysis but not really sure what to look for. I’m thinking the Kalmann filter went nuts… any help? Anyone seen this kind of behavior before? Here’s my build info:
Frame: QAV500 Frame w/540mm arms
Props: 12x5.5 Carbon Props
Motors: Tiger Motor MN3110-17 (700kv), 4mm shaft
Autopilot: Raspberry Pi 2 w/ Navio+ hat
ArduCopter Version: Copter V3.3
External Compass: HMC5883L Triple-axis Magnetometer (Compass) Board - HMC5883L : ID 1746 : $9.95 : Adafruit Industries, Unique & fun DIY electronics and kits
GPS: Ublox M8N + Garmin GA25 MCX GPS Antenna
Battery: Turnigy MultiStar 4S 14.8V 16Ah
Internet Cam: Raspberry Pi Cam
FPV Cam: 1/3 Inch SONY CCD Board Camera 700TVL
ESCs: Emax 4 in 1 30A w/SimonK Firmware
Telemetry: RFD900+ Long Range Radio Modem
FPV Tx/Rx: Boscam Thunderbolt 2000mW 5.8GHz
FPV Antenna: Boscam 5.8GHz Cloud Spirit Circular Polarized
RC Tx: FrSky Taranis 9XD
RC Rx: Dragon Link V2, UHF

Wanted to repost here in case the developers may understand the solution. [Original post is in DIY Drones]

Jeremy,

You have huge variations in baro readings. Do you have it covered as per our docs?

1 Like

Igor,

Here’s my latest experimentation. It was suggested that I try to do a compass motor calibration, which would be my next course of action. I think between that and locating the external compass even higher away from the motors is necessary to fix the problem. Any other suggestions? Feel free to take a look at my logs from yesterday’s flight which went MUCH more smoothly using the internal Navio+ compass. One question I had is, why does APM think that my Vcc is zero? I believe that is supposed to be the APM board voltage, which in this case should be 5v. I’m using the stock Emlid supplied Power Module to power the board an a 2.5a 5v step down converter for the servo rail.

From DIY Drones:
So I took comments into consideration and did a couple tests. I took the laptop out to the field and hooked up to telemetry via my RFD900+ serial modem. First I tried the compass configuration noted above. The main screen I was looking at was the EKF status screen (opens when you are in Mission Planner and you click on “EKF” on the HUD. First thing I noticed was that when you rev the motors from 0 to 50% or so (while on the ground) you get EKF very Red for Compass. So this definitely points to the compass as being part of the issue. If anyone has had luck using this compass for APM, let me know. Maybe its just too close to the battery wires (at least 200mm+ away though?). It is not currently on a mast, but I felt that it was far enough away from DC noise that interference there wouldn’t be an issue, while clearly it is if the EKF status is directly proportional to the throttle level!

The easiest thing to do was disconnect the external compass, use the internal compass, and calibrate. So that’s what I did. I performed the same test, and while the EKF definitely was poorer the faster I increased the throttle, it was noticeably less than using the external compass. So the sun setting, I take off and perform what I would consider a perfectly normal flight just using the internal compass. However, upon post flight analysis I get:

Log File C:/Program Files (x86)/Mission Planner/logs/QUADROTOR/1/2016-01-25 16-59-17.log
Size (kb) 9707.171875
No of lines 129429
Duration 0:09:36
Vehicletype ArduCopter
Firmware Version V3.3
Firmware Hash 340970fc
Hardware Type
Free Mem 0
Skipped Lines 0
Test: Autotune = NA -
Test: Balance/Twist = NA -
Test: Brownout = GOOD -
Test: Compass = FAIL - Large change in mag_field (191.71%)
Max mag field length (814.09) > recommended (550.00)
Test: Dupe Log Data = GOOD -
Test: Empty = GOOD -
Test: Event/Failsafe = GOOD -
Test: GPS = GOOD -
Test: IMU Mismatch = NA -
Test: Parameters = GOOD -
Test: PM = NA -
Test: Pitch/Roll = NA -
Test: Thrust = NA -
Test: VCC = FAIL - VCC below minimum of 4.6v (0.0v)

The compass failed WORSE than before… but it always pointed in the correct direction and the video clearly shows no erroneous altitude measurements nor AHI issues like before.

So the questions arise:

  1. Has anyone had luck with the HMC5883L Triple-axis Magnetometer (Compass) Board - HMC5883L : ID 1746 : $9.95 : Adafruit Industries, Unique & fun DIY electronics and kits compass running APM Copter v3.3+?
  2. If so, where did you mount the compass? Was there a certain orientation you used? Did you experience any bad mag/ekf data at first and need to adjust it?
    I think that with the right placement and tuning I’ll get my external compass to work properly and wont have a “Compass Fail” for variance post flight, but for now the Navio+ internal compass is doing the job until I can figure out what to do with the above external or replace it entirely.

Also here are the logs from my “good” flight. Also thanks again Craig/Kev for the log analyzer. Here are its output:

APM Log File Analiser v2.2.0.8
Log FileName: A:\guest\Videos\GoPro\QuadCopter\Fiesta Island\20160125\Logs\2016-01-25 16-59-17.log
Ardu Version: V3.3 Build: (340970fc)
Ardu Type: APM:Copter
Hardware: 259
Frame Type: X
No. Motors: 4
Data Found in APM Log File:-
IMU, GPS, CTUN, PM, CURR, MSG, MAG, CMD, ATT, MODE, STRT, EV, DU32, AHR2, EKF1, EKF2, EKF3, EKF4, UBX1, UBX2, RCIN, RCOU, BARO, RAD
Data NOT Found in APM Log File:-
NTUN, ATUN, ATDE, MOT, OF, INAV, D16, DU16, D32, DFLT, PID, CAM, ERR, GPS2, IMU2, IMU3, MAG2, MAG3, TERR, POWR, SIM
FAILSAFE SETTINGS:
Main Battery = Not Activated
Ground Control = Activated, with always RTL
Receiver = Activated at 975 PWM with always RTL
GEO Fence = Not Activated

PARAMETER ISSUES:-
ARMING_CHECK = 0
Warning: Pre-Arming Check is not Active (0=Off & 1=Fully On)

TUNE = 0
Information: CH6 Tuning is Disabled
06/01/1980 - 000560: Mode Changed to Stabilize
06/01/1980 - 000560: Mode Initialised at 0 0 Alt: 0 Spd:0
06/01/1980 - 000561: Armed
Limited Data Available - Mode Ignored
06/01/1980 - 000562: Mode Changed to Stabilize
06/01/1980 - 000562: Mode Initialised at 0 0 Alt: 0 Spd:0
24/01/2016 00:00:17 - 000579: Take Off
24/01/2016 00:00:17 - 000579: Mode Time Changed to 24/01/2016 00:00:17
24/01/2016 00:00:17 - 000588: Simple Mode is Disabled
24/01/2016 00:00:17 - 000588: APM Information: RC Pre-Arm checks have been completed.
24/01/2016 00:00:17 - 000588: APM Information: All Pre-Arm checks have been completed.
24/01/2016 00:00:17 - 000588: APM Information: USB is NOT Connected.
24/01/2016 00:00:17 - 000588: APM Information: System Time has been set from the GPS.
24/01/2016 00:00:17 - 000588: APM Information: Home is NOT Set
24/01/2016 00:00:18 - 000821: Super Simple Mode is On
24/01/2016 00:00:18 - 000821: APM Information: Home is Locked
24/01/2016 00:00:19 - 000978: *** Update Program to include this Event *** : 28
24/01/2016 00:00:19 - 001048: Auto Mode is waiting for Throttle.
24/01/2016 00:00:29 - 003332: LAND_COMPLETE_MAYBE
24/01/2016 00:00:30 - 003509: Landed
Alt(m ~ ft) Spd(m/s ~ mph) Dist(km ~ mi) Launch(m ~ ft) GPS-Sats GPS-Hdop Eff(mA/min)
Max 0 ~ 0 1 ~ 2 0.00 ~ 0.00 4.19 ~ 13.73 8 2.22 N/A
Avg -2 ~ -7 0 ~ 0 N/A ~ N/A N/A ~ N/A 7 1.70 ~~~
Min -6 ~ -20 0 ~ 0 N/A ~ N/A 0.00 ~ 0.00 6 1.23 N/A
Stabilize Flight Time (Session)= 13 seconds, 0:13
Stabilize Flight Time (Total)= 13 seconds, 0:13
WARNING: Less than 9 Satellites was detected during this mode.
24/01/2016 00:00:36 - 004853: Take Off
24/01/2016 00:00:36 - 004860: Auto Mode is waiting for Throttle.
24/01/2016 00:00:39 - 005396: SET_HOME
24/01/2016 00:00:39 - 005397: Set Home as 32.7774900 -117.2217000 Alt:3.85
24/01/2016 00:00:40 - 005732: *** Update Program to include this Event *** : 28
24/01/2016 00:09:52 - 128890: LAND_COMPLETE_MAYBE
24/01/2016 00:09:52 - 129070: Landed
Alt(m ~ ft) Spd(m/s ~ mph) Dist(km ~ mi) Launch(m ~ ft) GPS-Sats GPS-Hdop Eff(mA/min)
Max 91 ~ 298 17 ~ 38 3.04 ~ 1.89 202.73 ~ 665.11 18 1.8 N/A
Avg 11 ~ 36 5 ~ 11 N/A ~ N/A N/A ~ N/A 16 0.81 554.57
Min -6 ~ -20 0 ~ 0 N/A ~ N/A 2.70 ~ 8.85 9 0.71 N/A
Stabilize Flight Time (Session)= 556 seconds, 9:16
Stabilize Flight Time (Total)= 569 seconds, 9:29
24/01/2016 00:09:54 - 129429: DisArmed
Overall Flight Summary:
Alt(m ~ ft) Spd(m/s ~ mph) Dist(km ~ mi) Launch(m ~ ft) GPS-Sats GPS-Hdop
Max 91 ~ 298 17 ~ 38 3.04 ~ 1.89 202.73 ~ 665.11 18 2.22
Avg 0 ~ 0 0 ~ 0 N/A ~ N/A N/A ~ N/A 15 0.83
Min 0 ~ 0 0 ~ 0 N/A ~ N/A N/A ~ N/A 6 0.72
WARNING: Less than 9 Satellites was detected during this mode.
Power Summary:
Battery(V) Vcc(V) Current(A) Used Cap(mAh) Eff(mA/mim)
Max 16.15 0.00 56.86 5176 N/A
Avg 0.00 0.00 32.71 N/A 545.80
Min 13.60 0.00 0.62 N/A N/A
Overall Flight Time = 569 seconds, 9:29
*** Vibration Analysis has been removed from v2.2 onwards, you should now
*** review the “Vibration Graph” instead to understand your levels of vibration.

all this explanation but you have not answered igor’s question…