I am letting the community know of a potential serious issue with the Navio2 that can result in a fly away condition. I have discussed with the developers over the course of several weeks now and have refrained from posting this specific issue until they could come up with a fix.
I have come up with a temporary fix, but this requires modification and compiling of the ArduPilot code, which I will post below.
I have uploaded the modified files here ArduPilot Modified Code that you can download and replace/modify within ArduPilot code.
Github repository setup here with the latest code: Github ArduPilot Modificaiton
The files above are based on a 10 second (.1hz) update instead of the 1 second (1hz) update described below.
The Issue
The specific issue I have found is specific to the Navio 2 boards, with the additional MCU co-processor. The issue can arise if for any reason the ArduPilot program running on the raspberry pi experiences a crash or no longer responds. The MCU will hold the last state of the PWM outputs that were sent by ArduPilot, which can result in a fly away condition.
How to Test this Issue
Before doing this test, please remove all props from your airframe!!
This issue can be tested fairly easily. First launch the ArduPilot (ArduCopter, ArduPlane, etc) from the Raspberry pi shell (./ArduCopter -A udp:0.0.0.0:14550) without the & sign, this will run it in the shell and keep the process in the foreground.
Remove your props!!!
Next arm your airframe and go full or partial throttle. You are going to go back to the raspberry pi shell and hit Control-C. This will essentially kill the ArduPilot application running on your raspberry pi.
You will now be in the situation where there is no flight software running however the throttle is still wide open. This is the issue and will result in a fly away condition that you have no control over… wave your aircraft goodbye!!
Remove power from the raspberry pi or issue the sudo shutdown -h now command to disable the PWM outputs.
The Temporary Fix
The fix is not perfect and a proper fix will be to modify the MCU to add a fail safe if the ArduPilot program dies. There is currently a fail safe implemented in the MCU co-processor if the raspberry pi dies (powers off) or if the Kernel panics, but not if the ArduPilot program dies.
The temporary fix essentially enables the watchdog timer in the raspberry pi and the ArduPilot application “pats the dog” or resets the watchdog timer from timing out and re-booting the raspberry pi. This will reboot the raspberry pi in the event that the ArduPilot application dies. I have included a paramter that can be enabled or disabled from Mission Planner in the event you need to frequently stop or start the applicaiton for testing purposes. The default paramter is set to disable the watchdog timer (ie. disabled = 0)
First you will load the watchdog kernel module by issuing the following command:
$ sudo modprobe bcm2708_wdog
This loads the watchdog kernel module, but doesn’t start the timer… I’ll explain how that happens in a bit.
lsmod
This should list all the kernel modules that are loaded, you should see the following:
bcm2708_wdog 3537 0
The watchdog kernel module should now be loaded, however you will have to make this permanent if a re-boot happens. To do this you will have to modify the /etc/modules file (using nano or vim) and include the following:
bcm2708_wdog
Close and save the file.
Next you will modify the following three files in the ArudPilot source tree.
APM_Config.h
Parameters.h
Parameters.cpp
Copter.h
UserCode.cpp
The watchdog timer will only start when you open the /dev/watchdog file. When it’s opened for read/write, the timer is started automatically and begins its countdown. At this point, if you fail to write any character (except the special “V” character) to the device file, the timer will timeout and reboot the Rpi. Writing a value to the file will reset the counter, this is known as “patting the dog” as you are constantly resetting the timer to stop it from timing out and resetting the Rpi. Writing the special character “V” will disable the watchdog timer.
Compile and download to your raspberry pi… Please note that I have not fully tested this on a running machine, but I have bench tested it.
To enable the watchdog timer, list the full parameter list and look for the parameter WDT_enable. Setting this to 1 will enable the watchdog timer while a 0 will disable the functionality.
Dale