Board randomly hangs after boot

  1. Image and ArduPilot version.

Linux navio 4.14.95-emlid-v7+ #1 SMP PREEMPT Mon Feb 4 15:59:56 MSK 2019 armv7l GNU/Linux

Vendor: Emlid Limited
Product: Navio 2
Issue: Emlid 2019-02-27 1548ce733215806c8ad171c8b78ebca0c7cf8395
Kernel: 4.14.95-emlid-v7+
RCIO firmware: 0xb09979ae

  1. Detailed problem description. How does observed behavior differ from the expected one?

Board randomly hangs up after the boot process is over. Flashing yellow led turns off completely, or stops flashing. No power issues. Ssh access is not compromised. All the processes seem to be running without issues. No clue records in logs (although I don’t know if I am actually checking the right place for those)

  1. Your step by step actions.
  • Connect the battery, wait for boot up completion.
  • When Led starts flashing yellow, connect CGS over UDP (tried both, Mission Planner and APM Planner)
  • After a while board hangs up, either with the led switched off or led solid yellow light. ESCs are not happy (lost control signal constant beeping)
  • GCS tries to reconnect and times out eventually.
  • Board never comes back online.
  • RPi3 works fine - no hiccups, nothing.
  • arducopter processes are alive (htop)
  • Executing sudo killall -9 arducopter
  • board reboots and gets back online for awhile. ESCs are now become silent. This time, the period board is alive, seem to be longer than when it boots for the first time.
  • Eventually it hangs again. Led is off or not flashing - ESCs are beeping…
  • Scratching a rather bald head in frustration, and starting the process all over again…

Need your help with troubleshooting. Please… I was never able to finish the setup and arm the copter at least once, and I already have this board for about a week.

Hi @vitalie_cartera,

Does this behavior persist, if you supply the Navio2 over RPi USB port?
Could you please share photos of your hardware setup?

Thanks for getting back to me Tatiana,

Same thing if RPi is just powered over usb port.
Here is the photo of the setup as requested (the board is mounted temporary like that, until I am sure it works properly)

Anyone?

I am beginning to regret purchasing this board…
Guys - perhaps you have some tools I could use to further diagnose the issue, when board is in this state. I would at least want to know it this is a hardware related problem or not. What can I do to query the board? All the h/w test are passing when using emlidtool, however, perhaps it gets hot or something, and fails later on?
Maybe I could try to re-flash it? I know it might sound silly but who knows, right?

Please get back to me with suggestions. I really would appreciate the support from community out there…

Difficult to diagnose. Sometimes we have some clues…

If default appear when Pi is connected to 2A external plug or when Navio is powered from Emlid power module, we can rule out this item.

You can try to remove the Navio module and extender, spray some contact fluid on the plugs then reassemble it. If problem is still there, try with another SD card with fresh Emlid image. The latest is fit and I did not have problems with it on my unit.

Marc

thanks Marc,
this is not it. Took the trouble cleaning up everything and although the fleemzy Navio2 connector type is a rather poor choice for this board (reliability wise) it seem to me that this is not the cause.
Perhaps I need to flush the board still? How do I do that?

Hi @vitalie_cartera,

Please run this example from our docs and share the results.
Also, please post here the output of cat /sys/kernel/rcio/adc/ch1

After that, may I ask you to unplug all connected to servo rail equipment and check if the issue remains?

Hi Tatiana,

That example only can be executed if APM is not running.
pi@navio:~/Navio2/Python $ python ADC.py
APM is running. Can't launch the example

Besides, I actually ran all those examples couple of days ago and have not seen any obvious issue.

When motors started to beep, this is the output you’ve asked:
pi@navio:~/Navio2/Python $ cat /sys/kernel/rcio/adc/ch1
19316

Disconnecting everything on servo rail, does not fix the problem. After a while, (randomly) board’s LED goes solid or switches off - with, or without servo rail components attached.

please advise.

Hi @vitalie_cartera,

May I ask you to disconnect everything from Navio2 at all and test it again?
Please check whether something is getting hot on Navio2.

sure, Tatiana, no problem (it was one of the first troubleshooting I’ve done with this board anyway).

All disconnected. Powered from micro USB on RPi. Same behaviour. No signs of overheating.
Although, now that you brought this up, I remember when I first plugged the board in, I checked the chips and it felt rather hot to the touch on the top side. Not on a particular chip. I even looked underneath and decided that it might be due to a poor air flow between the RPi and Navio. After that, every time I checked the boards, they were just warm to the touch, (as they are now). Just FYI…

pi@navio:~/Navio2/Python $ sudo emlidtool test
2019-04-18 19:44:37 navio root[905] INFO mpu9250: Passed
2019-04-18 19:44:37 navio root[905] INFO gps: Passed
2019-04-18 19:44:37 navio root[905] INFO lsm9ds1: Passed
2019-04-18 19:44:37 navio root[905] INFO ms5611: Passed
2019-04-18 19:44:37 navio root[905] INFO pwm: Passed
2019-04-18 19:44:37 navio root[905] INFO adc: Passed
2019-04-18 19:44:37 navio root[905] INFO rcio_status_alive: Passed
2019-04-18 19:44:37 navio root[905] INFO rcio_firmware: Passed

pi@navio:~/Navio2/Python $ python ADC.py
A0: 4.7190V A1: 0.0000V A2: 0.0160V A3: 0.0160V A4: 0.0160V A5: 0.0240V
A0: 4.7370V A1: 0.0000V A2: 0.0160V A3: 0.0160V A4: 0.0200V A5: 0.0260V
A0: 4.7430V A1: 0.0000V A2: 0.0160V A3: 0.0160V A4: 0.0220V A5: 0.0260V

One small thing - the following command
sudo systemctl stop arducopter.service
always takes forever to complete. Don’t know if it is normal.

Also, I see about 7 instances of arducopter processes when checking with htop. Is this normal?

Hi @vitalie_cartera,

Have you tested whether this behavior appears with another RPi or not?

Could you please also take high-resolution photos of Navio2? We’ll check if there are any connection issues with the components on the board.

yes, Tatiana, I have checked on two different RPi’s. Same problem.

Here are some pictures. If you like a macro of a specific area, just let me know and I’ll post it!

Hi @vitalie_cartera,

We’ll look into it and I’ll write back to you.

Thanks!

Hi @vitalie_cartera,

When did you purchase Navio2?
Did the issue appear from the beginning?

Navio2 board was purchased on March 9th. As stated in my first post, I was never able to complete the initial setup, nor arm the quad once.

@vitalie_cartera,

Please contact us on support@emlid.com and send the following information:

  • Order date and number;
  • Shipping address.

Thanks!