RCIO Failure

I’m in a situation similar to what was described here:

RCIO was working just fine last night. In fact, right afterward I backed up my image to a laptop. As of today:

I have tried modprobe, checked dmesg and lsmod. I tried to run the emlidtool force update and unloaded/loaded rcio_spi as suggested. There are no entries for rcio in either. If I run

sudo modprobe rcio_spi

It completes and then lsmod | grep rcio lists:
rcio_spi 16384 0
rcio_core 36864 1 rcio_spi

But it doesn’t matter:

(openvino) pi@navio:~ $ sudo emlidtool rcio update
Please verify that rcio_spi is loaded and /sys/kernel/rcio/status/crc exists

/sys/kernel/rcio/status/crc does not load under any condition. Here are the results of the commands suggested in the other topic:

(openvino) pi@navio:~ $ sudo emlidtool rcio update -f
Updating firmware using /lib/firmware/rcio.fw
Connect: done
Attach: done
Catch vectors: done
Erase: done
Flashing
Load: done
Run: done
100% (100 of 100) |########################################################################################################| Elapsed Time: 0:00:13 Time: 0:00:13
You have successfully updated RCIO firmware

You need to reboot your device

(openvino) pi@navio:~ $ sudo modprobe -r rcio_spi && sudo modprobe rcio_spi
(openvino) pi@navio:~ $ dmesg | grep rcio
[ 475.300455] rcio_core: loading out-of-tree module taints kernel.
(openvino) pi@navio:~ $

I think somehow this HW is now dead. Is there anything else I should try? I reflashed my image from the working copy from last night and repeated all of the steps above without any luck.

If you have a logic probe that might be useful to Emlid for immediate debugging to get the output of maybe the rcio pins? I’m not sure where they would want the probes. I would imagine on whatever pins are for connecting the navio to the raspi. I would see what Emlid has to say. When my board broke the raspi-wifi broke as well which makes me think of some part recieved to much current. #72 Making the invisible visible: Tools for debugging and re-engineering electronic designs - YouTube might be helpful, about six mins in is a part on logic analyzers.

Hi @jleibund,

Please post here the output of the following commands:

  • readlink -f /usr/bin/arducopter
  • readlink -f /usr/bin/arduplane
  • readlink -f /usr/bin/ardurover

Last login: Thu Jun 6 06:54:17 2019 from 192.168.86.214
[setupvars.sh] OpenVINO environment initialized
(openvino) pi@navio:~ $ readlink -f /usr/bin/arducopter
/opt/ardupilot/navio/arducopter-3.6/bin/arducopter
(openvino) pi@navio:~ $ readlink -f /usr/bin/arduplane
/opt/ardupilot/navio/arduplane-3.9/bin/arduplane
(openvino) pi@navio:~ $ readlink -f /usr/bin/ardurover
/opt/ardupilot/navio2/ardurover-3.4/bin/ardurover
(openvino) pi@navio:~ $

@jleibund,

Please disable all ArduPilot services with the following commands:

  • sudo systemctl disable arducopter
  • sudo systemctl disable arduplane
  • sudo systemctl disable ardurover

And reboot the device. After rebooting, run emlidtool with sudo emlidtool ardupilot.

@tatiana.andreeva

OK - done. I disabled all three. Then rebooted then ran the command – complains about no crc:

(openvino) pi@navio:~ $ sudo emlidtool ardupilot
Traceback (most recent call last):
File “/usr/local/lib/python3.5/dist-packages/emlid/rcio/versionchecker.py”, line 43, in init
with open(self.crc_path, ‘r’) as f:
FileNotFoundError: [Errno 2] No such file or directory: ‘/sys/kernel/rcio/status/crc’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/bin/emlidtool”, line 11, in
sys.exit(main())
File “/usr/local/lib/python3.5/dist-packages/emlid/emlidtool.py”, line 355, in main
emlidtool.parse_args()
File “/usr/local/lib/python3.5/dist-packages/emlid/emlidtool.py”, line 258, in parse_args
getattr(self, args.command)()
File “/usr/local/lib/python3.5/dist-packages/emlid/emlidtool.py”, line 286, in ardupilot
self.func(ardupilot, args)
File “/usr/local/lib/python3.5/dist-packages/emlid/emlidtool.py”, line 333, in func
args.func(args)
File “/usr/local/lib/python3.5/dist-packages/emlid/util/util.py”, line 37, in wrapped
return fn(*args, **kwargs)
File “/usr/local/lib/python3.5/dist-packages/emlid/emlidtool.py”, line 49, in do_ardupilot
if FirmwareVersionChecker().update_needed():
File “/usr/local/lib/python3.5/dist-packages/emlid/rcio/versionchecker.py”, line 29, in wrapped
fn(*args, **kwargs)
File “/usr/local/lib/python3.5/dist-packages/emlid/rcio/versionchecker.py”, line 47, in init
’ {crc_path} exists’.format(crc_path=self.crc_path))
emlid.rcio.versionchecker.CrcNotFoundError: Please verify that rcio_spi is loaded and /sys/kernel/rcio/status/crc exists

@jleibund,

Could you please share photos of your hardware setup?

I’ll post some tonight but they aren’t much different from what I already posted. When running the commands above:

  • RPi using standard 5v power supply in wall socket
  • Navio fully seated in header
  • RPi has an Intel NCC2 attached to USB
  • picamera in video port

When on rover:

  • Power from 7.4v Lipo using UBEC through servo rail- tested that for 5V. Max draw 3A.
  • receiver SBUS to servo rail 0
  • servos on servo rail 1 and 3 pwm
  • 2 mb1240 sonar on servo rail 4 and 5- those ports exported to gpio for triggering sonar independently

On the rover, of course, because rcio is dead the sonar and servos are no longer controllable. Will post photos.

Here is the RPi, navio, NCC2 and Camera - with 5v RPI power source (used in the tests/cmds above). There is a closeup of the RPi / navio2 header. Two of the sonar. One of the car that shows the 5v 3A BEC soldered to the ESC power. I clipped the red wire between the ESC’s 6V BEC out and servo rail (not shown)… so only one power source through the BEC shown to the servo rail. One photo of the battery / harness as well. That’s everything. @tatiana.andreeva

Sorry-- the ADC harness is attached. I run ADC3 and 4 to the sonar TX channel. The sonar trigger channel is what is attached to Servo rail to the pins setup as GPIO as described above so that they can be triggered independently.

@tatiana.andreeva do you see any issues with the HW configuration above?

Hi @jleibund,

Sorry for the delayed response.

Could you please disconnect all the hardware from Navio2 and check whether it changes something or not?

Do you have a possibility to test Navio2 with another RPi? Please also check that the header isn’t damaged.

@tatiana.andreeva thanks for your help so far. I have disconnected all hardware and attached pictures so you can see the header and RPi pins.

Nothing appears to be damaged. I unplugged the picamera and USB extensions, reseated the navio2 (pic of that), booted and attempted to run ‘sudo emlidtool ardupilot’ but its still the crc issue:

(openvino) pi@navio:~ $ sudo emlidtool ardupilot
Traceback (most recent call last):
File “/usr/local/lib/python3.5/dist-packages/emlid/rcio/versionchecker.py”, line 43, in init
with open(self.crc_path, ‘r’) as f:
FileNotFoundError: [Errno 2] No such file or directory: ‘/sys/kernel/rcio/status/crc’

I am able to access the navio2 IMU via python. But the rcio, servo rail, and ability to access/control the LED are still not there (from the navio2 python examples).

I’m going to need to continue this project soon - do I need to purchase another navio2 from your website? I had bought mine via Amazon.

Here are the pics.

@tatiana.andreeva I don’t have a 2nd RPi to test this with unfortunately.

@jleibund,

Could you please post the output of sudo emlidtool test once again?

@tatiana.andreeva sure thing- when I’m back home tonight. Will post it.

@tatiana.andreeva here is the output:

Last login: Tue Jun 11 06:57:22 2019 from 192.168.86.214
[setupvars.sh] OpenVINO environment initialized
(openvino) pi@navio:~ $ sudo emlidtool test
2019-06-12 00:30:43 navio root[29520] ERROR mpu9250: Failed
– Reason: No connection!
2019-06-12 00:30:43 navio root[29520] ERROR adc: Failed
– Reason: rcio_adc module wasn’t loaded
2019-06-12 00:30:43 navio root[29520] ERROR rcio_firmware: Failed
– Reason: Please verify that rcio_spi is loaded and /sys/kernel/rcio/status/crc exists
2019-06-12 00:30:43 navio root[29520] INFO ms5611: Passed
2019-06-12 00:30:43 navio root[29520] ERROR lsm9ds1: Failed
– Reason: Bus error on LSM9DS1
2019-06-12 00:30:43 navio root[29520] ERROR rcio_status_alive: Failed
– Reason: rcio_status wasn’t loaded
2019-06-12 00:30:43 navio root[29520] ERROR pwm: Failed
– Reason: rcio_pwm module wasn’t loaded
2019-06-12 00:30:43 navio root[29520] ERROR gps: Failed
– Reason: Timeout
(openvino) pi@navio:~ $

@tatiana.andreeva anything else I can provide?

Hi @jleibund,

This behavior looks quite strange. We need to figure out if this is Navio2 or RPi issue somehow.

Have you tried to reflash with the latest stable image from docs? If not, may I ask you to check it?

Is anything getting hot on Navio2 after you supply it with power?

Could you please measure the voltage on L2 inductor as well (check the attached photo)?

@tatiana.andreeva L2 inductor voltage is 0.0mV. I don’t really feel anything getting hot anywhere. I’ll reflash from the original emlid image in a moment and run emlidtool test.