HAT not detected - EEPROM possibly corrupt?

Hello all!

Recently while trying to troubleshoot the IMUs on my Navio2, I found emlidtool was not detecting the HAT anymore.

pi@navio:~ $ sudo emlidtool
emlidtool version: 1.0.0
Vendor: ¯\_(ツ)_/¯ Unknown, but most likely Emlid
Product: Seems like you booted without Navio properly screwed!
Issue: Emlid 2017-09-22 94b6586ea415c117b57d0d9eed9fc2df0bf1bcfd
Kernel: 4.9.45-94f47ec-emlid-v7+
emlidtool -h to get more help

I confirmed that the HAT part of the device tree wasn’t showing up:

pi@navio:~ $ ls /proc/device-tree/
#address-cells  clocks              fixedregulator_5v0  memreserve     serial-number  system
aliases         compatible          interrupt-parent    model          #size-cells
axi             cpus                leds                name           soc
chosen          fixedregulator_3v3  memory              __overrides__  __symbols__
pi@navio:~ $ stat /proc/device-tree/hat/
stat: cannot stat '/proc/device-tree/hat/': No such file or directory
pi@navio:~ $ stat /proc/device-tree/hat/product
stat: cannot stat '/proc/device-tree/hat/product': No such file or directory
pi@navio:~ $ stat /proc/device-tree/hat/vendor
stat: cannot stat '/proc/device-tree/hat/vendor': No such file or directory

I also tightened all of the screws. No dice. (And it’s on the Pi3 very snug as it is.)

To check the I2C bus, I turned on i2c_vc=on and rebooted and can see that the EEPROM is definitely there:

pi@navio:~ $ sudo i2cdetect -y 0
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: 50 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --                         
pi@navio:~ $ sudo i2cdump -y 0 0x50 i
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 00 02 00 69 00 00 00 01 00 00 00 2d 00 00 00 db    .?.i...?...-...?
10: c9 e0 f0 21 ae 44 ae 25 41 d2 51 e6 16 8b 0f 03    ???!?D?%A?Q?????
20: 00 01 00 0e 07 45 6d 6c 69 64 20 4c 69 6d 69 74    .?.??Emlid Limit
30: 65 64 20 4e 61 76 69 6f 20 32 62 4b 02 00 01 00    ed Navio 2bK?.?.
40: 20 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00     ....?..........
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
60: 00 00 6d 6d ff ff ff ff ff ff ff ff ff ff ff ff    ..mm............
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................

100 bytes seems oddly short for a dtb. Does the Navio2 EEPROM only store the product/vendor strings? Or did I somehow corrupt/truncate my EEPROM? I don’t have a copy of the known-valid EEPROM data - could anybody with a working Navio2 share the proper EEPROM contents? Does someone with better HAT expertise have any insights as to what causes the Raspberry Pi bootloader not to populate the /hat node?

Hi, Sam!

Awesome investigation of the issue. Thanks a lot for your elaborate report.

Yep, that’s the case. Other configuration stuff is stored at /boot. This is actually more or less standard practice. When first Navios were assembled the whole EEPROM with DTBs mechanism hadn’t existed as even device tree was not used at the time.

sudo vcdbg log msg with a dtdebug=1 in /boot/config.txt can also help us figuring out the root cause.

Anyway, sudo i2cdetect -y 1 and a more detailed description of the problem with the IMUs will be of great help.

Hey George! Thanks for your quick reply. :slight_smile:

Agh, I’ve been looking for this! Every time I searched for how to look at the boot logs in a Raspberry Pi, I’d find Stack Exchange answers “helpfully” directing me to dmesg. Now that I know the proper command, I’ve pastebinned that log here.

I’m not even sure there’s a hardware issue with the IMUs. The main problem I’m seeing is Ardupilot throwing bad AHRS and “gyros still settling” quite often. Sometimes the attitude indicator spins around randomly and on some boots it goes 10+ minutes without getting a GPS lock (even though u-center confirms the uBlox has a fix). Ardupilot’s error messages are pretty obtuse (and I’m a newbie to Ardupilot as it is) so I decided to start with the hardware and work my way up, which is when I noticed the problem with emlidtool.

Here’s the i2cdetect output for bus 1:

pi@navio:~ $ sudo i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- 3d 3e -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- 77   

0x77 is, of course, the barometer. 0x3d and 0x3e are navigation/safety/strobe lights that I connected to the I2C port.

Of note is that my I2C wiring job is pretty basic and the 1MHz clock may be causing a fair amount of ringing in the line. Sometimes one of the navigation lights will skip a strobe - but I’m not sure if that’s due to wiring or the lights themselves just not liking 1MHz clocks. Still, I can’t rule out the possibility that the barometer is erroring when polled by Ardupilot due to the mismatched impedance on the bus from my awful wires. But that would only make sense if I was seeing altimeter problems - I have AHRS problems. AHRS should be entirely independent from the barometer, right?

I could try dropping the i2c-1 rate, but I think I’d rather identify what’s going on with the other subsystems (HAT ID, IMUs, GPS, …) before I start tweaking the one bus I’m pretty confident is fine.

Any chance you have a 433MHz telemetry connected at the same time?

That’s an oddly-specific question… Why 433MHz in particular?

There’s a 915MHz radio on there, but it’s connected to the UART and that’s working just fine.

How might a telemetry radio affect the other buses? Is one of these components known to be susceptible to interference in the 70cm band?

More info:

It seems turning off the light strobing daemon (which writes to the I2C bus about 32 times per second, in short bursts) causes the AHRS and GPS tracking to work a little bit better. I say “it seems” because that’s an unscientific observation and could easily be confirmation bias on my part. But this could make sense if the I2C operations to the lights cause occasional blocking/errors when ArduPilot reads the barometer, which in turn upsets the sensor polling loop and throws all of the sensor readings out of balance. (But that’s a flight-stack concern and deviates from the main topic here, which is that the HAT isn’t detected.)

It’s still a mystery to me why the RPi bootloader doesn’t detect the HAT ID EEPROM, because that points to issues deeper in the system than the software. Even if it’s just a coincidence (and after all, the only thing it really affects is emlidtool, which I can fix by just hardcoding the product/vendor strings), I’d still like to know what’s going on for the sake of understanding.

I have some spare SD cards. I’ll put a fresh Emlid image on one and see what that does, although since I hadn’t really changed the /boot partition until I noticed the HAT wasn’t being autodetected, I doubt I’ll see anything different.

This topic was automatically closed after 100 days. New replies are no longer allowed.