Ran out all six config attempts

The firmware was on the eMMC. The bootloader was in boot0. The flash had reported success at every step. And the board did nothing.

With a real serial console attached and the programming port unplugged, it turned out it had not been doing nothing at all. It had been saying this:

DDRINFO: Ran out all [6] cfg attempts. A non supported configuration

with a measured mr5=0xff. That is SPL — the small first-stage loader inside imx-boot — failing to train the LPDDR4 memory. It never hands off to U-Boot proper. No U-Boot banner, no Linux, no console past SPL, nothing on the network.

Five words, and they name the problem precisely. I just had not been listening, for the reason in the previous part: I had convinced myself the board was up because something on my network was answering, and that something was a different machine.

Why memory training fails at all#

Before the fix, briefly why this is a thing that can go wrong. LPDDR4 is not simply wired up and used. At every power-on, SPL runs a training sequence — timings, drive strengths, delays — against the specific memory parts on the specific module. The parameters it tries come from a configuration compiled into the bootloader, describing which memory variant this board has.

Give it the configuration for a different variant and it will try, honestly and repeatedly, and fail. Six attempts, then that message. It cannot fall back, because there is nowhere to fall back to: without working DRAM there is no code to run.

So the message means: the memory on this board is not the memory I was told to expect.

Comparing against something known to work#

The useful move here was not to read my own config harder. It was to compare it against a bootloader that demonstrably boots this exact module — the factory one, which I had backed up before touching anything.

Its SPL identifies itself:

U-Boot SPL 2023.04-IOTDIN-IMX8P-1.3-g70a7183523f

with DDR firmware 202006. Mine was built from u-boot-compulab commit 1241cf66, the v2023.04 tip. Same U-Boot version. Different commit. Same DDR firmware blobs, which I checked — so it was not the blobs.

That left the configuration, and this is where it got interesting.

A Kconfig choice that had been split underneath me#

The recipe I had worked from set a single symbol:

CONFIG_DRAM_D2D4=y

That symbol does not exist at my pinned commit. Between the version the recipe was written against and 1241cf66, CompuLab split it into a Kconfig choice in board/compulab/plat/imx8mp/Kconfig, with three options:

CONFIG_DRAM_D2
CONFIG_DRAM_D4
CONFIG_DRAM_D1D8

A Kconfig choice always resolves to something. Mine resolved to the default, DRAM_D2, which does not train a 4 GB module. My defconfig said nothing about DRAM at all, so nothing looked wrong in it — the wrong value was the one I never wrote.

This is the failure mode that makes pinning a version simultaneously essential and dangerous. Pinning gave me a reproducible build. It also meant I was building against a Kconfig structure that had changed shape since the documentation I was following, and the change was invisible in my own files.

The bdinfo output from part three had told me the answer a week earlier — DRAM approximately 4 GB — I just had not connected it to a symbol I did not know existed.

The fix:

CONFIG_DRAM_D4=y

Rebuild U-Boot, reflash with fastboot, watch tio. The whole cycle is a few minutes, which is the one mercy of this class of bug: verifying a guess is cheap.

Two more that were hiding behind it#

The DRAM failure was masking two further bugs. Neither could show itself while SPL was dying, and both would have stopped the board at the next stage. Three fixes in total got it to boot, and the reference system had all three — I had missed all three.

The environment was too small, in the wrong place#

CONFIG_ENV_SIZE=0x20000     # was 0x4000
CONFIG_SYS_MMC_ENV_DEV=2

fwup writes a 128 KB (0x20000) U-Boot environment containing the console setting and the A/B boot logic. U-Boot was compiled expecting 0x4000 — 16 KB — and to look at a different MMC device. It would have read a default environment instead of mine, which means no console on the right port and no A/B boot.

This was the risk I had written down at the start, in three separate files, as "confirm the environment location at first boot". I was right that it was a risk and wrong about the mechanism: I was worried about the byte offset and the choice of user area over boot partition, and the actual bug was the size and the device number.

U-Boot could not read the root filesystem#

CONFIG_FS_SQUASHFS=y

The Nerves root filesystem is a compressed SquashFS, and U-Boot has to read /boot/Image and the device tree out of it before Linux exists to do it. Without SquashFS support compiled in, that fails with Can't set block device.

Obvious in hindsight. Entirely invisible while SPL was still failing.

It boots#

Power-cycle, and the console produces a Nerves banner:

compulab_cs_gw 0.1.0
Platform: iotdin_imx8p arm
Firmware: Valid (A)

38 applications started, a live iex(1)> prompt on the DBG console, and ifconfig showing eth0 up with a DHCP address of 192.168.1.235 — MAC 00:01:c0:3e:81:66, which I had written down precisely so that I could believe it this time. can0 and wwan0 both present.

From that point on there is no more bench work. Updates go over the air:

MIX_TARGET=iotdin_imx8p mix upload 192.168.1.235

What I took from it#

A choice in Kconfig always has a value, whether or not you set one. An absent symbol in a defconfig is not a neutral state. When following a recipe written against an older tree, the symbols that were renamed or restructured are more dangerous than the ones that were removed, because a removed symbol produces a warning and a restructured one silently resolves to a default.

Diff against something that works. I could have read my defconfig for another week without finding this. Fifteen minutes comparing my SPL banner with the factory SPL banner gave me the commit difference, and the commit difference gave me the Kconfig split.

Get a real console before forming any theory. Every hour I spent believing the board had booted was an hour spent debugging a machine that was not the machine in front of me. The board had been printing the answer, in plain language, the entire time.