Provisioning the next board

Everything in the previous four parts was learning. This part is the distilled version — what I actually do now to turn a factory unit into one running our firmware, once per board, after which it never happens again because updates go over the air.

It also contains the one bug I would most likely have shipped without noticing, because nothing failed.

The mental model, in thirty seconds

Two artifacts from one mix firmware:

• compulab_cs_gw.fw — base system rootfs fused with the application release. Goes to the eMMC user area, /dev/mmcblk2.
• imx-boot — SPL, ARM Trusted Firmware, OP-TEE and U-Boot in one container. The only artifact not inside the .fw. Goes to the eMMC hardware boot partition, mmcblk2boot0.

Provisioning is therefore two writes, and both go through the board's own U-Boot. uuu and SDP are not involved on a unit that has a working bootloader — which is every unit except a genuinely bricked one.

And the ports, once more, because it is still the thing most likely to waste an hour:

• DBG is the serial console, /dev/cu.usbserial-*.
• OTG is the USB gadget: ums, fastboot, SDP.
• OTG connected at power-on forces SDP. Silent, no eMMC boot, no console. Unplug it to boot, plug it back in once U-Boot is up.

1. Build

cd ~/projekti/compulab_gw/compulab_cs_gw
unset MIX_EXS
export MIX_TARGET=iotdin_imx8p
mix deps.get
mix firmware </dev/null

The unset and the </dev/null are both there for reasons covered in part two. Neither is optional.

Copy two files across:

_build/iotdin_imx8p_dev/nerves/images/compulab_cs_gw.fw
../nerves_system_iotdin_imx8p/.nerves/artifacts/*/build/uboot-custom/flash.bin

I rename the second one to imx-boot locally, because flash.bin tells you nothing six months later.

Host tools needed: brew install fwup android-platform-tools tio.

2. Back up the factory image first

Only meaningful on a pristine unit, and only once per SKU. It takes minutes and it is the difference between "this is recoverable" and "this is a paperweight". The factory bootloader in that backup is also what let me diagnose the DRAM failure in part four, by giving me a known-good SPL banner to compare against.

3. Get a U-Boot prompt

OTG unplugged. DBG connected.

tio /dev/cu.usbserial-XXXX

Power on, press a key to stop autoboot. A factory unit shows CompuLab's prompt, a provisioned one shows iotdin-imx8p=>. Either is fine — both have ums and fastboot.

4. The user area, and the bug I nearly shipped

=> ums 0 mmc 2

Now plug the OTG cable into the laptop. The eMMC appears as an external disk.

diskutil list                     # the ~31 GB EXTERNAL physical disk
diskutil unmountDisk /dev/diskN
sudo fwup -a -d /dev/rdiskN -i compulab_cs_gw.fw -t complete

Check the disk identifier twice. The wrong answer here is your own machine's disk.

Here is the part that matters, and it is not obvious. My fwup.conf marks the application-data partition — mmcblk2p3, f2fs, mounted at /root — with expand = true, so it grows to fill whatever eMMC it lands on.

That expansion only happens when fwup writes the partition table directly to a real block device, because that is the only time it can see how large the device actually is.

If you take the other route — build a firmware.img with fwup -t complete on the build machine and flash that image with uuu — you have baked in a fixed-size partition table. The minimum, 512 MB. And nothing anywhere reports a problem. The board boots, the application runs, and the data partition is fifty-five times smaller than it should be.

I found this by looking, not by failing:

cmd("df -h /root")

A uuu-flashed board showed 512 MB. Reflashed the ums and fwup-direct way, the same board showed 28.2 G.

That is a real production bug that would have gone to a customer, because the only symptom is a number nobody checks until the disk fills up in eighteen months. It is now item one on the validation checklist below.

When fwup prints Success, press Ctrl-C at the U-Boot console to end ums.

Optionally, to burn in a per-unit asset serial:

sudo NERVES_SERIAL_NUMBER=GW-0001234 fwup -a -d /dev/rdiskN -i compulab_cs_gw.fw -t complete

Without it the SoC fuse serial# is used automatically, which is zero-touch and what I use. Getting that working needed a fix of its own: my boardid.config had been written in INI format, which is wrong. It takes boardid's argument form:

-b uboot_env -u "serial#"

With that, hostnames come out as nerves-<serial> instead of nerves-00000000.

5. The bootloader

=> fastboot usb 0

fastboot flash bootloader imx-boot
fastboot reboot

No offset, no mmc partconf. The factory ext_csd already boot-enables boot0 and we overwrite it in place.

6. Validate, don't assume

Unplug OTG, power-cycle, attach tio. In IEx, where cmd/1 runs a shell command:

Both of the first two rows exist because I got them wrong once. That is generally how a checklist earns its entries.

Noise that is not a problem

Worth writing down, because on a first boot it all looks alarming:

F2FS ... Can't find valid ... superblock
pca953x 4-0021: failed writing register
tpm ... error (256)
imx-bus-devfreq ... clk -2

In order: the wiped data partition being formatted for the first time, the digital I/O slots that are physically not fitted, a TPM nothing is using yet, and a clock message from the bus frequency driver. All benign, all present on a working unit.

7. After this, it's over the air

unset MIX_EXS; export MIX_TARGET=iotdin_imx8p; mix firmware
mix upload <device>

This writes the inactive A/B slot, switches to it and reboots. It never rewrites the partition table, the data partition, or imx-boot. Which means an update cannot undo the provisioning work, and cannot change the data partition size either — so getting step 4 right is a one-time-only opportunity per board.

A bad update is recoverable. Nerves.Runtime.revert(reboot: true) on the serial console rolls back to the previous slot, and the startup guard reverts unvalidated firmware automatically.

8. When there's no working U-Boot

If the eMMC is genuinely blank or the boot partition is destroyed, there is no ums and no fastboot, and uuu over SDP is the only way in. On an Apple Silicon Mac that means a direct USB connection and no hubs, and even then expect trouble — a Linux or Windows host is far more reliable.

Afterwards, redo step 4 the ums and fwup-direct way, or you will have the 512 MB data partition again.

Looking back at the whole thing

Five parts, and the shape of it is not what I expected when I started. I had written down two risks: whether binman could assemble the boot container, and where the U-Boot environment should live. The first resolved itself on the first complete build with no intervention. The second was a real bug, but not the one I had described.

What actually cost time was: four packaging bugs in files I had written myself, a vendor tool that does not work on my laptop, a Kconfig option that had been restructured upstream and silently defaulted to the wrong memory variant, and an hour spent debugging a completely different machine on my own network.

None of those were predictable from the datasheet. Which is, I think, the argument for writing the notes down while it is happening rather than afterwards — the failures are the part worth keeping, and they are exactly the part you stop being able to remember once the thing works.