Authoring a Nerves system by hand

I wanted to run Elixir on a CompuLab IOT-DIN-IMX8PLUS — an industrial gateway that lives on a DIN rail in a cabinet, takes 24 V, and is expected to sit there for years without anyone looking at it. There is no published Nerves system for this board. There is one for a sibling board from the same vendor, which is close but not the same hardware.

So the first real decision was what to do about that, and I want to explain it before any of the technical detail, because everything afterwards follows from it.

Fork it, or write it

The fast option is to fork the sibling system and change what looks wrong. The slow option is to author every file myself — nerves_defconfig, the kernel config, the U-Boot config, fwup.conf, the rootfs overlay — using the sibling system as read-only reference and the actual hardware documentation as the source of truth.

I chose the slow option. Not out of principle: a fork hides which values are deliberate and which are inherited. When a board does not boot, that distinction is the whole debugging problem. If I have written each line, then every value is one I can defend or change. If I have forked, then half the file is somebody else's answer to a slightly different question, and I cannot tell which half.

That decision cost me a week and it was correct. Parts two through four of this series are almost entirely me discovering things about my own files. Every one of those discoveries would have been harder to reason about had I inherited the file.

The board

Built around NXP's i.MX8M Plus: four Cortex-A53 cores, a Cortex-M7 for real-time work, and an NPU nobody is using yet. 4 GB of LPDDR4 on the system-on-module. That last number matters much more than it looks like it does — see part four.

Storage is a 29 GB eMMC, and its layout is the part worth internalising early, because the firmware model depends on it:

• a small hardware boot partition (mmcblk2boot0) that the SoC boot ROM reads first, always, no matter what else is on the chip
• a second, identical boot partition as a backup
• the regular user area (mmcblk2), which is where a partition table and filesystems live

Two micro-USB ports, which will cause more trouble than any other single feature of this hardware. One is a serial console (DBG). The other is a USB gadget port (OTG, labelled UART/USB). Part three is largely about them.

Base image, app firmware, runtime data

Nerves splits a device into layers, and being precise about them prevents a lot of confusion later. Two artifacts come out of one mix firmware:

• compulab_cs_gw.fw — the base system rootfs (kernel, device tree, busybox, Erlang, udev rules, erlinit) fused with the application release. This is what OTA updates replace. It goes to the eMMC user area.
• imx-boot — the boot container: SPL, ARM Trusted Firmware, OP-TEE and U-Boot, assembled into one image. This is the only artifact not inside the .fw, and it goes to the eMMC hardware boot partition, separately and once.

The consequence is worth stating plainly because I got it wrong in my own notes at first: fwup never touches the bootloader. It writes the user area only. Flashing imx-boot is a one-time provisioning concern, not something an update does. There is therefore no "enable the boot partition in fwup" work to do, which was an entry on my list for a while before I understood the split.

Pinning, and one config choice that mattered

I pinned the kernel to 6.6.23 and U-Boot to v2023.04, matching the known-good device tree and CompuLab's own iotdin-imx8p_defconfig. Reproducibility on an industrial device is not optional; a build that silently follows a branch tip is a build you cannot recreate in two years when a customer reports something.

For the kernel configuration there were two candidate starting points, and picking the wrong one would have cost me days of driver debugging:

• CompuLab's in-tree compulab_v8_defconfig at the pinned commit
• a fragment from their meta-layer, at 6.6.3

I compared them, which is the only reason I noticed that the meta-layer fragment was missing USB_NET_QMI_WWAN, TI_ADS1015 and RTC_DRV_ABX80X — the LTE modem, the ADC and the real-time clock. Three peripherals that would simply not have existed, and which I would have gone looking for in the device tree.

The in-tree v8 config is the complete one. It already enables SquashFS, overlayfs, ext4, vfat, and every peripheral on the board: FTDI and Option USB serial for the RS485 modules, GPIO_PCA953X for the digital I/O, FlexCAN, the Infineon TPM over I²C, the Marvell SDIO WiFi part.

On top of it I added a nineteen-line Nerves addendum, documented in place:

CONFIG_F2FS_FS=y
CONFIG_USB_WDM=y
CONFIG_SQUASHFS_ZSTD=y

F2FS for the expanding application-data partition, which was genuinely absent. USB_WDM explicitly, because that is what gives you /dev/cdc-wdm0 for the modem's QMI control channel. ZSTD because the Nerves rootfs is a compressed SquashFS.

What I expected to be hard

I wrote down two risks at the start, so that I would be honest with myself later about whether I had predicted correctly.

Risk one: assembling the boot container. The i.MX8M imx-boot image is built by binman from a description inside U-Boot, and I did not know whether that path worked on CompuLab's fork. If it did not, the fallback was NXP's imx-mkimage with an explicit container config.

Risk two: where the U-Boot environment lives. The factory image keeps it in the hardware boot partition. Nerves expects it in the user area, so that fwup and OTA can manage it the standard way. I deliberately diverged and put it in the user area at the same byte offset, 0x3F0000, and wrote in three separate files that this needs confirming at first boot.

Both predictions were wrong in an instructive way. binman worked on the first complete U-Boot build, no intervention needed. The environment location turned out to be a real bug — but a different one than I had written down, and not the reason the board failed to boot.

The thing that actually stopped it was a number in a Kconfig menu I had never looked at.

Before any of that, though, the build had to produce a file I could flash at all, and it took four separate bugs before it did.