Building the Nerves system: the first attempt

What Are We Actually Building?

Before I show you the first build attempt (and its failures), I need to clear up something that confused me at first. When you run mix compile in a Nerves system project, you're not just compiling some Elixir code. You're orchestrating a massive cross-compilation pipeline that downloads and builds an entire Linux distribution from source.

Here's the thing: there is no single "project" we're building. There's a stack of projects, each wrapping the next:

Your Nerves System (phase1_nerves_system/)
  │  mix.exs defines dependencies and config
  │
  ├── nerves (Elixir library)
  │     Orchestrates the build, manages artifacts
  │
  ├── nerves_system_br (Elixir library + Buildroot source)
  │     Contains the actual Buildroot build system (~200MB)
  │     Buildroot is the engine that does the real work:
  │       ├── Downloads Linux kernel 6.6 source from kernel.org
  │       ├── Downloads TF-A source from GitHub
  │       ├── Downloads U-Boot source from denx.de
  │       ├── Downloads Erlang/OTP source
  │       ├── Downloads BusyBox, iproute2, dropbear, etc.
  │       └── Cross-compiles ALL of the above for ARM Cortex-A7
  │
  └── nerves_toolchain_armv7_nerves_linux_gnueabihf (the cross-compiler)
        Uses crosstool-ng to build GCC from source
        The resulting GCC runs on your x86_64 machine
        but produces ARM binaries

When I first ran mix deps.get, I thought "okay, downloading some Elixir libraries." But nerves_system_br isn't just an Elixir library — it's a 200MB package that contains the entire Buildroot build system. And nerves_toolchain_armv7_nerves_linux_gnueabihf isn't a pre-built toolchain — it's a recipe for building GCC from source using crosstool-ng.

So when mix deps.compile runs, the first thing it does is compile a cross-compiler. Not compile your code. Not compile the kernel. It builds the tool that will later be used to build everything else. This alone takes 15-30 minutes.

The actual Nerves system build (kernel, bootloaders, rootfs) happens later, when you run mix compile after all the config files are in place. That build takes another 30-60 minutes and downloads even more source code.

Why Not Just Download a Pre-Built Toolchain?

You might wonder: why build GCC from source? Why not download a pre-compiled ARM toolchain like Linaro or ARM's official one?

Nerves does this for reproducibility. If you use a random toolchain from the internet, the exact versions of GCC, glibc, binutils, and their compile-time options become unknowns. Different toolchains produce subtly different binaries. By building from source with pinned versions, every developer building this Nerves system gets identical results. The toolchain is part of the system's definition, not an external dependency you hope is compatible.

It also means the toolchain version is locked in your mix.exs:

{:nerves_toolchain_armv7_nerves_linux_gnueabihf, "~> 13.2.0", runtime: false}

That 13.2.0 corresponds to GCC 13.2. Changing this version is a deliberate decision, not an accident of whatever happened to be installed on your machine.

The First Error: Missing flex

I created the mix.exs, ran mix deps.get (which downloaded everything successfully), and then ran mix deps.compile. The toolchain build started — crosstool-ng cloned its repository, applied patches, ran ./configure... and then:

checking for flex... no
configure: error: missing required tool: flex

crosstool-ng needs flex (a lexical analyzer generator) and bison (a parser generator) to build the cross-compiler. These are development tools that aren't always installed on desktop Linux systems.

The fix:

sudo apt-get install flex bison

I also learned to install a few more packages that crosstool-ng might need depending on your system:

sudo apt-get install texinfo help2man gawk libtool-bin

These are all build-time dependencies for compiling GCC from source. You'd never need them for normal Elixir development, but when you're building a cross-compilation toolchain, you're essentially building a compiler — and compilers have a lot of dependencies.

The full host dependency list for building a Nerves system from scratch on Ubuntu:

sudo apt-get install git build-essential bc libssl-dev libncurses5-dev \
  unzip wget cpio rsync file python3 cmake u-boot-tools device-tree-compiler \
  squashfs-tools dosfstools mtools gdisk picocom \
  flex bison texinfo help2man gawk libtool-bin

The Second Error: LD_LIBRARY_PATH

After installing flex and bison, I retried. This time crosstool-ng got much further — it generated its configuration, started merging defconfigs... and then:

[ERROR]  Don't set LD_LIBRARY_PATH. It screws up the build.

Well, that's direct.

crosstool-ng absolutely refuses to run when the LD_LIBRARY_PATH environment variable is set. And for good reason: LD_LIBRARY_PATH tells the dynamic linker to look for shared libraries in specific directories before the default system directories. When you're building a cross-compiler, this can cause the build to accidentally link against the wrong libraries — host libraries instead of the ones crosstool-ng built, or vice versa. The result would be a broken toolchain that produces corrupted binaries. So crosstool-ng takes the safe approach: if LD_LIBRARY_PATH is set, refuse to build.

On my system, it was set to:

/usr/local/cuda-12.4/lib64

I have NVIDIA CUDA installed for machine learning work. CUDA's installer adds LD_LIBRARY_PATH to a system-wide profile script (usually something in /etc/profile.d/) so that CUDA libraries are always available. This is fine for CUDA applications but it breaks crosstool-ng.

How to Fix It

For a single command:

env -u LD_LIBRARY_PATH mix deps.compile nerves_toolchain_armv7_nerves_linux_gnueabihf --force

The env -u flag runs the command with a specific variable unset. Your shell's LD_LIBRARY_PATH is unchanged afterward.

For the current terminal session:

unset LD_LIBRARY_PATH
# Now all commands in this terminal run without it
mix deps.compile

This lasts until you close the terminal. Opening a new terminal will have LD_LIBRARY_PATH set again (because the system profile script runs on every new shell).

For this project permanently (using direnv):

If you use direnv (which I recommend for any project that needs specific environment settings), create an .envrc file in the project root:

# .envrc
unset LD_LIBRARY_PATH

Then direnv allow . and every time you cd into the project, LD_LIBRARY_PATH gets unset automatically. When you cd out, it's restored.

What NOT to do: Don't remove the CUDA LD_LIBRARY_PATH from your system profile permanently. You'll break CUDA. The right approach is to unset it specifically for Nerves builds.

Will It Come Back After a Reboot?

Yes. Whatever system profile script sets LD_LIBRARY_PATH for CUDA runs every time you log in or open a new terminal. So after a reboot, you'll need to unset LD_LIBRARY_PATH again before building. This is why direnv is the best solution — it handles it automatically when you're in the project directory.

This is one of those issues that's infuriating when you first hit it. The error message is helpful ("Don't set LD_LIBRARY_PATH") but it doesn't tell you why it's set or how to unset it permanently for just this project. And if you don't have CUDA or similar software installed, you'll never encounter it — which makes it hard to find in troubleshooting guides.

What Happens Next

After clearing these two hurdles, the toolchain build runs for real. crosstool-ng downloads the GCC 13.2 source, binutils, glibc, and the Linux kernel headers, then cross-compiles everything. On my machine this takes about 20 minutes.

Once the toolchain is built, the next step is creating all the configuration files — nerves_defconfig, fwup.conf, kernel config fragment, U-Boot config — and then running the actual system build. That's where Buildroot downloads the Linux kernel, TF-A, and U-Boot source code and cross-compiles them using the toolchain we just built.

Nine minutes later, the toolchain was ready:

[INFO ]  Build completed at 20260313.183025
[INFO ]  (elapsed: 9:07.11)

The Configuration Files — What Each One Does

With the toolchain built, I needed to create the configuration files that tell Buildroot how to build the system. Here's every file I created, what it does, and why it exists.

The Project Directory Structure

phase1_nerves_system/
├── mix.exs                          # Elixir project definition
├── VERSION                          # "0.1.0"
├── nerves_defconfig                 # THE central config — tells Buildroot everything
├── linux-6.6.defconfig              # Kernel driver additions for STM32MP1
├── fwup.conf                        # Partition layout + update rules
├── fwup-revert.conf                 # Slot revert task
├── fwup_include/
│   └── provisioning.conf            # Product name, platform ID
├── post-build.sh                    # Runs after rootfs is assembled
├── post-createfs.sh                 # Runs after rootfs image — calls fwup
├── rootfs_overlay/
│   └── etc/
│       ├── erlinit.config           # Nerves init configuration
│       └── fw_env.config            # U-Boot environment location
└── uboot/
    ├── uboot.fragment               # U-Boot config additions
    └── uEnv.txt                     # Boot commands and kernel parameters

mix.exs — The Elixir Entry Point

This is where it all starts. The mix.exs file declares this project as a Nerves system package:

nerves_package: [
  type: :system,
  platform: Nerves.System.BR,
  platform_config: [defconfig: "nerves_defconfig"]
]

The critical line is platform: Nerves.System.BR — this tells Nerves to use the Buildroot-based build system. When you later run mix compile, the Nerves.System.BR module reads nerves_defconfig and drives the entire Buildroot build.

The env section defines the target architecture:

env: [
  {"TARGET_ARCH", "arm"},
  {"TARGET_CPU", "cortex_a7"},
  {"TARGET_OS", "linux"},
  {"TARGET_ABI", "gnueabihf"}
]

These environment variables are passed to Buildroot and the Nerves toolchain. gnueabihf means "GNU, EABI, hard float" — the ARM calling convention that uses hardware floating point registers.

nerves_defconfig — The Heart of the System

This is the file I spent the most time on. It's a Buildroot defconfig — a list of BR2_* key-value pairs that configure every aspect of the build.

I started by looking at two reference files:

1. Buildroot's upstream configs/stm32mp157c_dk2_defconfig — for the STM32MP1-specific settings (TF-A, U-Boot, kernel DTS name)
2. nerves_system_bbb/nerves_defconfig (BeagleBone Black) — for the Nerves-specific settings (skeleton, overlay, packages)

Then I merged them, adapting each section. The key sections:

Toolchain: Points to our Nerves-built GCC instead of Bootlin's pre-built toolchain. The ${NERVES_TOOLCHAIN} variable is set by the Nerves build system.

TF-A (Trusted Firmware-A): Configures the first-stage bootloader. The DTB_FILE_NAME=stm32mp157c-dk2.dtb must match the kernel's device tree file name — the C variant, not F.

U-Boot: Uses the stm32mp15_trusted defconfig (designed to work with TF-A), not stm32mp15_basic (which uses U-Boot's own SPL and bypasses TF-A).

Kernel: Uses multi_v7_defconfig as the base with our custom fragment on top. The BR2_LINUX_KERNEL_INTREE_DTS_NAME="st/stm32mp157c-dk2" tells Buildroot which device tree to compile.

Root filesystem: SquashFS with LZ4 compression. Read-only by design — this is what enables Nerves' A/B update scheme.

linux-6.6.defconfig — The Kernel Config Fragment

This is not a complete kernel configuration. It's a fragment — a list of options to add or override on top of multi_v7_defconfig. Only STM32MP1-specific drivers and Nerves requirements go here.

Some highlights:

• CONFIG_SERIAL_STM32=y and CONFIG_SERIAL_STM32_CONSOLE=y — without these, no serial console
• CONFIG_STMMAC_ETH=y and CONFIG_DWMAC_STM32=y — the Ethernet driver
• CONFIG_STM32_RPROC=y and CONFIG_RPMSG_CHAR=y — M4 coprocessor support (we enable this now even though we won't use it until Phase 2)
• CONFIG_BRCMFMAC=m — WiFi driver as a module (loaded on demand)
• CONFIG_SQUASHFS=y — required for Nerves' root filesystem format

fwup.conf — The Partition Map

This defines how the microSD card is partitioned and how firmware updates work. I covered the partition layout in detail in Part 2, but the key insight for reproducibility: the partition offsets are in 512-byte blocks, and they must align precisely. The ROM code scans the GPT for partitions named fsbl1 and fsbl2 — if these offsets are wrong or the names don't match, the board won't boot.

The three tasks (complete, upgrade.a, upgrade.b) define what happens during different flash scenarios. Remember the lesson from Part 2: use -t complete once, then -t upgrade.a/b for everything after.

erlinit.config — Nerves Init

The -c ttySTM0 line tells erlinit to attach the IEx console to UART4 (the ST-LINK virtual COM port). If this is wrong, you'll see the kernel boot on serial but then... silence. The BEAM starts, but IEx is attached to a UART that doesn't exist or isn't connected.

The --hang-on-exit flag keeps the console accessible when the BEAM crashes during development. In production, you'd remove this so erlinit auto-restarts the VM.

post-build.sh and post-createfs.sh — Build Pipeline Scripts

These run during the Buildroot build process:

• post-build.sh runs after Buildroot assembles the root filesystem directory but before it's packaged into an image. We use it to copy uEnv.txt into /boot/.
• post-createfs.sh runs after the rootfs image is created. It calls the nerves-common script that invokes fwup to package everything into the final .fw firmware file.

Both must be executable (chmod +x). Forgetting this is a common build failure.

Ready for the First Build

All files are in place. The moment of truth. From the phase1_nerves_system directory:

unset LD_LIBRARY_PATH
mix compile

This will trigger Buildroot, which will:

1. Download Linux kernel 6.6.80 source (~150MB)
2. Download TF-A v2.10.0 source
3. Download U-Boot 2024.01 source
4. Download Erlang/OTP, BusyBox, and other packages
5. Cross-compile everything using our ARM toolchain
6. Build the root filesystem (SquashFS)
7. Run fwup to create the final .fw firmware file

Expected time: 30-60 minutes for the first build. Subsequent builds are faster because Buildroot caches downloaded sources and intermediate build artifacts.

Will it work on the first try? Almost certainly not. Let's see what breaks.

The Third Error: rng-tools Notice

The build got surprisingly far. Buildroot's configuration system (conf) compiled, the .config was generated, and the build directory was created. Then:

** (Mix) Nerves encountered an error while constructing the artifact
nerves-config.mk:29: *** "The BR2_PACKAGE_RNG_TOOLS option is no longer a forced
dependency in Nerves. Linux kernels >= 5.6 may not need it. Please add
BR2_PACKAGE_NERVES_CONFIG_ACCEPT_RNG_NOTICE=y to your nerves_defconfig to indicate
that you've seen this message.".  Stop.

This one is interesting because it's not really an error — it's a policy change in nerves_system_br. Here's the story:

What Is rng-tools?

rng-tools is a daemon that feeds entropy from hardware random number generators into the Linux kernel's entropy pool. The kernel needs entropy (randomness) for cryptographic operations — SSH keys, TLS handshakes, anything that calls /dev/random or /dev/urandom.

Older Linux kernels (before 5.6) had a problem: at boot, the entropy pool was empty. The kernel would block programs that needed cryptographic randomness until enough entropy accumulated. This could cause the system to hang at boot for 30-60 seconds waiting for randomness, which is terrible for embedded systems that need to boot fast.

The solution was rng-tools: it reads from hardware RNG devices (like the STM32's built-in TRNG) and feeds that entropy to the kernel immediately at boot. Nerves used to force this as a dependency because without it, early network operations (like SSH) would stall.

Why It's No Longer Required

Starting with Linux 5.6, the kernel added RANDOM_TRUST_CPU — a config option that tells the kernel to trust the CPU's built-in random number generator (like x86's RDRAND or ARM's equivalent) for initial entropy seeding. With this enabled, the entropy pool is populated almost immediately at boot without needing a userspace daemon.

Since we're using kernel 6.6 (well past 5.6), the kernel handles entropy on its own. The rng-tools daemon is unnecessary overhead.

The Fix

Add this to nerves_defconfig, right after BR2_PACKAGE_NERVES_CONFIG=y:

# Acknowledge that rng-tools is no longer a forced dependency in Nerves.
# Linux kernels >= 5.6 have built-in entropy sources (RANDOM_TRUST_CPU),
# so the userspace rng-tools daemon is typically unnecessary.
BR2_PACKAGE_NERVES_CONFIG_ACCEPT_RNG_NOTICE=y

This is a pattern you'll see in Buildroot and Nerves: when a default changes, instead of silently doing the new thing, the build system forces you to acknowledge the change. It's annoying in the moment but it prevents subtle breakage — imagine if rng-tools was silently removed and your device started hanging at boot because your kernel didn't have RANDOM_TRUST_CPU enabled.

The Fourth Error: Cannot execute cross-compiler

With rng-tools acknowledged, the build got further — skeleton extracted, host tools built (fakeroot, makedevs), custom skeleton installed. Then:

>>> toolchain-external-custom  Configuring
Cannot execute cross-compiler '/armv7-nerves-linux-gnueabihf-gcc'

Look at that path carefully: /armv7-nerves-linux-gnueabihf-gcc. That's not a real path — there's no directory before the binary name, just a lone /. Something is terribly wrong with how Buildroot is finding the toolchain.

How I Debugged This

My first instinct was to look at what I'd been using as a reference. The BeagleBone Black's nerves_defconfig has this line:

BR2_TOOLCHAIN_EXTERNAL_PATH="${NERVES_TOOLCHAIN}"

And our defconfig has the identical line. ${NERVES_TOOLCHAIN} is a shell-style variable reference that Buildroot stores literally in its .config file. When make runs, GNU Make imports environment variables as Make variables, so ${NERVES_TOOLCHAIN} expands to whatever the NERVES_TOOLCHAIN environment variable is set to.

So where is NERVES_TOOLCHAIN supposed to come from?

Down the Rabbit Hole: Tracing the Nerves Build System

I spent quite a while tracing through the Nerves source code to understand this. Here's the chain:

1. mix compile triggers the :nerves_package compiler
2. Compile.NervesPackage calls Nerves.Artifact.build(package, toolchain)
3. This calls Nerves.System.BR.build(pkg, toolchain, opts)
4. Which calls make(:linux, pkg, _toolchain, opts) — note the underscore: it ignores the toolchain argument!
5. It then runs shell("make", ...) which inherits environment variables from the Elixir process

The critical insight: Nerves.System.BR.make doesn't set NERVES_TOOLCHAIN — it expects it to already be in the environment. For application projects (when you're building firmware, not the system itself), Nerves.Env.bootstrap() sets it. But when you're building the system itself, bootstrap hasn't run yet.

Official Nerves systems (like BBB) sidestep this entirely: they ship pre-built artifacts. When you mix deps.get nerves_system_bbb, you download a tarball — the system is never built from source on your machine. Only system maintainers trigger the actual Buildroot build, and they set up the environment explicitly.

When you're creating a custom system from scratch (like we are), there are no pre-built artifacts. The build happens locally. And nobody sets NERVES_TOOLCHAIN for you.

Where Is the Toolchain?

The toolchain was built by crosstool-ng earlier (the 9-minute build). It's cached as a symlink:

~/.nerves/artifacts/nerves_toolchain_armv7_nerves_linux_gnueabihf-linux_x86_64-13.2.0
  → .../x-tools/armv7-nerves-linux-gnueabihf/

Inside that directory, there's a bin/ folder with our cross-compiler:

bin/armv7-nerves-linux-gnueabihf-gcc
bin/armv7-nerves-linux-gnueabihf-g++
bin/armv7-nerves-linux-gnueabihf-ld
... etc

This is exactly what BR2_TOOLCHAIN_EXTERNAL_PATH needs to point to.

The Fix

Set NERVES_TOOLCHAIN before building:

export NERVES_TOOLCHAIN=$HOME/.nerves/artifacts/nerves_toolchain_armv7_nerves_linux_gnueabihf-linux_x86_64-13.2.0
unset LD_LIBRARY_PATH
mix compile

To find the exact path on your system, look in ~/.nerves/artifacts/ for a directory matching your toolchain name and version.

Making It Permanent with direnv

At this point I updated my .envrc file to handle both the CUDA and toolchain issues:

# .envrc for Nerves system builds
# Unset LD_LIBRARY_PATH — CUDA sets it system-wide, but crosstool-ng refuses to build with it
unset LD_LIBRARY_PATH

# Set NERVES_TOOLCHAIN — needed when building custom systems from source
# The path comes from ~/.nerves/artifacts/ after the toolchain is built
export NERVES_TOOLCHAIN=$HOME/.nerves/artifacts/nerves_toolchain_armv7_nerves_linux_gnueabihf-linux_x86_64-13.2.0

Then direnv allow . and it's handled automatically.

Why This Is Easy to Miss

This issue only affects people building custom Nerves systems from scratch. If you're using an official system (BBB, RPi, etc.), you download pre-built artifacts and never trigger a Buildroot build. The NERVES_TOOLCHAIN variable gets set by Nerves.Env.bootstrap() when compiling your application firmware, but by then the system is already built.

The Nerves documentation mentions setting NERVES_TOOLCHAIN when using mix nerves.system.shell, but not when doing a plain mix compile in a system project. It's one of those things that's obvious once you understand the architecture but completely opaque when you're just following tutorials.

The Fifth Error (and Sixth): Toolchain Property Mismatches

With the toolchain path fixed, Buildroot now finds the cross-compiler. It starts configuring the external toolchain, and then:

Incorrect selection of kernel headers: expected 6.1.x, got 5.4.x

And after fixing that, immediately:

Incorrect selection of gcc version: expected 15.x, got 13.2.0

Two errors, same root cause. To understand why these happen, you need to understand how Buildroot, the Nerves toolchain, and your defconfig relate to each other.

The Three-Way Relationship: Buildroot, Toolchain, and Defconfig

Here's the fundamental thing that confused me: the Nerves toolchain and Buildroot are independently built, but they must agree on certain properties for the final system to work. Your nerves_defconfig is the place where you declare this agreement.

The timeline goes like this:

1. Nerves toolchain is built first (by crosstool-ng, which is completely separate from Buildroot). When building it, the Nerves team chose specific versions: GCC 13.2, glibc, kernel headers 5.4, etc. These choices are frozen into the toolchain binary.

2. Buildroot runs second, using the already-built toolchain as an "external toolchain." Buildroot needs to know the properties of this toolchain — not to build it, but to configure itself correctly. For example, Buildroot needs to know the kernel headers version so it can set up the right glibc compatibility, enable the right features, and avoid generating code that uses interfaces the toolchain's headers don't know about.

3. Your nerves_defconfig is where you tell Buildroot: "the external toolchain I'm giving you has these properties." If you lie — if you say the headers are 6.1 but they're actually 5.4 — Buildroot catches the mismatch and refuses to continue. This is a safety check, not bureaucracy.

Think of it like a job interview. The toolchain is the candidate. Buildroot is the employer. The defconfig is the candidate's resume. If the resume says "10 years of Go experience" but the interview reveals the candidate has only done Python, the employer stops the process. That's what Buildroot is doing here.

Why Buildroot Cares About Toolchain Properties

Buildroot isn't just being pedantic. It needs accurate toolchain information because:

Kernel headers version determines which system calls and kernel data structures are available to userspace code. If Buildroot thinks the toolchain has 6.1 headers, it might enable features that rely on interfaces introduced between 5.4 and 6.1. The compiled binaries would reference system call numbers or structure fields that don't exist in the toolchain's headers, causing either build failures or — worse — subtle runtime bugs.

GCC version affects which compiler flags, optimizations, and language features Buildroot can use. GCC 15 supports different -march options, different sanitizers, and different warning flags than GCC 13. Buildroot adapts its build commands based on the GCC version.

glibc vs musl vs uclibc changes the entire C library compatibility landscape. Getting this wrong would produce binaries linked against the wrong libc.

All of these BR2_TOOLCHAIN_EXTERNAL_* settings are essentially Buildroot asking: "Tell me exactly what I'm working with so I don't make assumptions."

What Are Kernel Headers, Actually?

When the cross-compilation toolchain was originally built by crosstool-ng, one of the inputs was a set of Linux kernel headers. These aren't the kernel source code — they're a sanitized subset of kernel header files that define the userspace ABI (Application Binary Interface):

• System call numbers (e.g., __NR_openat = 322 on ARM)
• ioctl command constants
• Data structures shared between kernel and userspace (e.g., struct sockaddr, struct stat)
• Constants for file flags, socket options, signal numbers

These headers get installed into the toolchain's sysroot at usr/include/linux/ and usr/include/asm/. Every program compiled with this toolchain — from glibc itself to BusyBox to Erlang — includes these headers when it needs to talk to the kernel.

How to Find the Actual Kernel Headers Version

I checked the toolchain's kernel headers by reading version.h from the toolchain's sysroot:

cat ~/.nerves/artifacts/nerves_toolchain_armv7_nerves_linux_gnueabihf-linux_x86_64-13.2.0/\
  armv7-nerves-linux-gnueabihf/sysroot/usr/include/linux/version.h

Output:

#define LINUX_VERSION_CODE 328959

The version code is encoded as (major << 16) + (minor << 8) + patch. Decoding:

v = 328959
major = v >> 16          # 5
minor = (v >> 8) & 0xff  # 4
patch = v & 0xff          # 255
# → kernel headers 5.4.x

My defconfig had BR2_TOOLCHAIN_EXTERNAL_HEADERS_6_1=y because I'd guessed based on the fact that we're building kernel 6.6. Wrong guess.

"But Wait — Aren't We Building Kernel 6.6?"

When I first saw this error, I panicked. Are we using an old kernel? Did something go wrong?

No. The kernel headers version in the toolchain and the kernel version we're building for the board are two completely different things.

Here's the distinction:

• Kernel headers in the toolchain (5.4): These were baked into the cross-compiler when crosstool-ng built it. They define the userspace ABI — the contract between compiled programs and the kernel. "Programs compiled with this toolchain only rely on kernel interfaces that existed as of version 5.4."

• Kernel running on the board (6.6): This is the actual Linux kernel that Buildroot will cross-compile and put on the microSD card. It's a completely separate compilation step. Version 6.6 supports everything from 5.4 and adds new features on top.

The Linux kernel guarantees backward compatibility with older userspace interfaces. A program compiled against 5.4 headers will work perfectly on a 6.6 kernel — the kernel never removes or changes existing system call interfaces. That's Linus Torvalds' famous "we don't break userspace" rule, and it's one of the most strictly enforced policies in all of open source.

What you can't do is the reverse: if the toolchain had 6.6 headers, a program might use a system call that was introduced in 6.1, and that would fail on a 5.4 kernel. The rule is simple: toolchain headers must be ≤ running kernel version. Our setup (5.4 headers, 6.6 kernel) satisfies this comfortably.

The Nerves toolchain uses intentionally conservative kernel headers. This means the same toolchain can target boards running various kernel versions without compatibility issues. It's a deliberate design choice, not a bug.

The GCC Version: Same Story

After fixing the kernel headers, the next build attempt produced:

Incorrect selection of gcc version: expected 15.x, got 13.2.0

I hadn't specified BR2_TOOLCHAIN_EXTERNAL_GCC_13=y in the defconfig. Without an explicit setting, Buildroot 2025.11.2 defaults to the newest GCC it knows about — version 15. But our Nerves toolchain nerves_toolchain_armv7_nerves_linux_gnueabihf version 13.2.0 was built with — you guessed it — GCC 13.2.

The version number is right there in the Hex package name (~> 13.2.0 in mix.exs), but I hadn't connected the dots that this needed to be declared in the Buildroot defconfig too.

The Complete Fix

In nerves_defconfig, the toolchain section needs all three properties to match reality:

BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="armv7-nerves-linux-gnueabihf"
BR2_TOOLCHAIN_EXTERNAL_HEADERS_5_4=y    # ← was 6_1, fixed to match toolchain
BR2_TOOLCHAIN_EXTERNAL_GCC_13=y          # ← was missing, Buildroot defaulted to 15
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_TOOLCHAIN_EXTERNAL_PATH="${NERVES_TOOLCHAIN}"

Lesson: The Defconfig Is a Contract, Not a Wishlist

Every BR2_TOOLCHAIN_EXTERNAL_* setting must match the actual toolchain binary sitting in ~/.nerves/artifacts/. These aren't configuration choices you make — they're declarations of fact about a toolchain that already exists.

To figure out the right values:

• GCC version: look at the Nerves toolchain Hex package name (13.2.0 → GCC 13)
• Kernel headers: read version.h from the toolchain's sysroot (see the decoding trick above)
• C library: Nerves uses glibc (not musl or uclibc)
• C++ support: Nerves toolchains include C++ support

If you leave a setting unspecified, Buildroot picks whatever default the current Buildroot version prefers — which is almost certainly wrong for a pre-built toolchain from Nerves. Always be explicit.

The Seventh and Eighth Errors: Fortran and OpenMP

After fixing the kernel headers and GCC version, I expected smooth sailing. Instead, Buildroot's toolchain validation kept going:

Fortran support is not selected but is available in external toolchain

And right after fixing that:

OpenMP support is not selected but is available in external toolchain

Same pattern as before — Buildroot inspects the actual toolchain binaries and compares them to what the defconfig declares. The Nerves toolchain includes both Fortran (libgfortran) and OpenMP (libgomp) support because they come "for free" when building GCC from source — the Nerves team enables them since they don't add meaningful overhead but they're useful for scientific computing packages.

My first reaction was "I need Fortran for an embedded IoT gateway?" The answer is no — you don't need Fortran. But the toolchain has Fortran support compiled in, and Buildroot needs to know that. These settings are declarations of fact about the toolchain, not features you're choosing to use.

The Fix

Add both to nerves_defconfig:

BR2_TOOLCHAIN_EXTERNAL_FORTRAN=y
BR2_TOOLCHAIN_EXTERNAL_OPENMP=y

At this point I went and proactively checked what else the toolchain had. I looked for additional shared libraries in the toolchain's sysroot:

ls ~/.nerves/artifacts/nerves_toolchain_armv7_nerves_linux_gnueabihf-linux_x86_64-13.2.0/\
  armv7-nerves-linux-gnueabihf/sysroot/lib/

I found libgfortran.so (Fortran) and libgomp.so (OpenMP), which confirmed those two. I did not find libssp.so (stack smashing protection), libasan.so (address sanitizer), or libubsan.so (undefined behavior sanitizer) — so no additional declarations were needed. Proactively checking saved me from more round-trips through the 30-minute build.

The Ninth Error: RPC Support — A Lesson in Kconfig Syntax

Next up:

RPC support not available in C library, please disable BR2_TOOLCHAIN_EXTERNAL_INET_RPC

This one is about Sun RPC (Remote Procedure Call) support. Modern glibc (2.32+) removed built-in Sun RPC support, moving it to a separate library called libtirpc. Buildroot defaults to assuming the toolchain's glibc has RPC support, but ours doesn't.

I needed to explicitly disable the BR2_TOOLCHAIN_EXTERNAL_INET_RPC option. And here's where I learned something the hard way about Kconfig syntax.

My First Attempt (Wrong)

# is not set BR2_TOOLCHAIN_EXTERNAL_INET_RPC

Build failed with the same error. Why?

Kconfig Syntax: The Comment That Isn't a Comment

In Kconfig defconfig files, there's a very specific syntax for explicitly disabling an option:

# BR2_SOME_OPTION is not set

This looks like a comment. It starts with #. But it's actually a Kconfig directive — a machine-readable instruction that means "this option should be explicitly disabled." The parser looks for exactly this pattern: # , then the option name, then is not set.

My mistake — # is not set BR2_TOOLCHAIN_EXTERNAL_INET_RPC — had the words in the wrong order. To Kconfig, that really was just a comment. It was ignored completely, and the option defaulted to "enabled."

The Correct Fix

# BR2_TOOLCHAIN_EXTERNAL_INET_RPC is not set

The order matters. The option name comes right after #, followed by is not set. This is one of those syntax rules that bites everyone exactly once. I've seen it trip up Linux kernel developers too — it's just unusual enough that you don't remember the exact format until you've gotten it wrong.

Why Not Just Omit It?

You might wonder: if the default is "enabled" and I want "disabled," why can't I just add BR2_TOOLCHAIN_EXTERNAL_INET_RPC=n?

In Kconfig, =n doesn't always work as expected in defconfig files. The canonical way to disable a boolean option is the # ... is not set syntax. Using =n might work in some contexts but it's not guaranteed. The # ... is not set syntax is what make savedefconfig generates, and it's what the parser reliably handles.

The Tenth Error: Missing swig

The build got past all the toolchain validation and started compiling real code. TF-A built successfully. Then U-Boot started building and:

error: command 'swig' failed: No such file or directory

U-Boot uses SWIG (Simplified Wrapper and Interface Generator) to create Python bindings for libfdt, the flattened device tree library. These bindings are used by U-Boot's build system to manipulate device tree files during compilation.

The Fix

sudo apt-get install swig

This is a host dependency that isn't always obvious. You wouldn't think a bootloader needs a Python binding generator, but U-Boot's build system is complex and uses Python scripts extensively for device tree processing. I added swig to the host dependency list in the prerequisites section.

The Eleventh Error: The ext-toolchain Symlink

After installing swig, the build progressed much further — TF-A compiled, U-Boot compiled, the Linux kernel compiled (this took a while — building a full ARM kernel with multi_v7_defconfig). Then, during the nerves-config package build:

cp: cannot create regular file '.../host/opt/ext-toolchain/bin/echo-gcc-args': No such file or directory

This is a fascinating bug that reveals a mismatch between how nerves_system_br expects the toolchain to be set up and how we're actually providing it.

What's Happening

The file deps/nerves_system_br/package/nerves-config/nerves-config.mk contains a hardcoded path:

$(HOST_DIR)/opt/ext-toolchain/bin/echo-gcc-args

This path (host/opt/ext-toolchain/) is where Buildroot puts external toolchains when it downloads them (using BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y). Buildroot downloads the toolchain tarball, extracts it, and symlinks or copies it to $(HOST_DIR)/opt/ext-toolchain/.

But we're using BR2_TOOLCHAIN_EXTERNAL_PATH="${NERVES_TOOLCHAIN}" — we're pointing Buildroot to a toolchain that already exists on disk. In this case, Buildroot uses the toolchain directly from its installed location and never creates the host/opt/ext-toolchain/ directory.

The nerves-config package doesn't account for this. It assumes the directory exists.

The Fix

Create a symlink that bridges the gap:

# Find the Buildroot output directory
ARTIFACT_DIR=~/.nerves/artifacts/nerves_system_stm32mp157f_dk2-portable-0.1.0

# Create the expected directory structure
mkdir -p "$ARTIFACT_DIR/host/opt"

# Symlink to the actual toolchain
ln -sf $HOME/.nerves/artifacts/nerves_toolchain_armv7_nerves_linux_gnueabihf-linux_x86_64-13.2.0 \
  "$ARTIFACT_DIR/host/opt/ext-toolchain"

This creates the directory path that nerves-config.mk expects, pointing it back to the real toolchain. It's a workaround for a quirk in how nerves_system_br handles pre-installed toolchains.

Why This Is Worth Noting

This bug only appears when building a custom Nerves system from source with a local toolchain. Official Nerves systems ship pre-built artifacts, so this code path is rarely exercised. The nerves-config.mk file makes a reasonable assumption — that the toolchain lives in the standard Buildroot location — but that assumption breaks for custom builds.

If you're maintaining a custom Nerves system long-term, you'll want to either:

1. Keep this symlink creation as part of your build script
2. Submit a fix upstream to nerves_system_br that handles both toolchain locations

The Twelfth Error: post-createfs.sh — Victory Snatched at the Finish Line

The build ran for another 20 minutes. I watched TF-A compile, U-Boot compile, the Linux kernel compile (all 7000+ source files for multi_v7_defconfig), Erlang/OTP cross-compile, BusyBox build, erlinit, nbtty, nerves_heart, rootfs overlay installation, and finally SquashFS image creation (29.47 MB, LZ4 compressed). The entire root filesystem was built. Then:

>>>   Executing post-image script .../phase1_nerves_system/post-createfs.sh
Error: /home/tomaz/razvoj/elixir_projects/nerves/stm32-moj/phase1_nerves_system not found

Everything compiled. The complete Linux system — kernel, bootloaders, root filesystem — was built and sitting in the images directory. And then the very last step, packaging it into a .fw firmware file, failed because of how post-createfs.sh passes arguments to the nerves-common script.

What Went Wrong

Our post-createfs.sh was:

#!/bin/sh
set -e
NERVES_DEFCONFIG_DIR="$1"
. "$BR2_EXTERNAL_NERVES_PATH/board/nerves-common/post-createfs.sh" "$NERVES_DEFCONFIG_DIR"

The nerves-common script expects two arguments:

• $1 = the images directory (where to put the .fw output)
• $2 = the path to fwup.conf (a file)

But here's how Buildroot calls post-image scripts. Looking at the Buildroot Makefile:

$(EXTRA_ENV) $(s) \
    $(BINARIES_DIR) \
    $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))

So our script receives:

• $1 = BINARIES_DIR (the images directory)
• $2 = NERVES_DEFCONFIG_DIR (from BR2_ROOTFS_POST_SCRIPT_ARGS)

Our script then used . (source) to include the nerves-common script. In /bin/sh (dash on Ubuntu), when you source a script, the positional parameters of the parent script bleed through. So the nerves-common script inherited:

• $1 = BINARIES_DIR (correct — it's the images dir)
• $2 = /home/tomaz/.../phase1_nerves_system (wrong — it's the directory, not the fwup.conf file)

The check [ ! -f "$FWUP_CONFIG" ] correctly reported "not found" — because a directory is not a file.

The Fix

Rewrite post-createfs.sh to explicitly construct the correct arguments:

#!/bin/sh
set -e

# Buildroot post-image scripts receive:
#   $1 = BINARIES_DIR (the images directory)
#   $2 = BR2_ROOTFS_POST_SCRIPT_ARGS (= NERVES_DEFCONFIG_DIR)
#
# The nerves-common post-createfs.sh expects:
#   $1 = images directory
#   $2 = path to fwup.conf FILE

FWUP_CONFIG="$2/fwup.conf"

"$BR2_EXTERNAL_NERVES_PATH/board/nerves-common/post-createfs.sh" "$1" "$FWUP_CONFIG"

The key changes:

1. Construct the full path to fwup.conf by appending it to the defconfig directory
2. Execute the nerves-common script (not source it) to avoid positional parameter inheritance issues
3. Pass $1 (BINARIES_DIR) as the images directory and the constructed fwup.conf path as the second argument

The Lesson: Shell Sourcing Is Subtle

The . (source) command in shell scripting is deceptively different from executing a script. When you execute a script, it gets its own process with its own positional parameters. When you source a script, it runs in the current shell's context — and the behavior of positional parameters depends on which shell you're using (bash vs dash vs sh). The original script was fragile because it relied on this behavior.

The Complete .envrc File

After all these errors, the .envrc file that makes everything work:

# .envrc for Nerves system builds
#
# Unset LD_LIBRARY_PATH — CUDA sets it system-wide via /etc/profile.d/,
# but crosstool-ng refuses to build with it set because it can cause
# the toolchain build to link against wrong libraries.
unset LD_LIBRARY_PATH

# Set NERVES_TOOLCHAIN — needed when building custom systems from source.
# Official systems ship pre-built artifacts so this is never needed for them.
# The path comes from ~/.nerves/artifacts/ after the toolchain is built.
export NERVES_TOOLCHAIN=$HOME/.nerves/artifacts/nerves_toolchain_armv7_nerves_linux_gnueabihf-linux_x86_64-13.2.0

The Final nerves_defconfig Toolchain Section

For reference, here's the complete toolchain section after all fixes:

BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="armv7-nerves-linux-gnueabihf"
BR2_TOOLCHAIN_EXTERNAL_HEADERS_5_4=y
BR2_TOOLCHAIN_EXTERNAL_GCC_13=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_TOOLCHAIN_EXTERNAL_FORTRAN=y
BR2_TOOLCHAIN_EXTERNAL_OPENMP=y
# BR2_TOOLCHAIN_EXTERNAL_INET_RPC is not set
BR2_TOOLCHAIN_EXTERNAL_PATH="${NERVES_TOOLCHAIN}"

Every single BR2_TOOLCHAIN_EXTERNAL_* line was earned through a build failure. The original defconfig had only four of these. By the end, there are eleven. Each one represents a property of the toolchain that Buildroot needs to know about — and each one that was missing caused a build failure with a clear error message pointing to exactly what was wrong.

Summary: All Build Errors and Fixes

Here's the complete list, in order. If you're following along and building this system, you can apply all these fixes upfront and skip the pain:

That's eleven errors across five builds (some required full rebuilds, others just a re-run). Total wall-clock time from first mix compile to successful build: about 4 hours, mostly waiting for Buildroot to compile.

Was it worth it? Absolutely. Each error taught me something about how Buildroot, Nerves, and the toolchain interact. By the end, I understand the build system well enough to debug future issues confidently. And you, dear reader, can now apply all eleven fixes upfront and get a clean build on the first try.

Reality Check: What Did We Actually Build?

Let me pause and take stock. After hours of fighting build errors, what do we actually have? It's easy to lose sight of the big picture when you're knee-deep in Kconfig syntax and toolchain symlinks.

What's On Disk

The build produced a 6 GB artifact directory at ~/.nerves/artifacts/nerves_system_stm32mp157f_dk2-portable-0.1.0/. Here's what's inside:

~/.nerves/artifacts/nerves_system_stm32mp157f_dk2-portable-0.1.0/
├── build/      (5.3 GB)  ← All the compiled source code. Throwaway.
├── host/       (526 MB)  ← Cross-compiler, host tools (fwup, mksquashfs, etc.)
├── staging/    (265 MB)  ← Sysroot: headers + libraries for cross-compiling Elixir NIFs
├── target/     (55 MB)   ← The root filesystem tree (before SquashFS compression)
├── images/     (98 MB)   ← The final outputs:
│   ├── tf-a-stm32mp157c-dk2.stm32   (209 KB)  ← First-stage bootloader
│   ├── fip.bin                        (1.2 MB)  ← FIP: sp_min + U-Boot
│   ├── u-boot-nodtb.bin + u-boot.dtb            ← U-Boot pieces
│   ├── zImage                         (11.4 MB) ← Linux kernel 6.6.80
│   ├── stm32mp157c-dk2.dtb           (68 KB)   ← Device tree blob
│   ├── rootfs.squashfs                (30.9 MB) ← Compressed root filesystem
│   ├── fwup.conf + fwup-revert.conf             ← Firmware update configs
│   └── rootfs.tar                     (56 MB)   ← Uncompressed rootfs (for debugging)
├── nerves-env.sh                     ← Shell script to set up cross-compilation env
├── .config                           ← The complete Buildroot configuration
└── staging -> host/.../sysroot       ← Symlink to the cross-compilation sysroot

It's Not Firmware Yet — It's a Platform SDK

Here's the thing that surprised me: we haven't built something we can flash to the board yet. What we built is a Nerves system — think of it as a platform SDK. It contains everything needed to build firmware, but it's not firmware itself.

The distinction matters:

• Nerves system (what we built): Linux kernel + bootloaders + root filesystem + cross-compilation tools. It's a platform that Elixir applications run on top of. It's like an OS — you don't flash an OS alone, you flash an OS with an application installed.

• Nerves firmware (what we need next): A .fw file that combines the system with an Elixir application. This is what you actually flash to the microSD card. To create firmware, you need a separate Elixir project that depends on this system.

The analogy: building a Nerves system is like building a custom Raspberry Pi OS image. Building firmware is like installing your application on that OS and shipping the whole thing.

Is This Buildroot? Nerves? Both?

Both. Here's the layer cake:

1. Buildroot is the build engine. It downloaded and cross-compiled Linux 6.6.80, TF-A v2.10.0, U-Boot 2024.01, Erlang/OTP, BusyBox, and dozens of other packages for ARM Cortex-A7. The 5.3 GB build/ directory is pure Buildroot output.

2. nerves_system_br is the glue. It's an Elixir package that bundles Buildroot and adds Nerves-specific packages (erlinit, nbtty, nerves_heart, nerves_config) plus a custom root filesystem skeleton. It drives Buildroot from the Elixir/Mix build system.

3. Our custom system (nerves_system_stm32mp157f_dk2) is the configuration layer. The nerves_defconfig file, kernel config fragment, U-Boot fragment, fwup.conf, and rootfs overlay — these tell Buildroot what to build for our specific board.

So: Buildroot does the heavy lifting. nerves_system_br integrates it into the Elixir ecosystem. Our system tells it what to build. The result is a Nerves-flavored Linux distribution, cross-compiled for the STM32MP157F-DK2.

Can I Move This to Another Machine?

This is the practical question. The answer has two parts:

The source (portable): The phase1_nerves_system/ directory — mix.exs, nerves_defconfig, kernel config, fwup.conf, scripts, overlays — is everything you need to reproduce the build. Push this to a Git repository and anyone with the right host dependencies can mix deps.get && mix compile to rebuild everything from scratch. It'll take 30-60 minutes, but it works.

The built artifact (also portable, with one command): You can create a portable tarball of the built system:

cd phase1_nerves_system
mix nerves.artifact

This runs mksystem.sh, which packages the essential outputs (images, staging, config — about 350 MB compressed) into a tarball. The 5.3 GB build/ directory is not included — it's only needed during compilation.

This tarball is exactly what official Nerves systems ship. When you mix deps.get a system like nerves_system_rpi4, you download this same kind of tarball from GitHub Releases. You never build it from source.

How Official Nerves Systems Handle Distribution

Here's the workflow that the Nerves core team uses:

1. Source code lives in a GitHub repo (e.g., nerves-project/nerves_system_rpi4)
2. CI builds the system from source (just like we did, but on GitHub Actions)
3. mix nerves.artifact creates the portable tarball
4. The tarball is uploaded to GitHub Releases (tagged with the version)
5. mix.exs has artifact_sites: [{:github_releases, "nerves-project/nerves_system_rpi4"}] — this tells Nerves where to download pre-built artifacts

Our mix.exs already has this configured:

artifact_sites: [
  {:github_releases, "tomazbracic/nerves_system_stm32mp157f_dk2"}
]

So if we push the source to that GitHub repo, run mix nerves.artifact to create the tarball, and upload it as a GitHub Release, then anyone (including you at work) can use this system by adding it as a dependency:

{:nerves_system_stm32mp157f_dk2, "~> 0.1", runtime: false, targets: :stm32mp157f_dk2}

They'll never need to build from source. mix deps.get will download the pre-built tarball.

How To Use This At Work

You have several options, from simplest to most robust:

Option 1: Copy the artifact tarball (quick and dirty)

# On your home PC:
cd phase1_nerves_system
mix nerves.artifact --path /tmp
# Creates: /tmp/nerves_system_stm32mp157f_dk2-portable-0.1.0-CHECKSUM.tar.gz

# Copy that file to your work machine, place it in ~/.nerves/dl/
# Nerves will find it there and skip downloading

Option 2: GitHub Releases (the right way)

1. Push the system source to GitHub
2. Create the artifact: mix nerves.artifact
3. Create a GitHub Release tagged v0.1.0
4. Upload the tarball to the release
5. At work, just add it as a Mix dependency — Nerves downloads it automatically

Option 3: Rebuild from source at work

Just clone the repo and mix compile. Same 30-60 minute build, but fully reproducible. You need the same host dependencies installed.

Option 4: Private artifact hosting

If you can't use public GitHub, Nerves also supports {:prefix, "https://your-company-server/artifacts/"} as an artifact site. Upload the tarball to any HTTP server and point to it.

What Happens Next

To actually boot the board, we need to:

1. Create a Nerves application project (a normal Elixir project that depends on this system)
2. Build firmware: mix firmware — this packages the system + application into a .fw file
3. Flash the microSD: mix firmware.burn or fwup nerves_system_stm32mp157f_dk2.fw
4. Insert the card, power up, watch serial console

The system build was the hard part. Building firmware from a working system is fast (seconds, not hours) and rarely fails.

From System to Firmware: The Last Mile

With the system built, I expected "the hard part is over." It mostly was — but there were a few more lessons to learn.

What's a System vs Firmware?

This distinction kept confusing me, so let me be very explicit:

• System = the platform (Linux kernel + bootloaders + root filesystem + cross-compilation sysroot). This is what Buildroot produces. Think of it as "the operating system." Takes 30-60 minutes to build.
• Firmware = the system + your Elixir application, packaged together into a single .fw file that you flash to the microSD card. Takes seconds to build.

You can't flash the system alone. It's like having an OS installer but no hard drive to install it to. The firmware is the complete, flashable package.

To create firmware, I needed a separate Elixir project — phase1_test_firmware/ — that depends on the system:

# In phase1_test_firmware/mix.exs
{:nerves_system_stm32mp157f_dk2,
 path: "../phase1_nerves_system", runtime: false, targets: :stm32mp157f_dk2,
 nerves: [compile: true]}

The nerves: [compile: true] flag is important. Without it, Nerves tries to download a pre-built system artifact from GitHub Releases (like official systems do). Since we haven't published ours yet, it would fail. This flag says "I built the system locally, use that."

The firmware project also includes nerves_pack — a meta-package that pulls in everything you need for a basic connected device: networking (vintage_net), SSH (nerves_ssh), time synchronization (nerves_time), and device discovery (mdns_lite).

More Missing Packages: The NIF Problem

When mix firmware runs, it cross-compiles the Elixir dependencies using the ARM cross-compiler from our system's sysroot. Some dependencies have NIFs (Native Implemented Functions) — C code that gets compiled alongside the Elixir code. If those C files #include a header that's not in our sysroot, the build fails.

This is how I discovered that our system was missing two networking libraries:

Error 13: libmnl.h: No such file or directory — The nerves_uevent package (part of nerves_pack) uses netlink sockets to listen for kernel device events (like "USB device plugged in" or "network interface came up"). Netlink is a Linux IPC mechanism for communication between the kernel and userspace. libmnl (Minimalistic Netlink Library) provides a clean API for it.

Fix: Add BR2_PACKAGE_LIBMNL=y to nerves_defconfig, rebuild the system.

Error 14: netlink/genl/genl.h: No such file or directory — The vintage_net_wifi package (WiFi management for Nerves) uses generic netlink to talk to the WiFi driver for scanning access points. libnl is a larger netlink library that includes generic netlink support.

Fix: Add BR2_PACKAGE_LIBNL=y to nerves_defconfig, rebuild the system.

After each new package, rebuilding the system takes just seconds — Buildroot only compiles the new package and regenerates the rootfs. Then mix firmware picks up the new headers in the sysroot.

These packages aren't needed by the base system — they're needed by the application's NIFs. Official Nerves systems (like the BeagleBone Black system) already include them because nerves_pack is such a common dependency. When building a custom system, you discover these requirements one by one.

The fwup.conf Fixes

Three more issues in fwup.conf needed fixing before firmware could be built.

Problem 1: The Include Path

My fwup.conf had:

include("fwup_include/provisioning.conf")

This worked when fwup ran from the system directory, but during the firmware build, Buildroot copies fwup.conf to the images/ directory — and fwup_include/ wasn't copied alongside it. The include() path is relative to wherever fwup reads the file from, so it broke.

The Buildroot mechanism (BR2_NERVES_ADDITIONAL_IMAGE_FILES) uses cp without -r, so it can't copy directories. I tried adding the directory to the additional files list and got cp: -r not specified; omitting directory.

Fix: I inlined the provisioning variables directly into fwup.conf. For four define() statements, a separate file was over-engineering.

Problem 2: Missing Partition GUIDs

GPT partitions have two types of identifiers that look similar but serve different purposes:

• type — the partition type GUID. This says "what kind of partition is this?" For example, 0fc63daf-8483-4772-8e79-3d69d8477de4 means "Linux filesystem." The STM32MP1 ROM code uses partition names (not types) to find bootloader partitions, but the type GUID must be set for a valid GPT.

• guid — the partition's unique identifier. Every partition needs its own unique GUID. This is how the OS distinguishes between two partitions of the same type (e.g., rootfs-a and rootfs-b are both "Linux filesystem" type, but they're different partitions).

I had type on every partition but forgot guid. fwup error: partition 0 must have a valid guid.

Fix: Add a unique guid to each partition definition.

Problem 3: The U-Boot Environment Block

This one is subtle and ties together three different configuration files.

U-Boot stores its environment variables (like "which rootfs slot is active?") in a specific region of the microSD card. This region isn't a GPT partition — it's a raw area at a fixed byte offset. Three different tools need to agree on exactly where this region lives:

1. U-Boot reads/writes its environment here. Configured by CONFIG_ENV_OFFSET=0x480000 and CONFIG_ENV_SIZE=0x2000 in uboot/uboot.fragment.

2. Linux userspace (fw_printenv/fw_setenv) reads/writes the same region. Configured by /dev/mmcblk0 0x480000 0x2000 in rootfs_overlay/etc/fw_env.config.

3. fwup reads/writes the same region during firmware updates to switch the active slot. Configured by the uboot-environment block in fwup.conf.

My fwup.conf had upgrade tasks that referenced uboot-env:

require-uboot-variable(uboot-env, "nerves_fw_active", "b")

But I hadn't defined the uboot-environment block:

uboot-environment uboot-env {
    block-offset = 2304
    block-count = 16
}

The numbers: 0x480000 bytes / 512 bytes per block = block 2304. 0x2000 bytes / 512 bytes per block = 16 blocks. fwup works in 512-byte blocks (the standard sector size for SD cards), while U-Boot and fw_env.config use byte offsets. Same physical location, different units.

Storage: Why microSD and Not eMMC

The STM32MP157F-DK2 is a discovery board — designed for evaluation and learning, not production. It has a microSD card slot but no eMMC.

For production devices, you'd want eMMC (embedded MultiMediaCard):

• Soldered to the board — no physical connector to vibrate loose
• Built-in wear leveling and error correction
• Significantly faster than microSD (especially random writes)
• Available on the STM32MP157F-EV1 evaluation board or custom PCBs

For development, microSD is actually convenient:

• Pull it out, reflash on your PC, put it back in seconds
• Swap between different firmware versions by swapping cards
• SquashFS rootfs is read-only, so no wear concerns on the system partition
• The app partition uses F2FS (Flash-Friendly File System), designed for NAND storage

If we later move to a production board with eMMC, the fwup.conf would change the device references from mmcblk0 (microSD) to mmcblk1 (eMMC), but the partition layout and update logic stay the same.

Error 15: fwup raw_write Called Twice on Same Resource

The firmware built fine, but when I tried to actually flash it to the microSD card:

$ sudo fwup -a -d /dev/sda -i phase1_test_firmware.fw -t complete
  0% [                                    ]
fwup: raw_write didn't write anything and was likely called twice in an on-resource for 'tf-a.stm32'. Try a "cp" function.

The problem is in how fwup handles resources. fwup is a streaming tool — it reads each resource from the .fw archive once, like a tape. When you have:

on-resource tf-a.stm32 {
    raw_write(${FSBL_PART_OFFSET})    # writes to fsbl1
    raw_write(${FSBL2_PART_OFFSET})   # nothing left to write!
}

The first raw_write consumes the entire tf-a.stm32 data stream. The second raw_write gets zero bytes because the stream is already exhausted. fwup detects this and errors out rather than silently writing an empty partition.

Why do we need two copies? The STM32MP1 ROM bootloader expects two redundant copies of the first-stage bootloader (TF-A). If fsbl1 is corrupted, the ROM falls back to fsbl2. It's a safety mechanism baked into the silicon.

The fix: define the same physical file as two separate fwup resources. Each resource gets its own stream:

file-resource tf-a-1.stm32 {
    host-path = "${NERVES_SYSTEM}/images/tf-a-stm32mp157c-dk2.stm32"
}
file-resource tf-a-2.stm32 {
    host-path = "${NERVES_SYSTEM}/images/tf-a-stm32mp157c-dk2.stm32"
}

Then in the task:

on-resource tf-a-1.stm32 {
    raw_write(${FSBL_PART_OFFSET})
}
on-resource tf-a-2.stm32 {
    raw_write(${FSBL2_PART_OFFSET})
}

Same file, two independent streams, two independent writes. The .fw archive ends up with two copies of the TF-A binary inside it, which adds ~250 KB — trivial compared to the 28 MB total.

Aside: Running fwup with sudo and asdf

If you installed fwup via asdf, sudo fwup won't work — sudo doesn't inherit your user's PATH, so it can't find the asdf shim. And even sudo $(which fwup) fails because the asdf shim script doesn't work under sudo's environment.

The fix: resolve the actual binary path, bypassing the shim entirely:

sudo $(asdf which fwup) -a -d /dev/sda -i firmware.fw -t complete

asdf which fwup returns something like ~/.asdf/installs/fwup/1.10.2/bin/fwup — the real ELF binary, not the shim wrapper.

Firmware Built!

After fixing everything:

Building phase1_test_firmware.fw...
Firmware UUID: fall-earth (3d48c051-a7ab-547f-9256-8a5b36669289)
Firmware built successfully!

A 28 MB firmware file. It contains the entire boot chain (TF-A first-stage bootloader, FIP with sp_min + U-Boot, Linux kernel 6.6.80), a 30 MB SquashFS root filesystem with Erlang/OTP 28 and our test application, device tree blob for the DK2, and fwup metadata for A/B updates.

Flashing the microSD Card

With the firmware rebuilt (including the tf-a fix), time to flash:

$ mix firmware
==> nerves_system_stm32mp157f_dk2
Generated nerves_system_stm32mp157f_dk2 app
==> phase1_test_firmware
Nerves environment
  MIX_TARGET:   stm32mp157f_dk2
  MIX_ENV:      dev
Generated phase1_test_firmware app
|nerves| Building OTP Release...
Building phase1_test_firmware.fw...
Firmware UUID: cushion-armor (40117cc3-cdf8-5aab-f68b-5df9ef95b766)
Firmware built successfully!

Then flash to the microSD card. I used a fresh 32 GB card (keeping the original ST demo card as reference):

$ sudo $(asdf which fwup) -a -d /dev/sda -i _build/stm32mp157f_dk2_dev/nerves/images/phase1_test_firmware.fw -t complete
100% [====================================] 28.49 MB in / 32.95 MB out
Success!
Elapsed time: 4.917 s

28 MB in, 33 MB out (fwup decompresses the SquashFS rootfs during write). Under 5 seconds. The -t complete task wrote the GPT table, both TF-A copies, the FIP binary, and the rootfs to slot A — exactly what our fwup.conf defined.

Next step: put the card in the DK2, connect serial, and see if it boots.

From System to Firmware to... Silence

I put the card in, connected picocom, powered on, and... partial success. TF-A came up perfectly:

NOTICE:  CPU: STM32MP157FAC Rev.Z
NOTICE:  Model: STMicroelectronics STM32MP157C-DK2 Discovery Board
...
NOTICE:  BL2: TF-A Runtime loaded
NOTICE:  SP_MIN: Initializing runtime services
NOTICE:  SP_MIN: Preparing exit to normal world

TF-A (BL2) initialized DDR, loaded the FIP, and handed off to SP_MIN. SP_MIN set up the secure monitor and prepared to jump to the "normal world" (U-Boot). And then... nothing. Complete silence. No U-Boot banner, no error, no panic. Just a blinking cursor.

This is the worst kind of embedded debugging problem. When something crashes with an error message, you at least know what went wrong. When it hangs silently, you're left guessing.

Error 16: U-Boot Environment CRC Mismatch on Fresh Card

But before the hang debugging, let me backtrack. The first flash attempt actually failed earlier with a different error. When fwup tried to set U-Boot environment variables on the freshly written card:

fwup: uboot_setenv failed: CRC32 mismatch

The problem: uboot_setenv expects a valid U-Boot environment block on the card. But on a fresh card, the environment area contains random data with no valid CRC32 checksum. U-Boot's environment format starts with a CRC32 of the data that follows — if the CRC doesn't match, the environment is considered corrupt.

The fix: call uboot_clearenv() first. This writes a blank environment with a valid CRC, then uboot_setenv works:

on-finish {
    uboot_clearenv(uboot-env)
    uboot_setenv(uboot-env, "nerves_fw_active", "a")
    uboot_setenv(uboot-env, "bootcmd", "load mmc ...")
    # ... etc
}

This is the kind of thing that only bites you on the first flash. Once U-Boot has written its own environment at least once, the CRC is always valid. But for factory provisioning, you need uboot_clearenv first.

Error 17: U-Boot Silent Console

Back to the hang. My first theory: U-Boot is running but not printing anything. The stm32mp15_trusted defconfig in U-Boot 2024.01 has some suspicious settings:

CONFIG_DISABLE_CONSOLE=y
CONFIG_SILENT_CONSOLE=y

These literally disable all console output. Why would a default board config do this? Probably for production — you don't want a boot menu appearing on a customer's device. But for development, it means you're flying blind.

I added these to uboot/uboot.fragment:

# CONFIG_DISABLE_CONSOLE is not set
# CONFIG_SILENT_CONSOLE is not set
# CONFIG_SPL_SILENT_CONSOLE is not set
# CONFIG_TPL_SILENT_CONSOLE is not set
# CONFIG_SYS_CONSOLE_IS_IN_ENV is not set
CONFIG_CONS_INDEX=1

Rebuilt, reflashed, rebooted. Still nothing after SP_MIN. So the console silencing wasn't the real problem — or at least, not the only one.

Error 18: U-Boot Hangs After SP_MIN (UNRESOLVED)

This is where I spent hours. TF-A and SP_MIN work perfectly. U-Boot never produces any output. The board isn't hard-locked (the power LED stays on, ST-LINK stays connected), but nothing comes out of UART4.

Here's everything I tried, in order:

Attempt 1: Disable OP-TEE and SCMI. The stm32mp15_trusted defconfig enables OP-TEE (a Trusted Execution Environment) and SCMI (System Control and Management Interface). We use SP_MIN instead of OP-TEE. SCMI is a protocol for U-Boot to talk to the secure monitor about clock and power management. If U-Boot tries to make SCMI calls and SP_MIN doesn't handle them, U-Boot could hang during board init — before it ever prints anything.

# CONFIG_OPTEE is not set
# CONFIG_SCMI_AGENT_OPTEE is not set
# CONFIG_RNG_OPTEE is not set
# CONFIG_SCMI_FIRMWARE is not set
# CONFIG_SCMI_AGENT_SMCCC is not set

The annoying thing about SCMI: even after disabling CONFIG_SCMI_FIRMWARE, Kconfig pulls it back in through dependencies. CONFIG_SCMI_AGENT_SMCCC depends on CONFIG_ARM_SMCCC, which is always enabled on ARM. I had to trace through the Kconfig dependency tree to find all the entry points.

Still hung.

Attempt 2: Pre-populate U-Boot environment. U-Boot's default boot sequence (distro_bootcmd) scans for extlinux.conf and boot.scr across multiple devices. Maybe it was getting stuck in a scan loop. I pre-populated the environment via fwup with explicit boot commands:

uboot_setenv(uboot-env, "bootcmd", "load mmc 0:4 ...")
uboot_setenv(uboot-env, "bootargs", "console=ttySTM0,115200 ...")

Still hung. But this was good practice anyway — we'd need this for a proper Nerves boot.

Attempt 3: Fix invalid Buildroot TF-A option. I discovered that BR2_TARGET_ARM_TRUSTED_FIRMWARE_BL32_SP_MIN=y — the option I had in nerves_defconfig to tell Buildroot to build SP_MIN as BL32 — doesn't actually exist in Buildroot 2025.11. Grep through the entire Buildroot source: zero matches. It was silently ignored.

The upstream stm32mp157c_dk2_defconfig in Buildroot uses BR2_TARGET_ARM_TRUSTED_FIRMWARE_BL31=y. Looking at the Buildroot TF-A package code, BL31 is the generic "build the BL32 component" option, and the actual type of BL32 (SP_MIN vs OP-TEE) is controlled by AARCH32_SP=sp_min in the additional variables. We already had that. So I changed:

# Before (invalid — silently ignored)
BR2_TARGET_ARM_TRUSTED_FIRMWARE_BL32_SP_MIN=y

# After (correct — matches upstream)
BR2_TARGET_ARM_TRUSTED_FIRMWARE_BL31=y

I also added E=0 to the TF-A additional variables, which upstream uses to set the exception level for BL33 entry (EL0 = non-secure).

Rebuilt everything. Still hung.

Attempt 4: Verify the FIP contents. I used fiptool info to inspect the FIP binary and confirmed it contained the updated U-Boot. I checked the U-Boot DTB inside the build tree — stdout-path correctly pointed to serial0 = UART4 at 0x40010000, 115200n8. The load address (CONFIG_TEXT_BASE=0xC0100000) matched STM32MP_BL33_BASE in TF-A. Everything looked correct on paper.

Key observation: The original ST demo card (U-Boot 2022.10, from the Yocto-based OpenSTLinux) boots perfectly on the same hardware. So the board, the cables, and the serial console are all fine. The problem is specific to our U-Boot 2024.01 build.

Another observation: The upstream Buildroot stm32mp157c_dk2_defconfig uses U-Boot 2025.10, not 2024.01. Our version choice was slightly arbitrary. It's possible that U-Boot 2024.01 has a specific issue with SP_MIN on STM32MP1 that's fixed in newer versions. Or maybe a config option we're missing.

This is where the first debugging session ended. The board boots TF-A and SP_MIN correctly but U-Boot 2024.01 produces no output and appears to hang. Time to get systematic.

The U-Boot Version Upgrade (Error 18 Continued)

The upstream Buildroot stm32mp157c_dk2_defconfig uses U-Boot 2025.10, not our 2024.01. Maybe it's a version issue? I upgraded:

BR2_TARGET_UBOOT_CUSTOM_VERSION_VALUE="2025.10"

This also required adding dependencies that U-Boot 2025.10 needs:

BR2_TARGET_UBOOT_NEEDS_PYLIBFDT=y
BR2_TARGET_UBOOT_NEEDS_OPENSSL=y
BR2_TARGET_UBOOT_NEEDS_GNUTLS=y

And the U-Boot device tree path changed in newer versions:

# Before (2024.01)
BR2_TARGET_UBOOT_CUSTOM_MAKEOPTS="DEVICE_TREE=stm32mp157c-dk2"
# After (2025.10)
BR2_TARGET_UBOOT_CUSTOM_MAKEOPTS="DEVICE_TREE=st/stm32mp157c-dk2"

I also simplified the uboot.fragment — many of the overrides I'd added (SCMI disable, OP-TEE disable) kept fighting with Kconfig dependencies. For the clean test, I stripped the fragment back to just the essentials: autoboot settings, SquashFS command, and env storage location.

Clean rebuild:

cd /home/tomaz/razvoj/elixir_projects/nerves/stm32-moj/phase1_nerves_system/.nerves/artifacts/nerves_system_stm32mp157f_dk2-portable-0.1.0
make uboot-dirclean && make arm-trusted-firmware-dirclean && make

The TF-A timestamp confirmed the new build was being used:

NOTICE:  BL2: Built : 07:08:56, Mar 14 2026

Same hang. No U-Boot output.

Systematic Isolation: What's Different from Upstream?

At this point I stopped guessing and started comparing. I had the exact upstream Buildroot stm32mp157c_dk2_defconfig in my source tree. Let me compare it against our config line by line.

TF-A section — nearly identical:

# Upstream
BR2_TARGET_ARM_TRUSTED_FIRMWARE_ADDITIONAL_VARIABLES="STM32MP_SDMMC=1 AARCH32_SP=sp_min DTB_FILE_NAME=stm32mp157c-dk2.dtb E=0 BL33_CFG=$(BINARIES_DIR)/u-boot.dtb"

# Ours
BR2_TARGET_ARM_TRUSTED_FIRMWARE_ADDITIONAL_VARIABLES="STM32MP_SDMMC=1 AARCH32_SP=sp_min DTB_FILE_NAME=stm32mp157c-dk2.dtb E=0 BL33_CFG=$(BINARIES_DIR)/u-boot.dtb STM32MP15=1"

The only difference: we add STM32MP15=1, which is harmless — TF-A auto-detects this from the DTB filename anyway.

U-Boot section — identical except we have a config fragment. Upstream doesn't.

FIP contents verified with fiptool info:

Secure Payload BL32 (Trusted OS): offset=0x100, size=0x8B00
Non-Trusted Firmware BL33: offset=0x8C00, size=0x1066F8
FW_CONFIG: offset=0x10F2F8, size=0x226
HW_CONFIG: offset=0x10F51E, size=0x122D0
TOS_FW_CONFIG: offset=0x1217EE, size=0x37CB

U-Boot binary: 1,074,936 bytes. Device tree: 74,448 bytes. Both present, both correct sizes. The U-Boot binary starts with valid ARM exception vectors (0xb80000ea = b reset), addresses in the expected DDR range (0xC0100000).

The U-Boot device tree has stdout-path = "serial0:115200n8" pointing to UART4 at 0x40010000. Same UART that TF-A uses successfully.

So what IS different?

1. Our U-Boot config fragment (AUTOBOOT_KEYED, SQUASHFS, ENV settings)
2. The toolchain (Nerves toolchain vs Bootlin's)
3. Everything else is identical

Test 1: Remove the Config Fragment

Simplest test first. Comment out our U-Boot config fragment entirely:

# TEMPORARILY DISABLED for debugging
# BR2_TARGET_UBOOT_CONFIG_FRAGMENT_FILES="${NERVES_DEFCONFIG_DIR}/uboot/uboot.fragment"

Clean rebuild U-Boot and TF-A. Rebuild firmware. Flash. Boot.

NOTICE:  BL2: v2.10.0    (release):v2.10.0
NOTICE:  BL2: Built : 07:40:48, Mar 14 2026
NOTICE:  BL2: Booting BL32
NOTICE:  SP_MIN: v2.10.0    (release):v2.10.0
NOTICE:  SP_MIN: Built : 07:40:48, Mar 14 2026

Same hang. This rules out our config fragment — the vanilla stm32mp15_trusted_defconfig with zero modifications still produces a U-Boot that hangs.

Test 2: Build Vanilla Buildroot (The Nuclear Option)

If it's not our config, it must be the toolchain. The Nerves toolchain (GCC 13.2, armv7-nerves-linux-gnueabihf) is used to compile U-Boot. The upstream Buildroot config uses Bootlin's external toolchain instead.

To test this, I built the exact upstream stm32mp157c_dk2_defconfig completely outside the Nerves layer, using Buildroot directly:

cd phase1_nerves_system/deps/nerves_system_br/buildroot-2025.11.2
make O=/tmp/buildroot-stm32-test stm32mp157c_dk2_defconfig
make O=/tmp/buildroot-stm32-test -j$(nproc)

This downloads Bootlin's pre-built toolchain instead of using ours, and builds everything from scratch. The output includes a complete sdcard.img ready to flash:

sudo umount /dev/sda*
sudo dd if=/tmp/buildroot-stm32-test/images/sdcard.img of=/dev/sda bs=1M status=progress

And then:

U-Boot 2025.10 (Mar 14 2026 - 07:54:58 +0100)

CPU: STM32MP157FAC Rev.Z
Model: STMicroelectronics STM32MP157C-DK2 Discovery Board
Board: stm32mp1 in trusted - stm32image mode (st,stm32mp157c-dk2)
Board: MB1272 Var4.0 Rev.C-03
DRAM:  512 MiB
Clocks:
- MPU : 650 MHz
...
Hit any key to stop autoboot: 0
Boot over mmc0!
...
Starting kernel ...
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 6.12.53 ...
...
Welcome to Buildroot
buildroot login:

It boots. U-Boot, kernel, full Linux — everything works perfectly.

The Root Cause: The Nerves Toolchain

This was the moment everything clicked. The only meaningful difference between our failing build and this working one was the toolchain. But to understand why, you need to understand what a "toolchain" actually is.

What Is a Cross-Compiler Toolchain, Really?

Imagine you speak English and you need to write a letter in Japanese. You can't just write it yourself — you need a translator. A cross-compiler is that translator: it's a program that runs on your x86 PC (English) but produces code that runs on an ARM chip (Japanese).

A "toolchain" is the translator plus all the reference dictionaries it needs:

A cross-compiler toolchain contains:
├── GCC              (the translator — turns C code into ARM machine code)
├── binutils         (helper tools — linker, assembler, like a typesetter)
├── glibc            (the C standard library — common phrases the translator knows)
├── Linux headers    (the kernel's API dictionary — how to talk to the OS)
└── built by some tool (crosstool-ng, Buildroot, or pre-packaged)

None of these components are proprietary or special. GCC is GCC. glibc is glibc. It's the same open-source software regardless of who packaged it. The differences are:

• Which versions of each component
• What build options were used when compiling GCC itself
• What defaults GCC uses when you don't specify flags explicitly

The "Nerves toolchain" isn't a Nerves-made compiler. It's a standard GCC built from source by crosstool-ng, with specific version pins chosen by the Nerves team for reproducibility. The -nerves- in the name armv7-nerves-linux-gnueabihf is just a vendor tag — like a label saying "this particular build of GCC was assembled by the Nerves project." It could just as easily be armv7-acme-linux-gnueabihf and the resulting binaries would be identical.

Here's what's inside each:

Both produce ARM code. Both use hard-float ABI (gnueabihf). Both link against glibc. For 99% of software, they're interchangeable.

But not for bootloaders.

Why Bootloaders Are Special

The same U-Boot source code (2025.10), the same defconfig (stm32mp15_trusted), the same TF-A (v2.10), the same FIP assembly — compiled with one toolchain it boots, compiled with another it crashes silently before producing any serial output.

Here's why bootloaders are uniquely sensitive: they run in a world with no safety nets.

When a normal Linux application crashes, the kernel catches the fault, prints a stack trace, and kills the process. The kernel provides virtual memory, exception handlers, and a stable environment. Even a badly compiled application usually crashes visibly.

A bootloader has none of that. U-Boot runs directly on the hardware:

• No operating system — no exception handlers, no memory protection
• No MMU — a bad pointer doesn't segfault, it corrupts random memory
• No console yet — early initialization happens before the serial port is configured
• Direct hardware access — wrong register writes can lock up the chip

So if GCC generates one wrong instruction during U-Boot's early startup — maybe a Thumb instruction where ARM mode is expected, or a floating-point instruction on a core that hasn't enabled the FPU yet — the CPU takes an unhandled exception and locks up. No error message. No crash dump. Just silence.

This is exactly what we saw: TF-A and SP_MIN (which are simpler programs) worked fine. U-Boot, which does complex board initialization with clocks, DRAM, and peripheral setup, died silently.

The Nerves toolchain works perfectly for RPi4, BeagleBone, and every other Nerves-supported board. This isn't a "broken" toolchain — it's a specific incompatibility between GCC 13.2's code generation and the STM32MP1's early boot environment. It might be a GCC 13.2 bug fixed in 14.3, or a subtle difference in default code generation flags. Without deep-diving into the disassembly of both U-Boot binaries, we can't pinpoint the exact cause. But we don't need to — we have a working solution.

The Fix: Switch to Bootlin's Toolchain

We switched our nerves_defconfig from the Nerves custom toolchain to Bootlin's pre-built toolchain. This is what the upstream Buildroot STM32MP1 defconfig uses, so it's well-tested.

The change in nerves_defconfig:

# BEFORE: Nerves toolchain (broken U-Boot on STM32MP1)
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="armv7-nerves-linux-gnueabihf"
BR2_TOOLCHAIN_EXTERNAL_HEADERS_5_4=y
BR2_TOOLCHAIN_EXTERNAL_GCC_13=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_TOOLCHAIN_EXTERNAL_FORTRAN=y
BR2_TOOLCHAIN_EXTERNAL_OPENMP=y
# BR2_TOOLCHAIN_EXTERNAL_INET_RPC is not set
BR2_TOOLCHAIN_EXTERNAL_PATH="${NERVES_TOOLCHAIN}"

# AFTER: Bootlin toolchain (works for everything including bootloaders)
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV7_EABIHF_GLIBC_STABLE=y

That's it. Three lines instead of ten. Buildroot downloads the Bootlin toolchain automatically during the build. No manual setup, no crosstool-ng compilation, no 20-minute toolchain build step.

The trade-off: we lose the Nerves team's specific version pinning. But we gain a toolchain that's proven to work with our board's entire boot chain — and it's the same toolchain used by upstream Buildroot's official STM32MP1 support, maintained by Bootlin in partnership with STMicroelectronics.

To rebuild with the new toolchain, we did a clean build within the Nerves artifact directory:

cd phase1_nerves_system/.nerves/artifacts/nerves_system_stm32mp157f_dk2-portable-0.1.0
make clean && make

This is the full Buildroot system build — the big one (~45 minutes). It has to recompile everything from scratch because every binary in the system needs to be compiled with the same toolchain. You can't mix binaries from two different toolchains in the same rootfs — the glibc versions and ABI details need to match.

Here's what gets rebuilt:

1. Downloads Bootlin toolchain (~100MB pre-built GCC 14.3 for ARM)
2. Linux kernel 6.6.80 — the biggest piece, cross-compiled for Cortex-A7
3. TF-A v2.10 — first-stage bootloader (BL2 + SP_MIN)
4. U-Boot 2025.10 — second-stage bootloader (the one that was crashing!)
5. Erlang/OTP 28 — the entire BEAM VM, cross-compiled for ARM
6. BusyBox — minimal Unix utilities (shell, ls, cat, etc.)
7. Dropbear — lightweight SSH server
8. iproute2, libmnl, libnl — networking stack
9. nbtty, nerves_config — Nerves-specific packages
10. SquashFS rootfs — packs everything into a compressed read-only filesystem

The previous debugging builds we did (make uboot-dirclean && make) were incremental — they only rebuilt U-Boot and TF-A, taking just a few minutes. This clean build rebuilds the entire world. The source code for most packages is already downloaded from the earlier builds, so it skips the download step, but every last .c file gets recompiled.

Why can't we just rebuild the bootloaders? Because we changed the toolchain — the cross-compiler itself. Every compiled binary in the system was built with the old GCC 13.2. The new GCC 14.3 might produce slightly different code, link against a slightly different glibc, or use different default optimizations. Mixing binaries from two toolchains in the same rootfs is asking for subtle runtime crashes. Clean slate.

The Three-Layer Build System (or: Why There Are So Many Configs)

If you're following along and feeling confused about which config goes where and which command picks up which file — you're not alone. I stumbled through this myself. Here's how it actually works, explained as simply as I can.

Think of it like a restaurant kitchen with three people:

Layer 1: Your recipe book = nerves_defconfig

This is what you write. It says "I want Linux 6.6, U-Boot 2025.10, Bootlin toolchain, SquashFS, etc." It's a wish list — a simple text file with about 200 lines. It lives in your project directory and you edit it directly.

Layer 2: The head chef = the Nerves build system (mix compile)

The head chef reads your recipe book and sets up the kitchen. When you run mix compile, it:

• Creates a working kitchen (the build directory at .nerves/artifacts/.../)
• Expands your recipe into a full detailed plan — the .config file, which is about 6000 lines of resolved Buildroot configuration
• Calls the sous chef (Buildroot) to do the actual cooking

Layer 3: The sous chef = Buildroot (make)

Buildroot does the real work. It reads the .config (the detailed plan) and downloads source code, compiles everything with the toolchain, and puts the results in images/.

The flow looks like this:

nerves_defconfig  ──(mix compile)──>  .config  ──(make)──>  binaries
   (your wish)                      (full plan)            (actual output)

Here's the critical thing that tripped me up: when you edit nerves_defconfig, nothing happens automatically. It's just a file on disk. The .config in the build directory doesn't update itself.

• mix compile reads your nerves_defconfig and generates a new .config. That's the head chef reading the updated recipe.
• make (run directly in the artifact directory) reads .config — not your nerves_defconfig. The sous chef doesn't know your recipe book exists. If .config is stale (still says "Nerves toolchain"), make will happily use the old toolchain.

This is exactly what happened to us. We edited nerves_defconfig to switch to Bootlin's toolchain, then ran make clean && make directly in the build directory. But the .config still said "Nerves toolchain" because we hadn't run mix compile to update it. The build ran with the wrong toolchain, and we spent time wondering why nothing changed.

The correct sequence after editing nerves_defconfig:

# Step 1: Head chef reads the new recipe and updates the plan
cd phase1_nerves_system/
mix compile

# Step 2: Sous chef does a clean rebuild with the updated plan
cd .nerves/artifacts/nerves_system_stm32mp157f_dk2-portable-0.1.0/
make clean && make

Why both steps? Because mix compile updates .config and does a quick build (skipping anything Buildroot thinks is "already done"). But when you change the toolchain, everything needs to be recompiled — Buildroot doesn't know that. So you follow up with make clean && make to force a full rebuild.

The rule of thumb:

• Changed nerves_defconfig? → Run mix compile first (updates .config)
• Need a full rebuild? → Then cd to the artifact directory and run make clean && make
• Never run make alone after changing the defconfig — it won't see your changes

Error 18 Continued: U-Boot Gets Partially Unstuck

After the Bootlin toolchain switch and a full clean rebuild, something changed. Not a full fix, but progress. U-Boot now printed its banner:

U-Boot 2025.10 (Mar 14 2026 - 17:34:14 +0100)

CPU: STM32MP157FAC Rev.Z
Model: STMicroelectronics STM32MP157C-DK2 Discovery Board
Board: stm32mp1 in trusted - stm32image mode (st,stm32mp157c-dk2)
Board: MB1272 Var4.0 Rev.C-03
DRAM:  512 MiB

Then it stopped. No "Clk:" line, no autoboot, no prompt. But this was already a big step — with the Nerves toolchain we got nothing after SP_MIN. Now U-Boot was running, initializing DRAM, and talking to us on UART. It just froze during some later init step.

So there were TWO problems stacked on top of each other. The toolchain swap fixed the first one. Now we needed to find the second.

Error 19: The fwup.conf Math Bug — FIP Binary Corruption

I started comparing the FIP contents between our Nerves build and the working vanilla Buildroot build using fiptool info. Same component types, same sizes. So the FIP itself was fine. I tried swapping vanilla bootloader binaries onto our fwup-flashed card using dd — same hang. This seemed to prove the issue was in the disk layout, not the binaries.

But before chasing disk layout theories too far, I found something terrible in fwup.conf.

Our fwup.conf had a U-Boot environment block defined at line 107:

# Offset 0x480000 = block 2304 (at 512 bytes/block)
uboot-environment uboot-env {
    block-offset = 2304
    block-count = 16
}

See the comment? 0x480000 = block 2304. Let's check that math:

0x480000 = 4,718,592 bytes
4,718,592 / 512 = 9,216 blocks

The answer is 9216, not 2304. Someone (me) divided by 2048 instead of 512. A factor-of-four error.

So where does block 2304 actually land?

Block 2304 × 512 bytes = byte offset 0x120000

Our FIP binary is written at block 1058 and is about 2311 blocks long (1.18 MB). That means the FIP occupies blocks 1058 through ~3369. Block 2304 is right in the middle of the FIP binary — about 640 KB in, smack in the middle of the U-Boot code.

In the complete task's on-finish, fwup runs uboot_clearenv(uboot-env). This writes 8 KB of zeros at block 2304. It was zeroing out a chunk of U-Boot's machine code after writing the FIP.

TF-A loads the corrupted U-Boot into RAM. U-Boot starts executing from its entry point (which is before the corrupted area), prints the banner, CPU info, DRAM size — all that code is near the beginning of the binary. Then execution reaches the zeroed-out region and the CPU takes an unhandled exception. No error message, no crash dump — just the silence we'd been staring at.

The fix: Change block-offset from 2304 to 9216. At block 9216, the environment lands at byte offset 0x480000, which is within the FIP partition (blocks 1058–9249) but well past the end of the actual FIP binary (which ends around block 3369). There's about 5850 blocks of empty space in the tail of the FIP partition — that's where the environment should live.

uboot-environment uboot-env {
    block-offset = 9216    # was 2304 — WRONG (corrupted FIP binary!)
    block-count = 16
}

The environment was also shared between three configuration files that all needed to agree:

• fwup.conf: block-offset = 9216 (fixed)
• uboot/uboot.fragment: CONFIG_ENV_OFFSET=0x480000 (was already correct!)
• rootfs_overlay/etc/fw_env.config: /dev/mmcblk0 0x480000 0x2000 (was already correct!)

Both the U-Boot config and the Linux userspace config had the right offset all along. Only fwup had the wrong one, because its config uses 512-byte blocks while everything else uses byte offsets. Unit conversion bugs are a classic.

I also fixed the partition type GUIDs while I was in fwup.conf. We'd been using 8da63339-0007-60c0-c436-083ac8230908 ("Linux reserved") for the bootloader partitions. The upstream vanilla Buildroot uses 0fc63daf-8483-4772-8e79-3d69d8477de4 ("Linux filesystem") for all partitions, including bootloaders. The STM32MP1 ROM code finds bootloader partitions by name ("fsbl1", "fsbl2"), not by type GUID, so this probably didn't matter — but matching upstream is always safer than being creative.

Error 20: /dev/sda Was a Regular File

Rebuilt firmware with the fixed fwup.conf. Flashed. Booted. Same hang at "DRAM: 512 MiB".

Wait, what?

I spent another hour testing. Swapped in vanilla bootloaders via dd — still hung. Ran sgdisk -p /dev/sda to examine the GPT:

last usable sector is 3334

On a 32 GB card? Something was very wrong. I checked the partition table:

Warning! Secondary partition table overlaps the last partition by 1054491 blocks!
Problem: partition 4 is too big for the disk.
Problem: partition 5 is too big for the disk.

The GPT header thought the disk was only 3334 sectors (~1.6 MB). But the card was 32 GB. I checked blockdev --getsize64:

blockdev: ioctl error on BLKGETSIZE64: Inappropriate ioctl for device

That ioctl should never fail on a block device. And then lsblk told the truth:

lsblk: /dev/sda: not a block device

I ran ls -la /dev/sda:

-rw-r--r-- 1 root root 36438016 mar 14 18:06  /dev/sda

That - at the beginning. Not b (block device). Not c (character device). A regular file. A 36 MB regular file sitting in /dev, masquerading as the SD card device.

At some point during our debugging — probably when I pulled the SD card out while fwup was expecting a device at that path, or when dd created the path — the kernel's device node got replaced by a regular file. Every subsequent fwup, dd, and sgdisk command was reading and writing this 36 MB file on the NVMe drive, not the actual SD card.

lsblk showed the real sda block device existed in the kernel (major 8, minor 0, 29.7 GB). But the file at /dev/sda was shadowing it.

The fix:

sudo rm /dev/sda                  # delete the regular file
sudo mknod /dev/sda b 8 0         # recreate the block device node
sudo chown root:disk /dev/sda     # set proper ownership

After this, ls -la /dev/sda showed:

brw-rw---- 1 root disk 8, 0 ...  /dev/sda

That b at the beginning. Block device. The real one.

The Lesson: Always Verify Your Assumptions

This was humbling. I spent over an hour building theories about FIP corruption, GPT header sizes, and partition type GUIDs — and the actual problem was that /dev/sda was a regular file. Every test I ran was writing to a 36 MB file on my NVMe, not to the SD card. The board was still booting from whatever was on the card from the last time we wrote to the real device — which was the build with the corrupted FIP.

How to avoid this:

1. After any flash operation, verify the device: ls -la /dev/sda — look for b at the start
2. If block device commands fail, check the device type before debugging further
3. After pulling/reinserting the card, wait for udev to settle: udevadm settle

The irony: both the fwup.conf math bug AND the /dev/sda file issue were real problems that needed fixing. If I'd caught the /dev/sda issue first, I might have missed the math bug entirely — and it would have bitten us later when we re-enabled the U-Boot environment writes for real A/B slot switching.

Nerves Partition Philosophy: Why fwup, Not genimage

While debugging all this, I looked at how vanilla Buildroot creates the SD card image. It uses a tool called genimage that creates a complete disk image file (sdcard.img) with a proper GPT partition table, then you dd the entire image to the card:

# Buildroot's genimage.cfg.template for STM32MP1
image sdcard.img {
    hdimage { partition-table-type = "gpt" }
    partition fsbl1 { image = "tf-a-stm32mp157c-dk2.stm32" }
    partition fsbl2 { image = "tf-a-stm32mp157c-dk2.stm32" }
    partition fip   { image = "fip.bin"; size = 2M }
    partition rootfs { image = "rootfs.ext4"; bootable = "yes" }
}

Simple. Creates a 128 MB image, dd it to the card, done. But Nerves deliberately does NOT use genimage. Here's why:

genimage creates full disk images. To update firmware, you'd have to transfer the entire image — bootloaders, kernel, rootfs, everything — even if only the application changed. For a 128 MB image over a slow IoT network link, that's a problem.

fwup creates differential archives. The .fw file is a zip-like archive that contains only the data that changes. For an OTA update (upgrading from slot A to slot B), the .fw file contains just the new rootfs — maybe 30 MB compressed. The bootloaders, partition table, and app data partition are untouched. This is the upgrade.a / upgrade.b task in fwup.conf:

task upgrade.b {
    require-uboot-variable(uboot-env, "nerves_fw_active", "a")
    on-resource rootfs.img {
        raw_write(${ROOTFS_B_PART_OFFSET})
    }
    on-finish {
        uboot_setenv(uboot-env, "nerves_fw_active", "b")
        uboot_setenv(uboot-env, "nerves_fw_validated", "0")
    }
}

That's it. Write the new rootfs to the inactive slot, flip the active flag, mark as unvalidated. The device reboots into the new firmware. If it fails to validate (application crashes, can't reach the cloud, whatever), U-Boot boots back into the previous slot. No data loss, no bricked device.

This is why Nerves uses fwup instead of genimage. The standard Nerves partition layout — MBR with boot A/B, rootfs A/B, and app data — is specifically designed for this update mechanism. Our STM32MP1 adaptation uses GPT instead of MBR (the ROM bootloader requires it) and has extra bootloader partitions (fsbl1, fsbl2, fip), but the core A/B update concept is identical.

The full partition map on our microSD card:

GPT
├── fsbl1    (256 KiB)    TF-A first-stage bootloader
├── fsbl2    (256 KiB)    TF-A copy (ROM bootloader fallback)
├── fip      (4 MiB)      FIP: SP_MIN + U-Boot + DTB
│   └── [U-Boot env at tail: 8 KB at offset 0x480000]
├── rootfs-a (256 MiB)    SquashFS — active slot
├── rootfs-b (256 MiB)    SquashFS — inactive slot (for updates)
└── app      (remaining)  F2FS — persistent data, survives updates

Compare to the standard Nerves layout (e.g., Raspberry Pi):

MBR
├── Boot A/B (FAT32)      Kernel, bootcode, config
├── Rootfs A (SquashFS)   Active root filesystem
├── Rootfs B (SquashFS)   Inactive (for updates)
└── App data (EXT4)       Persistent data

Same concept, different boot chain requirements. The STM32MP1 needs three extra partitions for its TF-A-based boot chain, and uses GPT because the ROM bootloader searches GPT entries by name. But from Nerves' perspective — from fwup's perspective — it's the same pattern: write data to specific offsets, flip U-Boot env variables, reboot.

U-Boot Lives!

With the real block device restored and all three bugs fixed (toolchain, FIP corruption, /dev/sda), I reflashed:

ls -la /dev/sda    # verify: must start with 'b' (block device)
sudo umount /dev/sda*
sudo $(asdf which fwup) -a -d /dev/sda -i phase1_test_firmware.fw -t complete

Power cycled the board, opened picocom, and:

NOTICE:  CPU: STM32MP157FAC Rev.Z
NOTICE:  Model: STMicroelectronics STM32MP157C-DK2 Discovery Board
NOTICE:  Board: MB1272 Var4.0 Rev.C-03
NOTICE:  BL2: v2.10.5(release):lts-v2.10.5
NOTICE:  BL2: Built : 17:34:22, Mar 14 2026
NOTICE:  BL2: Booting BL32
NOTICE:  SP_MIN: v2.10.5(release):lts-v2.10.5
NOTICE:  SP_MIN: Built : 17:34:23, Mar 14 2026


U-Boot 2025.10 (Mar 14 2026 - 17:34:14 +0100)

CPU: STM32MP157FAC Rev.Z
Model: STMicroelectronics STM32MP157C-DK2 Discovery Board
Board: stm32mp1 in trusted - stm32image mode (st,stm32mp157c-dk2)
Board: MB1272 Var4.0 Rev.C-03
DRAM:  512 MiB
Clocks:
- MPU : 650 MHz
- MCU : 208.878 MHz
- AXI : 266.500 MHz
- PER : 24 MHz
- DDR : 533 MHz
optee optee: OP-TEE api uid mismatch
Core:  152 devices, 41 uclasses, devicetree: board
WDT:   Started watchdog@5a002000 with servicing every 1000ms (32s timeout)
NAND:  0 MiB
MMC:   STM32 SD/MMC: 0, STM32 SD/MMC: 1
Loading Environment from MMC... ** Read outside partition 2
Invalid ENV offset in MMC, copy=0
In:    serial
Out:   serial
Err:   serial
optee optee: OP-TEE api uid mismatch
Previous ADC measurements was not the one expected, retry in 20ms
****************************************************
*        WARNING 500mA power supply detected       *
*     Current too low, use a 3A power supply!      *
****************************************************

Net:   eth0: ethernet@5800a000

Hit any key to stop autoboot: 0
Boot over mmc0!
Saving Environment to MMC... Invalid ENV offset in MMC, copy=1
Failed (1)
switch to partitions #0, OK
mmc0 is current device
STM32MP>

U-Boot is fully alive. After hours of debugging three stacked bugs, the full boot chain is working: ROM → TF-A BL2 → SP_MIN → U-Boot. Let me break down what this output tells us.

Reading the U-Boot Output

The good:

• Clocks initialized: MPU at 650 MHz, DDR at 533 MHz — everything running at the right frequencies
• MMC detected: STM32 SD/MMC: 0 — the microSD card is visible
• Network detected: eth0: ethernet@5800a000 — the Gigabit Ethernet PHY is recognized
• Serial working: In: serial / Out: serial / Err: serial — console I/O is routed correctly
• Autoboot attempted: Boot over mmc0! — U-Boot tried to boot from the SD card

The warnings (all expected at this stage):

• "optee: OP-TEE api uid mismatch" — U-Boot's default config tries to talk to OP-TEE, but we're running SP_MIN instead. SP_MIN doesn't implement the OP-TEE API, so the UID check fails. Harmless — U-Boot falls back gracefully. This warning will go away when we re-enable our config fragment.

• "Invalid ENV offset in MMC" — U-Boot tried to load its saved environment from the SD card and couldn't find it. This is because our U-Boot config fragment (which sets CONFIG_ENV_IS_IN_MMC=y and CONFIG_ENV_OFFSET=0x480000) is currently disabled. Without it, U-Boot uses the default environment compiled into the binary — and the default env location doesn't match our partition layout. The boot commands we set via fwup's uboot_setenv are sitting on the card at the right offset, but U-Boot doesn't know to look there.

• "WARNING 500mA power supply detected" — The board is powered through the ST-LINK micro-USB (which provides only 500 mA from the PC's USB port). The DK2 has a USB-C power connector that expects a 3A supply for full operation. For basic serial console debugging this is fine, but once we start using Ethernet, WiFi, or the display, we'll need proper power.

Why U-Boot stopped at the prompt:

U-Boot ran its default distro_bootcmd, which scans for boot scripts (extlinux.conf, boot.scr) across all storage devices. Our rootfs is SquashFS — and without CONFIG_CMD_SQUASHFS=y (from our disabled config fragment), U-Boot can't even read files from it. The scan found nothing bootable and dropped to the interactive prompt.

What Needs to Happen Next

To get Linux booting, we need to re-enable the U-Boot config fragment (uboot/uboot.fragment) which adds:

• CONFIG_CMD_SQUASHFS=y — lets U-Boot read our SquashFS rootfs
• CONFIG_ENV_IS_IN_MMC=y — tells U-Boot where to read/write its environment
• CONFIG_ENV_OFFSET=0x480000 — matches our fwup.conf and fw_env.config
• Boot delay and autoboot settings

But first, let's try manually booting from the U-Boot prompt to validate the kernel and rootfs are there and loadable.

Manual Boot Attempt from U-Boot Prompt

At the STM32MP> prompt, I tried loading the kernel:

STM32MP> load mmc 0:4 0xc2000000 /boot/zImage
Can't set block device

"Can't set block device" — U-Boot can't access partition 4 of mmc device 0. This could mean:

1. U-Boot doesn't have SquashFS support compiled in (most likely — CONFIG_CMD_SQUASHFS=y is in our disabled config fragment)
2. Or the GPT partition table isn't being read correctly

Let's check what U-Boot sees. I ran part list mmc 0:

STM32MP> part list mmc 0

This will show whether U-Boot can read our GPT partition table at all, and how it numbers the partitions.

Note on power supply: The "WARNING 500mA power supply detected" message appeared because the USB-C port was connected via a USB-A to USB-C cable to the PC's front USB port. A USB-A port only provides 500 mA (USB 2.0) or 900 mA (USB 3.0). The DK2 board detects power source capability via USB Type-C Power Delivery ADC readings. For proper power, you need a USB-C PD charger (5V/3A) with a USB-C to USB-C cable. For serial console debugging, the current setup works fine — the warning is about current available, not current consumed.

Debugging from the U-Boot Prompt

When U-Boot drops to its STM32MP> prompt, you have a powerful debugging environment. Here are the two commands that tell you the most:

part list mmc 0 — shows the GPT partition table as U-Boot sees it:

STM32MP> part list mmc 0

Partition Map for mmc device 0  --   Partition Type: EFI

Part    Start LBA    End LBA        Name
    Attributes
    Type GUID
    Partition GUID
  1    0x00000022    0x00000221    "fsbl1"
    attrs:    0x0000000000000000
    type:    0fc63daf-8483-4772-8e79-3d69d8477de4
        (linux)
    guid:    b1a1e1f1-0001-4000-8000-000000000001
  2    0x00000222    0x00000421    "fsbl2"
    attrs:    0x0000000000000000
    type:    0fc63daf-8483-4772-8e79-3d69d8477de4
        (linux)
    guid:    b1a1e1f1-0002-4000-8000-000000000002
  3    0x00000422    0x00002421    "fip"
    attrs:    0x0000000000000000
    type:    0fc63daf-8483-4772-8e79-3d69d8477de4
        (linux)
    guid:    b1a1e1f1-0003-4000-8000-000000000003
  4    0x00002422    0x00082421    "rootfs-a"
    attrs:    0x0000000000000000
    type:    0fc63daf-8483-4772-8e79-3d69d8477de4
        (linux)
    guid:    b1a1e1f1-0004-4000-8000-000000000004
  5    0x00082422    0x00102421    "rootfs-b"
    attrs:    0x0000000000000000
    type:    0fc63daf-8483-4772-8e79-3d69d8477de4
        (linux)
    guid:    b1a1e1f1-0005-4000-8000-000000000005
  6    0x00102422    0x00102421    "app"
    attrs:    0x0000000000000000
    type:    0fc63daf-8483-4772-8e79-3d69d8477de4
        (linux)
    guid:    b1a1e1f1-0006-4000-8000-000000000006

This confirms U-Boot can read our GPT correctly. All six partitions are visible with the right names, offsets, and type GUIDs. The hex addresses convert back to our fwup.conf offsets: 0x22 = 34 (fsbl1 start), 0x2422 = 9250 (rootfs-a start), etc. If you ever suspect a partition layout issue, this is the first command to run.

mmc info — shows the SD card hardware details:

STM32MP> mmc info
Device: STM32 SD/MMC
Manufacturer ID: 3
OEM: 5344
Name: SD32G
Bus Speed: 50000000
Mode: SD High Speed (50MHz)
Rd Block Len: 512
SD version 3.0
High Capacity: Yes
Capacity: 29.7 GiB
Bus Width: 4-bit
Erase Group Size: 512 Bytes

This confirms the MMC controller is properly initialized and the card is fully detected — 29.7 GiB, running in High Speed mode at 50 MHz with 4-bit bus width. If you ever see issues with card detection or wrong capacity, this is where to look.

load mmc 0:4 0xc2000000 /boot/zImage — attempt to load the kernel:

STM32MP> load mmc 0:4 0xc2000000 /boot/zImage
Can't set block device

"Can't set block device" means U-Boot can see partition 4 in the GPT (we confirmed that with part list) but can't read files from it. That's because our rootfs is SquashFS and this U-Boot binary doesn't have SquashFS filesystem support compiled in — CONFIG_CMD_SQUASHFS=y is in our uboot.fragment which was disabled during debugging.

Other useful U-Boot debugging commands for future reference:

• printenv — show all environment variables (bootcmd, bootargs, etc.)
• mmc dev 0 / mmc rescan — switch/rescan MMC device
• md 0xC0100000 40 — memory dump (check if something was loaded)
• bdinfo — board info (RAM start/size, boot device, etc.)
• dm tree — device model tree (all detected hardware)

Re-enabling the U-Boot Config Fragment

With the debugging phase complete, it's time to re-enable our U-Boot config fragment and rebuild. The fragment adds the missing pieces:

# uboot/uboot.fragment
CONFIG_AUTOBOOT_KEYED=y
CONFIG_AUTOBOOT_PROMPT="Press any key to stop autoboot: %d\n"
CONFIG_BOOTDELAY=1
CONFIG_CMD_SQUASHFS=y          # ← lets U-Boot read our SquashFS rootfs
CONFIG_ENV_IS_IN_MMC=y         # ← tells U-Boot to read/write env from SD card
CONFIG_ENV_SIZE=0x2000         # ← 8 KB environment block
CONFIG_ENV_OFFSET=0x480000     # ← matches fwup.conf and fw_env.config

This is a targeted rebuild — only U-Boot needs recompiling, then TF-A reassembles the FIP:

# Step 1: Update .config from defconfig (picks up re-enabled fragment)
cd phase1_nerves_system/ && mix compile

# Step 2: Rebuild U-Boot and reassemble FIP
cd .nerves/artifacts/nerves_system_stm32mp157f_dk2-portable-0.1.0
make uboot-dirclean && make arm-trusted-firmware-dirclean && make

# Step 3: Build new firmware with updated U-Boot
cd phase1_test_firmware/ && mix firmware

# Step 4: Flash (always verify device first!)
ls -la /dev/sda    # must start with 'b'
sudo umount /dev/sda*
sudo $(asdf which fwup) -a -d /dev/sda -i _build/.../phase1_test_firmware.fw -t complete

After the rebuild and reflash, we can try booting again. With the config fragment active, U-Boot now has SquashFS support and can read files from our rootfs partition.

Error 21: DTB Path Mismatch — The st/ Subdirectory That Wasn't There

With SquashFS support enabled, I tried the manual boot sequence from the U-Boot prompt:

STM32MP> load mmc 0:4 0xc2000000 /boot/zImage
7296344 bytes read in 384 ms (18.1 MiB/s)

The kernel loaded! 7.3 MB, 18 MiB/s from SquashFS — that's great. Now the device tree:

STM32MP> load mmc 0:4 0xc4000000 /boot/st/stm32mp157c-dk2.dtb
** File not found /boot/st/stm32mp157c-dk2.dtb **

Wait, what? The file doesn't exist? Let me check what's actually in /boot/:

STM32MP> ls mmc 0:4 /boot/
<DIR>       4096 .
<DIR>       4096 ..
         7296344 zImage
           69251 stm32mp157c-dk2.dtb

There it is — stm32mp157c-dk2.dtb is in /boot/ directly, not in /boot/st/. The st/ subdirectory doesn't exist in the rootfs.

This is a subtle but important distinction between the kernel source tree and the installed rootfs:

• In the kernel source: the DTS file lives at arch/arm/boot/dts/st/stm32mp157c-dk2.dts. The st/ is a vendor subdirectory that Linux uses to organize device trees by manufacturer (there are also nxp/, ti/, broadcom/, etc.).
• In nerves_defconfig: we use BR2_LINUX_KERNEL_INTREE_DTS_NAME="st/stm32mp157c-dk2" — this tells Buildroot where to find the DTS in the source tree. The st/ prefix is part of the source path.
• In the rootfs: when Buildroot installs the compiled DTB, it strips the vendor prefix and puts it flat into /boot/. So st/stm32mp157c-dk2.dtb becomes just stm32mp157c-dk2.dtb.

This tripped us up because we used the st/ prefix consistently everywhere — in nerves_defconfig, in DEVICE_TREE, in DTB_FILE_NAME, and in fwup.conf's bootcmd. The first three are correct because they reference the source tree or the build system. But fwup.conf's bootcmd references the installed rootfs, where the prefix is stripped.

Here's the full picture of where st/ belongs and where it doesn't:

The fix is straightforward — in fwup.conf's on-finish section, the fdtfile environment variable needs to be just the filename without st/:

# WRONG (file not found — st/ subdir doesn't exist in rootfs):
uboot_setenv(uboot-env, "fdtfile", "st/stm32mp157c-dk2.dtb")

# CORRECT (Buildroot strips vendor prefix when installing to /boot/):
uboot_setenv(uboot-env, "fdtfile", "stm32mp157c-dk2.dtb")

We're not changing this right now because we're debugging iteratively and don't want to introduce new variables while chasing other bugs. But this fix is queued for the next rebuild.

For the manual boot, I used the correct path:

STM32MP> load mmc 0:4 0xc4000000 /boot/stm32mp157c-dk2.dtb
69251 bytes read in 8 ms (8.3 MiB/s)

Now both the kernel and device tree are loaded into RAM. Time to set the kernel command line and boot:

STM32MP> setenv bootargs console=ttySTM0,115200 root=/dev/mmcblk0p4 rootfstype=squashfs rootwait
STM32MP> bootz 0xc2000000 - 0xc4000000

Linux Boots! (And Then Crashes)

The bootz command started the kernel. For the first time, we see our custom Linux 6.6.80 kernel boot on the STM32MP157F-DK2:

Kernel image @ 0xc2000000 [ 0x000000 - 0x6f5558 ]
## Flattened Device Tree blob at 0xc4000000
   Booting using the fdt blob at 0xc4000000
Working FDT set to c4000000
   Loading Device Tree to c3f3e000, end c3f51e82 ... OK
Working FDT set to c3f3e000

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 6.6.80 (tomaz@JERINA) (arm-linux-gnueabihf-gcc.br_real (Bootlin ...
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: STMicroelectronics STM32MP157C-DK2 Discovery Board

The kernel identified the board correctly — "STM32MP157C-DK2 Discovery Board" (remember, the C and F variants are software-compatible, only the F has hardware crypto acceleration). The output continued with hardware initialization:

[    0.000000] Memory: 492572K/524288K available (14336K kernel code, ...)
[    0.000140] CPU: All CPU(s) started in SVC mode.
[    1.148058] STM32 USART driver initialized
[    1.320879] stm32-dwmac 5800a000.ethernet: User ID: 0x40, Synopsys ID: 0x50
[    1.373488] dwmac4: Master AXI performs any burst length
[    1.449581] mmc0: new high speed SDIO card at address 0001
[    1.457985] mmc1: new ultra high speed SDR104 SDHC card at address aaaa
[    1.466889] mmcblk1: mmc1:aaaa SD32G 29.7 GiB
[    1.474558] mmcblk1: p1 p2 p3 p4 p5 p6

All six of our GPT partitions were detected (p1 through p6). The kernel found the Ethernet MAC (5800a000.ethernet), the WiFi SDIO chip, and our microSD card with 29.7 GiB. Then:

[    2.206285] VFS: Mounted root (squashfs filesystem) readonly on device 179:4.

The SquashFS rootfs mounted successfully from partition 4 (179:4 = mmcblk1p4). Linux is fully up. Now erlinit takes over:

[    2.282661] Run /sbin/init as init process
erlinit: cmdline argc=6, merged argc=8
erlinit: Could not mount tmpfs on /tmp: No such file or directory
erlinit: Could not mount proc on /proc: No such file or directory
erlinit: Could not mount sysfs on /sys: No such file or directory
erlinit: No release found in /srv/erlang.
erlinit: Erlang installation not found. Check that /usr/lib/erlang exists
erlinit: FATAL ERROR. CANNOT CONTINUE.
erlinit: Not hanging due to --hang-on-exit...
erlinit: Rebooting...
[    2.309651] reboot: Restarting system

So close! Linux boots, the rootfs mounts, erlinit starts — but there's no Erlang. The error messages tell us two things:

1. "No release found in /srv/erlang" — no Elixir application release
2. "Erlang installation not found. Check that /usr/lib/erlang exists" — no Erlang runtime

My first instinct was to check Buildroot's build stamps — surely Erlang was compiled but not installed to the target? Running make erlang-show-info revealed something surprising:

"install_target": false,
"install_staging": true,

Erlang's target install is intentionally disabled. This isn't a bug — it's by design, enforced by a Nerves patch: 0008-erlang-don-t-install-to-target.patch in nerves_system_br/patches/buildroot/.

How Nerves Actually Gets Erlang Into the Rootfs

This is one of the most important architectural details to understand about Nerves, and it's not documented in any obvious place. Here's how it actually works:

Step 1: Buildroot creates a base rootfs — WITHOUT Erlang

When Buildroot runs, it compiles Erlang/OTP and installs it to staging/ only (the cross-compilation sysroot). The base rootfs.squashfs contains Linux, BusyBox, erlinit, device drivers, networking tools — but no Erlang runtime and no application code. This is the "system" artifact.

staging/usr/lib/erlang/     ← Present (for cross-compiling NIFs)
target/usr/lib/erlang/      ← Intentionally absent
images/rootfs.squashfs      ← Base rootfs, no Erlang

Step 2: mix firmware builds your Elixir release WITH embedded ERTS

When you run mix firmware in your application project, Mix creates an Elixir/OTP release that bundles the Erlang Runtime System (ERTS) inside it. The key is in mix.exs:

def release do
  [
    include_erts: &Nerves.Release.erts/0,  # ← Bundles cross-compiled ERTS
    steps: [&Nerves.Release.init/1, :assemble],
    # ...
  ]
end

Nerves.Release.erts/0 returns the path to the cross-compiled ERTS in the staging directory. So the release contains the ARM-compiled beam.smp, all the standard library .beam files, AND your application's compiled .beam files — everything the BEAM VM needs to run.

Step 3: rel2fw.sh merges the release onto the base rootfs

This is the magic step. After Mix creates the release, Nerves calls rel2fw.sh (from nerves_system_br/scripts/), which:

1. Copies the entire release into a temporary overlay at /srv/erlang
2. Runs scrub-otp-release.sh to remove unnecessary files (source code, docs, build artifacts)
3. Calls merge-squashfs to merge this overlay onto the base rootfs.squashfs
4. The result is combined.squashfs — a new SquashFS image with both the base system AND the Erlang release
5. Runs fwup to package combined.squashfs (not the original rootfs.squashfs) into the final .fw file

Base rootfs.squashfs (from Buildroot)
    + /srv/erlang/ overlay (your release + ERTS)
    = combined.squashfs (what actually goes on the SD card)
    → packaged into .fw by fwup

Step 4: erlinit finds the release at /srv/erlang

On boot, erlinit scans /srv/erlang for a release directory containing erts-*. It finds the bundled ERTS, sets up the paths, and starts the BEAM VM. There is no system-wide Erlang installation — the runtime is self-contained inside the release.

Why This Architecture?

This design is brilliant for several reasons:

1. OTA updates carry everything. When you push a firmware update, the .fw file contains the complete rootfs including your application AND the exact Erlang runtime it was compiled against. No version mismatches.

2. The system artifact is application-independent. The base rootfs.squashfs doesn't know or care what Elixir application will run on it. Different teams can share the same system artifact and build different applications on top.

3. Small firmware files. The scrub-otp-release.sh script strips documentation, source files, and unused OTP applications from the release. A typical Nerves firmware is 25-35 MB — tiny enough for OTA over cellular.

4. Reproducible builds. The ERTS version is locked by your application's Mix dependencies, not by whatever Erlang happens to be in the rootfs. Two developers building the same mix.lock get identical firmware.

So What's Actually Wrong?

Nothing is wrong with the build — this is working exactly as designed. The erlinit error we saw happens during manual boot because:

• We booted the kernel with manual U-Boot commands pointing at the base rootfs.squashfs (partition 4)
• This base image intentionally has no Erlang — it's the pre-merge image from Buildroot
• The combined.squashfs (with Erlang merged in) is what mix firmware packages into the .fw file

The real question is: when fwup writes the .fw to the SD card, is it writing the combined image (with Erlang) or the base image (without Erlang)? Let's check by looking at our fwup.conf:

file-resource rootfs.img {
    host-path = "${NERVES_SYSTEM}/images/rootfs.squashfs"
}

There's the problem. Our fwup.conf references ${NERVES_SYSTEM}/images/rootfs.squashfs — that's the base image from Buildroot, not the combined image with Erlang merged in. But wait — rel2fw.sh overrides this. Looking at line 181:

ROOTFS="$TMP_DIR/combined.squashfs" $FWUP -c -f "$FWUP_CONFIG" -o "$FW_FILENAME"

The ROOTFS environment variable is set to the combined squashfs. This means fwup.conf can reference ${ROOTFS} to get the merged image. Let's check if our fwup.conf uses this variable... it doesn't. It hardcodes ${NERVES_SYSTEM}/images/rootfs.squashfs.

Looking at other Nerves systems (like nerves_system_rpi4), their fwup.conf files use:

file-resource rootfs.img {
    host-path = "${ROOTFS}"
}

Not ${NERVES_SYSTEM}/images/rootfs.squashfs. The ROOTFS variable is set by rel2fw.sh to point at the combined SquashFS — the one with Erlang merged in.

Error 22: fwup.conf Uses Base Rootfs Instead of Combined Rootfs

Root cause: Our fwup.conf hardcodes the path to the base rootfs.squashfs from Buildroot, which doesn't contain Erlang. It should use the ${ROOTFS} environment variable, which rel2fw.sh sets to the merged SquashFS that contains both the base system and the Erlang release.

Fix:

# WRONG — base image without Erlang:
file-resource rootfs.img {
    host-path = "${NERVES_SYSTEM}/images/rootfs.squashfs"
}

# CORRECT — combined image with Erlang release merged in:
file-resource rootfs.img {
    host-path = "${ROOTFS}"
}

This one-line change is all that's needed. The ROOTFS variable is set by rel2fw.sh during mix firmware, pointing to the temporary combined.squashfs that has your Elixir release (including ERTS) merged onto the base Buildroot rootfs.

The Missing Directories: /tmp, /proc, /sys

The erlinit errors about missing /tmp, /proc, and /sys were a red herring from the earlier boot with the base rootfs image. Once the proper combined rootfs (with Erlang merged in) was flashed, these errors disappeared — the Nerves skeleton creates these mount points correctly.

It Boots! Nerves on the STM32MP157F-DK2

After fixing fwup.conf to use ${ROOTFS} and rebuilding the firmware, the moment of truth:

Erlang/OTP 28 [erts-16.3] [source] [32-bit] [smp:2:2] [ds:2:2:10] [async-threads:1]

(phase1_test_firmware@nerves-stm32mp1)1>

Nerves is running. The IEx prompt appears over the serial console. Let's break down what the boot log tells us:

Hardware detected:

• Both Cortex-A7 CPUs active (smp:2:2 — 2 schedulers, 2 online)
• 512 MiB DDR3L memory (424 MB available to Linux after kernel + CMA reservation)
• Ethernet: RTL8211F PHY at 1 Gbps/Full duplex, flow control active
• Display: STM DRM framebuffer (60x50 text console), Vivante GC400 GPU initialized
• Touch: EDT FT5x06 capacitive touchscreen on I2C
• USB: EHCI host controller + DWC2 OTG, USB hub detected
• Storage: 29.7 GiB microSD, all 6 GPT partitions recognized
• RTC: STM32 RTC (date needs initialization — expected, no battery backup)
• Crypto: STM32 CRC32 + HASH accelerator (the F variant's hardware crypto!)
• Remoteproc: M4 coprocessor registered (ready for Phase 2!)
• PMIC: STPMIC1 v0x21 managing power rails

Minor warnings to address later:

• latin1 locale — need to add --env ELIXIR_ERL_OPTIONS=+fnu to erlinit.config
• nerves_initd not found — can remove --pre-run-exec /usr/bin/nerves_initd from erlinit.config
• logger :backends deprecated — Elixir 1.19 config style change, update config.exs
• Read outside partition 2 — U-Boot env redundant copy issue, doesn't affect operation once manually booted

What's still manual: U-Boot's autoboot doesn't work yet because the environment prepopulated by fwup can't be read (the "Read outside partition 2" issue). We're booting via manual U-Boot commands for now. The autoboot fix is a U-Boot env configuration issue that we'll resolve in the next session.

The Full Boot Chain, Working

Let me trace the complete path from power-on to IEx prompt — this is everything we built in Phase 1:

Power on
  → ROM bootloader (STM32 internal) reads GPT, finds "fsbl1" partition
  → TF-A BL2 (tf-a-stm32mp157c-dk2.stm32) initializes DDR, clocks, PMIC
  → SP_MIN (BL32, inside FIP) sets up secure monitor
  → U-Boot (BL33, inside FIP) initializes MMC, Ethernet, serial
  → [manual: load kernel + DTB from SquashFS partition 4]
  → Linux 6.6.80 boots, mounts SquashFS rootfs readonly
  → erlinit (PID 1) finds release in /srv/erlang
  → BEAM VM starts with 2 schedulers on 2 CPUs
  → Elixir application starts (shoehorn → phase1_test_firmware)
  → IEx prompt on ttySTM0 (serial console via ST-LINK)

Every link in this chain — from the first-stage bootloader to the IEx prompt — is something we configured, debugged, and understood. That's the whole point of building a custom Nerves system from scratch.

Fixing the Minor Warnings — And One More Kernel Bug

With Nerves booting and the IEx prompt working, I went back to fix the minor warnings from the initial boot. Most were quick config changes. But one of them uncovered another missing kernel config — and this one taught me something about how VintageNet works under the hood.

Fix: IEx Instead of Eshell

The first boot dropped me into an Erlang shell (Eshell) instead of the Elixir IEx prompt I expected. This is because Elixir >= 1.17 changed how it starts the interactive shell. You need explicit VM flags in rel/vm.args.eex:

## Force UTF-8 native encoding (prevents latin1 warning)
+fnu

## Start the Elixir IEx shell (required for Elixir >= 1.17)
-noshell
-user elixir
-run elixir start_cli
-extra --no-halt

Without these, the BEAM VM starts its default Erlang shell. The -user elixir flag tells it to use Elixir's shell handler instead, and -run elixir start_cli kicks off the IEx CLI. The +fnu flag forces UTF-8 filename encoding — without it, Erlang defaults to latin1 and you get a warning on every boot.

Fix: nerves_initd Not Found

The erlinit.config had --pre-run-exec /usr/bin/nerves_initd, but this binary doesn't exist in our rootfs. This comes from the official Nerves systems (like nerves_system_rpi4) that include it, but our custom system doesn't build it. Removed the line — erlinit starts fine without it.

Fix: Logger :backends Deprecated

Elixir 1.19 deprecated the :backends configuration for Logger. The old config:

config :logger, backends: [RingLogger]

needed to be replaced. But here's the catch — you need two config lines, not one:

# Disable the default console handler (no console on embedded)
config :logger, :default_handler, false
# Add RingLogger as a handler (new Elixir 1.19 style)
config :logger, {:handler, :ring_logger, RingLogger, %{}}

The first line disables the default handler (which would try to write to stdout — not useful on embedded). The second adds RingLogger as a proper Erlang logger handler. I initially only had the first line, which is why RingLogger.next said "The RingLogger backend isn't running" on the first boot attempt. Without the handler config, RingLogger was loaded but never started.

Error 23: VintageNet Crashes — "Rule Family Not Supported"

This one was the most interesting. After fixing the minor issues, I tried to get networking working. VintageNet was configured in target.exs:

config :vintage_net,
  config: [
    {"eth0", %{type: VintageNetEthernet, ipv4: %{method: :dhcp}}}
  ]

But VintageNet.info() said "loaded, but not started." When I manually started it with Application.start(:vintage_net), it actually got an IP address via DHCP — 192.168.44.179 — proving the Ethernet hardware was fully working. Then it immediately crashed:

[error] Failed to update IP routing table due to Error: Rule family not supported.

Check that your Linux config includes:
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y

[error] GenServer VintageNet.RouteManager terminating
** (RuntimeError) Error: Rule family not supported.

VintageNet uses Linux policy routing to manage network interfaces. Policy routing goes beyond the simple routing table most people know about. Instead of one global routing table, you can have multiple tables and rules that select which table to consult for each packet. This is what ip rule commands manage.

Why does VintageNet need this? Because on an embedded device you might have Ethernet, WiFi, and a cellular modem — all with default gateways. VintageNet uses policy routing to prioritize interfaces: Ethernet gets priority 100, WiFi gets 200, cellular gets 300. Without multiple routing tables, you can't express "use Ethernet's gateway if it's up, otherwise fall back to WiFi."

The kernel options that enable this:

The standard multi_v7_defconfig doesn't enable these — they're not needed for a simple desktop or server setup. But Nerves (via VintageNet) absolutely requires them.

The fix was simple — add two lines to linux-6.6.defconfig:

# VintageNet uses Linux policy routing (ip rule) to manage multiple interfaces.
# Without these, VintageNet.RouteManager crashes with "Rule family not supported"
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y

But this means a kernel recompile and reflash. No way around it — the routing subsystem is compiled into the kernel, not a loadable module.

The debugging process here was actually satisfying. The DHCP worked perfectly — our Ethernet driver, PHY driver, and network stack were all fine. The crash was specifically in the routing layer, and VintageNet's error message told us exactly what was missing. No guessing, no shotgun debugging. Just read the error, add the config, rebuild.

What These Fixes Mean for Custom Nerves Systems

If you're building a custom Nerves system for a new board, here's the checklist of things that the official Nerves systems (like nerves_system_rpi4) have that you'll also need:

1. Kernel config: CONFIG_IP_ADVANCED_ROUTER=y and CONFIG_IP_MULTIPLE_TABLES=y — VintageNet won't work without these
2. vm.args.eex: IEx startup flags (-noshell -user elixir -run elixir start_cli -extra --no-halt) — without these you get Eshell instead of IEx
3. vm.args.eex: +fnu — forces UTF-8 filename encoding
4. Logger config: config :logger, backends: [RingLogger] — yes, it's deprecated in Elixir 1.19, but it still works. The new handler API ({:handler, :ring_logger, RingLogger, %{}}) doesn't work with Config.config/2 because it expects a keyword list, not a tuple. Accept the deprecation warning for now.
5. fwup.conf: Use ${ROOTFS}, never ${NERVES_SYSTEM}/images/rootfs.squashfs — the base image doesn't contain Erlang

These are the kind of things you'd never know from reading documentation alone. You find them by building, booting, and watching what breaks.

Error 24: U-Boot Autoboot — Works on This Card, Not on Fresh Flash

After fixing networking, there was one more annoyance: U-Boot wouldn't autoboot. It would count down, try to boot, fail to read the saved environment, and drop to the STM32MP> prompt. I had to manually type the boot commands every time.

My first fix was saveenv from the U-Boot prompt — save the bootcmd to MMC, and it persists across reboots. But that only helps this specific SD card. Flash a new card or give it to a colleague, and they'd have to type the commands manually too. Not acceptable for a system you want to share.

The proper fix: compile the bootcmd into U-Boot itself using CONFIG_BOOTCOMMAND. This sets the default bootcmd — the one U-Boot uses when there's no saved environment. Add to uboot/uboot.fragment:

CONFIG_BOOTCOMMAND="sqfsload mmc 0:4 ${kernel_addr_r} boot/zImage; sqfsload mmc 0:4 ${fdt_addr_r} boot/stm32mp157c-dk2.dtb; setenv bootargs root=/dev/mmcblk0p4 rootfstype=squashfs console=ttySTM0,115200; bootz ${kernel_addr_r} - ${fdt_addr_r}"

This means rebuilding U-Boot (which means rebuilding the FIP, since U-Boot is packed inside), then rebuilding firmware and reflashing. But the result is worth it — every fresh flash autoboots without manual intervention.

Ethernet and SSH — Task 11 Complete

After the kernel rebuild with policy routing support and the U-Boot autoboot fix, everything came together. Power on, U-Boot counts down, loads the kernel from SquashFS, Linux boots, erlinit starts the BEAM, VintageNet configures Ethernet via DHCP, and within 15 seconds of power-on we have a working IEx prompt with network connectivity:

iex(phase1_test_firmware@nerves-stm32mp1)> VintageNet.info()
VintageNet 0.13.9

All interfaces:       ["eth0", "lo", "sit0"]
Available interfaces: ["eth0"]

Interface eth0
  Type: VintageNetEthernet
  Present: true
  State: :configured (0:01:13)
  Connection: :internet (0:01:08)
  Addresses: fe80::12e7:7aff:fee1:a22b/64, 192.168.44.179/24
  MAC Address: "10:e7:7a:e1:a2:2b"
  Configuration:
    %{type: VintageNetEthernet, ipv4: %{method: :dhcp}}

The board has:

• Ethernet: 1 Gbps via RTL8211F PHY, DHCP, internet connectivity confirmed
• SSH: Available via nerves_ssh (configured with user's public key)
• mDNS: Advertised as nerves-stm32mp1.local via mdns_lite
• IEx: Full Elixir interactive shell over serial console
• All hardware detected: Display (DRM/DSI), touchscreen (EDT FT5x06), GPU (Vivante GC400), M4 coprocessor (remoteproc), crypto accelerator, USB hub, WiFi SDIO module

Deferred probes (normal, not errors):

• 10000000.m4 — M4 coprocessor firmware not loaded yet (Phase 2)
• 0-004a — Audio codec needs additional driver config
• sound — Depends on audio codec
• vdda: disabling — Unused voltage regulator being powered down (saves power)

Error 25: App Partition Not Mounted — SSH Host Keys Can't Persist

Even with VintageNet working and DHCP assigning an IP, SSH still failed:

[error] [NervesSSH] :ssd.daemon failed: {:error, ~c"No host key available"}
[error] /dev/mmcblk0p6: Can't lookup blockdev
[error] Executable mkfs.ext4 was not found. The Nerves System must be fixed to include it!
[warn]  [NervesSSH] Failed to save authorized_keys file: {:error, :erofs}

Three cascading failures:

1. /dev/mmcblk0p6 doesn't exist — The kernel couldn't enumerate the partition. Our fwup.conf defined APP_PART_COUNT = 0 (meaning "fill remaining space"), but the kernel may not handle this correctly for the last GPT partition.

2. mkfs.ext4 not found — Even if the device existed, nerves_runtime tried to format it on first boot and couldn't find the mkfs binary. Our Buildroot config didn't include any filesystem creation tools.

3. :erofs (read-only filesystem) — Without a mounted writable partition, NervesSSH falls back to /tmp for host keys, but can't persist them across reboots. Every reboot generates new host keys, and SSH clients reject the connection with "host key changed" warnings.

This is the kind of issue you'd never hit with official Nerves systems because they've already solved it. But building from scratch means you discover why each piece exists.

How Official Nerves Systems Handle This

Comparing against nerves_system_rpi5, the official approach has three layers:

Layer 1: erlinit mounts the partition early. The rpi5's erlinit.config has:

-m /dev/mmcblk0p7:/root:f2fs::

This mounts the app partition before the BEAM VM starts, so by the time nerves_runtime and nerves_ssh initialize, /root is already writable.

Layer 2: Buildroot includes filesystem tools. The rpi5's nerves_defconfig has BR2_PACKAGE_F2FS_TOOLS=y. If the partition is unformatted (first boot on a fresh flash), nerves_runtime can format it automatically using mkfs.f2fs.

Layer 3: fwup.conf sets the partition metadata. The U-Boot environment includes three variables per slot:

a.nerves_fw_application_part0_devpath = /dev/mmcblk0p7
a.nerves_fw_application_part0_fstype  = f2fs
a.nerves_fw_application_part0_target  = /root

These tell nerves_runtime which device to mount, what filesystem to expect, and where to mount it.

We had only layer 3 (the U-Boot env vars). We were missing layers 1 and 2.

Why f2fs instead of ext4? F2FS (Flash-Friendly File System) was designed by Samsung specifically for flash storage — SD cards, eMMC, SSDs. It handles wear leveling and flash-specific I/O patterns better than ext4, which was designed for spinning disks. Every official Nerves system uses f2fs for the app partition.

The Full Fix

1. Add BR2_PACKAGE_F2FS_TOOLS=y to nerves_defconfig (provides mkfs.f2fs)
2. Change fwup.conf env vars from ext4 to f2fs
3. Set APP_PART_COUNT to a real value instead of 0
4. Add -m /dev/mmcblk0p6:/root:f2fs:: to erlinit.config
5. Add cgroup mounts (like rpi5) for process management
6. Rebuild system + firmware, reflash with -t complete

What Else Was Missing — The Full rpi5 Comparison

After fixing the app partition, I did a systematic comparison of every file in our system against nerves_system_rpi5. Here's everything we were still missing from our erlinit.config:

And missing from the rootfs overlay:

The lesson: a Nerves system isn't just "Linux boots and Erlang runs." It's a complete embedded platform with persistent storage, crash recovery, terminal handling, and developer experience. The official systems have years of polish baked in. Building from scratch means rediscovering each of these pieces — which is exactly the point of this project.

OTA Updates — The Moment of Truth

With SSH working, the next milestone was OTA (Over-The-Air) firmware updates. No more pulling the SD card, no more fwup -t complete. Just mix upload nerves-stm32mp1.local from my development machine. This is the whole point of Nerves — iterate on firmware without physically touching the board.

The first OTA went smoothly. Build firmware, upload, board reboots into the new version on slot B. Validation works — Nerves.Runtime.validate_firmware() sets nerves_fw_validated=1 in the U-Boot env. Revert works — Nerves.Runtime.revert() switches back to slot A and reboots. The A/B partition scheme proved itself immediately.

But then things got interesting.

Modeling After osd32mp1 — "Just LOOK at Official Repos!"

My initial fwup.conf was functional but rough. Missing safety guards, no delta update support, no platform verification on upgrades. After some frustrating debugging cycles, I decided to properly model everything after the official nerves_system_osd32mp1 — the closest existing Nerves system to our hardware (also an STM32MP1 board).

This was a turning point. Instead of inventing solutions, I studied how the osd32mp1 system handles things:

Shared configuration via fwup_include/:

The osd32mp1 uses an fwup_include/fwup-common.conf file that both fwup.conf and fwup-revert.conf include. This keeps partition offsets, metadata defaults, and UUIDs in one place. I created the same structure:

phase1_nerves_system/
  fwup.conf                      # Main firmware config
  fwup-revert.conf               # Revert-only config
  fwup_include/
    fwup-common.conf             # Shared defines (both include this)
    provisioning.conf            # Device-specific provisioning

Safety tasks that prevent common mistakes:

task upgrade.unvalidated {
    require-uboot-variable(uboot-env, "nerves_fw_validated", "0")
    on-init { error("Please validate the running firmware before upgrading it again.") }
}

task upgrade.wrong {
    require-uboot-variable(uboot-env, "a.nerves_fw_platform", "${NERVES_FW_PLATFORM}")
    require-uboot-variable(uboot-env, "a.nerves_fw_architecture", "${NERVES_FW_ARCHITECTURE}")
    on-init { error("Please check the media being upgraded.") }
}

task upgrade.wrongplatform {
    on-init { error("Expecting platform=${NERVES_FW_PLATFORM} and architecture=${NERVES_FW_ARCHITECTURE}") }
}

fwup evaluates tasks in order and uses the first one whose require-* conditions match. So these safety tasks act as guards: if the firmware isn't validated yet, block the upgrade. If the platform doesn't match, block the upgrade. Only if all guards pass does the actual upgrade.a or upgrade.b task execute.

I hit the upgrade.unvalidated guard the hard way — tried to do an OTA before validating, and fwup refused. That's the safety working correctly.

Revert configuration — simpler than expected:

The osd32mp1's fwup-revert.conf does NOT have a "validate" task. Modern nerves_runtime (0.13+) validates firmware by writing nerves_fw_validated=1 directly to the U-Boot env via the UBootEnv Elixir library. No fwup involvement needed. The revert.fw only needs two tasks: revert.a and revert.b, plus error guards.

post-build.sh generates revert.fw:

# Copy fwup_include to images dir so fwup can resolve include() paths
cp -rf "$NERVES_DEFCONFIG_DIR/fwup_include" "$BINARIES_DIR"

# Compile revert.fw into the rootfs at /usr/share/fwup/revert.fw
NERVES_SDK_IMAGES="$BINARIES_DIR" "$HOST_DIR/usr/bin/fwup" \
    -c -f "$NERVES_DEFCONFIG_DIR/fwup-revert.conf" \
    -o "$TARGET_DIR/usr/share/fwup/revert.fw"

This is the file that Nerves.Runtime.revert() applies at runtime.

The fwup ${} Trap — The Hardest Bug to Find

Here's a bug that took hours to track down and taught me the most important thing about fwup configuration.

I wanted to set the U-Boot bootcmd via fwup — a smart bootcmd that handles A/B slot switching and boot-counting auto-revert. Something like:

if test x${nerves_fw_active} = xb; then setenv rootpart 5; else setenv rootpart 4; fi;
sqfsload mmc 0:${rootpart} 0xc2000000 boot/zImage; ...

So I put this in fwup.conf:

uboot_setenv(uboot-env, "bootcmd", "if test x${nerves_fw_active} = xb; then ...")

Built, flashed, booted. The board booted fine. But when I SSH'd in and checked the bootcmd via Nerves.Runtime.KV.get("bootcmd"), I saw:

if test x = xb; then setenv rootpart 5; else setenv rootpart 4; fi;
sqfsload mmc 0:  boot/zImage; ...

Every ${variable} reference was gone. Empty strings. The x prefix trick (which prevents U-Boot syntax errors on empty vars) was the only reason it didn't crash — test x = xb is just false, so it always fell through to setenv rootpart 4. And sqfsload mmc 0: happened to auto-detect the SquashFS partition. So the system booted, hiding the bug.

The root cause: fwup evaluates ${...} at compile time — when you run fwup -c to create the .fw file. It's fwup's own variable substitution, not something deferred to U-Boot. So ${nerves_fw_active} was evaluated by fwup (to empty string, since it's not a fwup variable), not passed through to U-Boot.

fwup does have \${...} syntax to defer evaluation, but that defers to fwup's apply time (when the .fw is written to the SD card), not to U-Boot's runtime. There is simply no way to write literal ${var} into a U-Boot env string via fwup's uboot_setenv.

This is why the osd32mp1 system doesn't set bootcmd via fwup either. The bootcmd goes into CONFIG_BOOTCOMMAND in the U-Boot config fragment — compiled directly into the U-Boot binary.

Boot-Counting Auto-Revert — Protecting Against Bad Firmware

The smart bootcmd isn't just about A/B slot switching. It implements boot-counting auto-revert, which is critical for OTA safety:

# Phase 1: Check for failed boot (revert if needed)
if test x${nerves_fw_booted} = x1; then
  if test x${nerves_fw_validated} != x1; then
    echo Reverting firmware...;
    # Switch to other slot
    if test x${nerves_fw_active} = xb; then
      setenv nerves_fw_active a;
    else
      setenv nerves_fw_active b;
    fi;
    setenv nerves_fw_validated 1;
    setenv nerves_fw_booted 1;
    saveenv;
  fi;
fi;

# Phase 2: Mark boot attempt
if test x${nerves_fw_booted} != x1; then
  setenv nerves_fw_booted 1;
  saveenv;
fi;

# Phase 3: Select partition and boot
if test x${nerves_fw_active} = xb; then
  setenv rootpart 5;
else
  setenv rootpart 4;
fi;
sqfsload mmc 0:${rootpart} 0xc2000000 boot/zImage;
sqfsload mmc 0:${rootpart} 0xc4000000 boot/stm32mp157c-dk2.dtb;
setenv bootargs console=ttySTM0,115200 root=/dev/mmcblk0p${rootpart}
  rootfstype=squashfs rootwait loglevel=4;
bootz 0xc2000000 - 0xc4000000

The flow for a normal OTA:

1. Upgrade task writes new rootfs to slot B, sets nerves_fw_active=b, nerves_fw_booted=0, nerves_fw_validated=0
2. Reboot → U-Boot checks: booted=0, so sets booted=1, saves env, boots slot B
3. New firmware starts, application calls Nerves.Runtime.validate_firmware() which sets validated=1
4. Next reboot → U-Boot checks: booted=1 and validated=1 → all good, boot normally

The flow for a bad OTA (firmware crashes before validating):

1. Same as above through step 2
2. New firmware crashes, board reboots (or panic=10 triggers reboot)
3. U-Boot checks: booted=1 and validated!=1 → REVERT! Switches back to previous slot, saves env
4. Old working firmware boots

The x prefix on every test is an old POSIX/U-Boot trick: if ${nerves_fw_booted} is undefined or empty, test x${nerves_fw_booted} = x1 becomes test x = x1 (false, but valid syntax) instead of test = 1 (syntax error).

The BootcmdMigration Module — Working Around fwup's Limitation

Since fwup can't write the smart bootcmd, and OTA can't update U-Boot, we need another way. The solution: an Elixir module that installs the bootcmd on first boot via Nerves.Runtime.KV.put, which writes raw strings to the U-Boot env without any ${} evaluation.

defmodule MojPrviStmFirmware.BootcmdMigration do
  require Logger

  @smart_bootcmd "if test x${nerves_fw_booted} = x1; then " <>
                   "if test x${nerves_fw_validated} != x1; then " <>
                   # ... full bootcmd with literal ${} references preserved ...
                   "bootz 0xc2000000 - 0xc4000000"

  def run do
    case Nerves.Runtime.KV.get("bootcmd") do
      val when is_binary(val) and byte_size(val) > 0 ->
        if String.contains?(val, "nerves_fw_active") do
          Logger.info("[BootcmdMigration] Smart bootcmd already installed")
        else
          install_smart_bootcmd()
        end
      _ ->
        install_smart_bootcmd()
    end
  end

  defp install_smart_bootcmd do
    Logger.info("[BootcmdMigration] Installing smart bootcmd")
    Nerves.Runtime.KV.put("bootcmd", @smart_bootcmd)
  end
end

Elixir's <> string concatenation preserves the literal ${nerves_fw_active} because there's no template engine evaluating it. The module runs at application startup (before the supervisor tree) and checks if the bootcmd already contains nerves_fw_active — if yes, it's already the smart version.

This is called from application.ex:

def start(_type, _args) do
  MojPrviStmFirmware.BootcmdMigration.run()
  # ... supervisor setup
end

The Mystery — U-Boot Ignores Its Own Saved Env

Here's the open puzzle. The BootcmdMigration successfully writes the 711-byte smart bootcmd to the U-Boot env. I can read it back via Nerves.Runtime.KV.get("bootcmd") and see the full command with all ${var} references intact.

But U-Boot doesn't use it.

The evidence: after reboot, nerves_fw_booted is still 0 (the smart bootcmd would set it to 1). The kernel command line shows root=/dev/mmcblk0p4 rootfstype=squashfs console=ttySTM0,115200 — missing rootwait and loglevel=4 that the smart bootcmd would add. U-Boot is executing the old CONFIG_BOOTCOMMAND compiled into the binary, not the saved env's bootcmd.

Most likely cause: the UBootEnv Elixir library and U-Boot 2025.10 use incompatible env serialization formats (different CRC or header structure). U-Boot reads the env, sees an invalid CRC, and falls back to the compiled-in defaults. The Elixir library reads its own writes successfully because it uses its own parser.

The practical fix: do a -t complete reflash with the updated U-Boot that has the smart bootcmd as CONFIG_BOOTCOMMAND. Then auto-revert works from the first boot regardless of the saved env format.

This mystery remains open and is tracked for investigation — it matters for OTA scenarios where the U-Boot binary can't be updated.

What Belongs in the Base System vs. the Firmware App?

While debugging all this, I had a moment of clarity about the Nerves architecture. When you do mix nerves.new with an official system like rpi4, you get a working firmware out of the box — OTA, SSH, A/B updates, validation, revert, all working. You focus on your application logic.

For our custom system, these features split into two categories:

Base system (in phase1_nerves_system/):

• Partition layout and fwup tasks (fwup.conf, fwup-revert.conf)
• U-Boot bootcmd (CONFIG_BOOTCOMMAND in uboot.fragment)
• Filesystem tools (f2fs-tools in defconfig)
• erlinit configuration
• revert.fw generation (post-build.sh)

Firmware app workaround (in moj_prvi_stm_firmware/):

• BootcmdMigration — because fwup can't write ${var} to U-Boot env

Everything except BootcmdMigration is a base system concern. Anyone building a firmware with mix nerves.new on this system should get OTA, validation, and revert for free — same as with official systems. The BootcmdMigration is our system's quirk until we solve the U-Boot env format mystery.

Phase 1 Complete — With Polish Pending

The core Phase 1 goal is achieved: a custom Nerves system that boots on the STM32MP157F-DK2 with networking and SSH. But "boots and works" is different from "production-ready." The remaining polish (writable app partition, MOTD, OTA testing, git packaging) is tracked as Phase 1b and will be addressed before moving to Phase 2 (M4 coprocessor).

The big open item: boot-counting auto-revert doesn't work yet because U-Boot ignores the saved env's bootcmd. The next -t complete flash with the updated U-Boot binary should fix this. Until then, manual revert via Nerves.Runtime.revert() works perfectly.