dm-verity on a signed rootfs, and two console messages that were simply untrue

I set out to close the last hole in the boot chain. I spent most of the day discovering that two things I'd been reading on the console for weeks were simply not true.

The hole#

Where I'd got to: bootloader verified, secure monitor verified, U-Boot verified, kernel verified. All of it proven by breaking it on purpose and watching the board refuse.

And then the board mounts a root filesystem nobody has checked at all.

That's where the BEAM lives. My Elixir release. The UI. Everything that is actually my product sits on the one partition with no protection on it whatsoever. I have a demo image from a few weeks back called TAMPERED-ROOTFS_BOOTS-ANYWAY that exists purely to make this point honestly, and it does exactly what the name says.

The fix is dm-verity. Hash every 4 KB block of the filesystem, hash those hashes, keep going until you're left with one value — a Merkle tree and its root hash. At runtime the kernel re-hashes each block as it reads it and walks the tree. Two things I like about it: nothing is checked at boot, so a 52 MB filesystem costs nothing at startup, and the integrity of the whole thing collapses to 32 bytes. Protect those, and you've protected everything.

That last sentence is the entire problem.

You can't put the hash of a filesystem inside that filesystem#

The 32 bytes have to live somewhere an attacker can't rewrite. The only signed thing I have is the FIT image holding the kernel. So: put the hash in the FIT.

Except the FIT is /boot/kernel.itb — a file inside the filesystem I'm hashing. Change the FIT, and the filesystem changes, and the hash changes, and now the FIT is wrong. Round and round. There is no fixed point.

I'm glad I noticed that before writing code rather than after. The escape is to move the kernel image out of the root filesystem into its own partition, which is a bigger change than I wanted today, so I split the work: get verity working first with the hash somewhere convenient, then go back and make it trustworthy. Same staging that worked for the kernel signing.

And I want to be blunt about what "somewhere convenient" means, because this is exactly the kind of thing that gets quietly glossed over: for now, the root hash rides on the kernel command line, which lives in the U-Boot environment, which is unsigned and writable. Someone who can rewrite my root filesystem can rewrite the hash that judges it, in the same sitting. What today's work actually buys is that a naive tamper — rewrite the filesystem, touch nothing else — now fails. That's real, and it's not the same as tamper-proof. The boot log will say dm-verity is active either way, which is precisely why I'm writing this paragraph.

Lie number one: "Press any key to stop autoboot"#

I needed the U-Boot shell to test something. So: reset, and start hammering keys at the prompt.

Nothing. It boots straight past. It doesn't even count down to zero — it prints 1 and goes.

I assumed my serial terminal wasn't sending keystrokes. It was; I'd been typing into the Elixir shell over the same cable all morning. Then I assumed one second was just too short a window, which is almost right and completely misleading.

Here's what's actually in the config:

CONFIG_AUTOBOOT_KEYED=y
CONFIG_AUTOBOOT_STOP_STR=""
CONFIG_AUTOBOOT_PROMPT="Press any key to stop autoboot: %d\n"

With AUTOBOOT_KEYED turned on, U-Boot doesn't stop on any key. It stops when you type an exact stop string. Mine is empty. And in the source:

delaykey[i].len = delaykey[i].str == NULL ? 0 : strlen(delaykey[i].str);
...
if (delaykey[i].len > 0 && presskey_len >= delaykey[i].len && memcmp(...) == 0)

strlen("") is zero. That condition can never be true. No key has ever stopped autoboot on this board.

And the prompt? It's CONFIG_AUTOBOOT_PROMPT, a plain printf format string that has no connection whatsoever to the logic. It would cheerfully print "Press any key to stop autoboot" if the stop string were hunter2.

I don't mind a misconfiguration. What bothers me is a console message that confidently describes behaviour the code doesn't implement — it sent me looking at my terminal, my cable, and my timing before I ever suspected the config. The countdown detail, too: %d prints the current value, so with a one-second delay you see 1 and that's it. There is no zero. That looked like a symptom. It's just arithmetic.

Two fixes. Right now, without rebuilding: the same function reads bootstopkey from the environment before falling back to the compiled value, and Nerves can write U-Boot variables from Elixir, so one line from the IEx prompt made autoboot interruptible. And for good: AUTOBOOT_KEYED off, delay bumped to three seconds. Now "any key" means any key and the prompt tells the truth.

Worth adding, because it's tempting to call the old state a security feature: it isn't. Blocking the U-Boot shell on a shipped device is a legitimate thing to want — but the environment is writable and bootstopkey overrides the compiled value, so anyone who can write it turns interruption straight back on. Accidental hardening isn't hardening.

Lie number two: the quotes#

To switch on dm-verity without an initramfs you pass the kernel a parameter that contains the whole device-mapper table:

dm-mod.create="vroot,,,ro,0 102448 verity 1 /dev/mmcblk0p4 ..."

Note the spaces. The kernel's command-line parser splits on whitespace, so those quotes aren't decoration — without them the kernel gets dm-mod.create=vroot,,,ro,0 and nothing else.

So I built the thing carefully: store the quoted parameter in a U-Boot variable, have the boot script paste it into the command line. I reasoned about how U-Boot's shell handles quoting and convinced myself it would survive.

Then I did something I'm glad I did. Instead of flashing it, I stopped at the U-Boot prompt and typed it out by hand.

STM32MP> printenv ba
ba=console=ttySTM0,115200 dm-mod.create=vroot,,,ro,0 1 2 3 root=/dev/dm-0

Gone. The kernel would have taken dm-mod.create=vroot,,,ro,0, failed to build the device, and panicked with no root filesystem.

Second attempt: escape the quotes with backslashes. Also gone — the shell treats the escaped quote as a real delimiter and removes it.

What finally told me where the problem was:

STM32MP> printenv t
t=dm-mod.create="vroot,,,ro,0 1 2 3"

The quotes were stored perfectly. The loss happens on expansion — the shell substitutes the variable and then strips quotes from the result. Which means the answer isn't better quoting. It's to never let the quoted text pass through ${...} at all.

So the flashing tool now writes an entire command into the environment — setenv bootargs console=... 'dm-mod.create="..."' root=/dev/dm-0 — and the boot script just runs it. The tool writes the environment directly, with no shell involved, so both kinds of quote land byte for byte. Running it is identical to typing it, and typing it works. I rehearsed that too, on the board, before flashing anything.

Why I bothered testing instead of flashing#

Because the boot script is shared by both firmware slots.

The A/B rollback that rescued me twice last week rescues you from a bad kernel. It cannot rescue you from a bad boot script, because the boot script is the thing that performs the rollback. Break it and both slots are unbootable, and the only way back is an SD card and a card reader.

Two wrong designs, both of which I'd reasoned my way into, caught in about four minutes at a prompt. That's the cheapest afternoon I've had on this project.

Four attempts#

I'll spare you the blow-by-blow, but the shape is worth having, because every single failure was a thing that looked configured correctly.

Attempt one: verity: Data device lookup failed (-ENODEV), three quarters of a second in. The code that sets up the verified device runs early in boot — earlier than the SD card driver finishes finding its partitions. There's a companion setting whose entire job is to wait for a named device first. Added it.

Attempt two: verity: metadata block 12806 is corrupted. Block 12806 was precisely where I'd told the kernel the hash tree began. I dumped that block, and it isn't the tree — it's a little header the hashing tool writes in front of it, with the algorithm and salt in plain sight. The tree starts one block later. The kernel doesn't know or care about that header; you have to point past it.

And here's the bit that stung: my offline check had passed on that broken image. The verification tool, given an offset, reads that header and works out where the tree is by itself. The kernel doesn't. Two different readings of the same bytes, and the friendlier tool hides the difference. I now run the check in its strict mode, where you supply every parameter by hand and it reads the disk the way the kernel does — and the build refuses to produce firmware that fails it.

Attempt three is the one I'll remember. It booted. Reached a shell. Everything fine. Then I rebooted and it died: Data device lookup failed (-ENXIO).

Same card. Same settings. Byte-identical command line — I checked, at the prompt.

The error code was the clue, and I'd skimmed past it. The function that looks up a device by name only ever reports "no such device". -ENXIO is different: it means the name was resolved and then opening it failed. Which shouldn't be possible... unless the name was resolved to something that doesn't exist.

And that's exactly what happens. If you ask for partition 4 and partition 4 isn't registered yet, the kernel strips the number off, looks up the disk, and hands back a device number for partition 4 anyway — on the reasoning that it might show up once someone opens the disk. So my "wait for partition 4" was satisfied the instant the card appeared, which is far too early. Pure race. First boot after a flash won it; the next one didn't.

The fix is to wait on the partition's UUID instead of its name. That lookup has no such optimism: no real partition, no answer.

It works#

device-mapper: init: waiting for device PARTUUID=b1a1e1f1-0004-... ...
device-mapper: init: all devices available
device-mapper: verity: sha256 using implementation "stm32-sha256"
device-mapper: ioctl: dm-0 (vroot) is ready

Every block of the root filesystem is now hashed as it's read and checked against a tree whose root is 32 bytes. Two boots in a row, clean.

I got a present I wasn't expecting, too. Look at stm32-sha256 in that third line — the hashing isn't happening in software. This chip has a SHA-256 engine in silicon, the kernel found it, and verity is using it. I'd enabled that driver months ago for something else entirely.

What I still don't have#

Those 32 bytes travel to the kernel on the command line, and the command line lives in the bootloader's environment, which is unsigned and writable. So someone who can rewrite my root filesystem can rewrite the number that judges it, in the same sitting, with the same screwdriver.

What I've actually bought is that the lazy attack — swap the filesystem, change nothing else — now fails. That's worth having. It is not the same as tamper-proof, and the boot log looks identical either way, which is exactly why I'm ending the post on it rather than the nice green lines above.

Closing it properly means moving the kernel image out of the root filesystem so the hash can live somewhere signed. That's the next chunk of work, and it comes with a decision I haven't made yet.

But first: the thing I haven't done is break it. Everything above proves verity is running. Next post, I corrupt a block and find out whether it actually refuses — which is the only claim that's worth anything.