Moving the kernel out of the rootfs, and feeding the bootloader garbage

The last post ended with dm-verity working and me refusing to call it finished, because the number it checks against was sitting somewhere anyone could edit. This post is about closing that, and the two-command test that proves it closed.

The hole, one more time#

dm-verity reduces a 52 MB filesystem to 32 bytes. Change any block and the board refuses to read it. Lovely — as long as those 32 bytes are out of reach.

They weren't. They rode in on the kernel's command line, and the command line came from the bootloader's environment: a chunk of the same SD card, unsigned, writable with a text editor. So the attack was never "defeat the hashing". It was "change the filesystem, then change the number, both on the same card, in the same sitting".

Worse, and this is the part that nagged at me: the boot log looks identical either way. There is no line that says "by the way, the number I'm checking against came from somewhere anyone can edit."

Two ways to fix it, and I picked the wrong one first#

The hash has to arrive inside something already signed. I had two candidates.

Compile it into the kernel. The kernel is signed, so if the command line is part of the kernel binary and the kernel is told to ignore whatever the bootloader hands it, the number is as protected as the kernel itself.

Or build a tiny signed startup image that sets verity up before handing over to the real system.

Both are equally secure. The only difference is cost, and I reasoned my way to the second one: the first needs a separate kernel per firmware slot, because the two slots live on different partitions, and I assumed two kernel builds meant minutes added to every single build.

Then I measured it. Forty-six seconds, and only twelve source files recompiled, because the build system tracks which files actually reference that setting and rebuilds only those.

Ninety seconds a build does not justify inventing a whole new boot stage. So: compile it into the kernel. The security argument also fits in one sentence — the command line is inside the signed kernel — and anyone auditing this later will want it in one sentence, so a short one has real value.

The lesson there isn't "I changed my mind". It's that I offered an opinion before measuring something I could have measured in five minutes.

The thing that had to move first#

There was an obstacle, and it's a nice one.

The hash describes the root filesystem. To be trustworthy it must live in a signed thing. The only signed thing I had was the kernel container — which was a file inside that same root filesystem.

So: hash the filesystem, put the hash in the container, which changes the container, which changes the filesystem, which changes the hash. Round and round. There is no arrangement that satisfies itself.

You cannot store the hash of a filesystem inside that filesystem. Once I wrote it down that plainly it stopped feeling like a puzzle and started feeling like arithmetic.

The fix is to move the kernel container out into its own partition. Two of them, one per slot. The bootloader now reads it straight off the card by block number rather than by filename.

That renumbered every partition after it, which I expected to be a miserable afternoon of chasing hardcoded numbers. It wasn't — because a day earlier, chasing an unrelated intermittent bug, I'd switched everything to referring to partitions by their unique identifier rather than their position. The filesystem moved from partition four to partition six and nothing noticed. That's the second time that one fix has paid for itself.

And a bonus: with the bootloader no longer reading anything out of the root filesystem, the copies of the kernel in the filesystem became dead weight. Deleting them took the image from 53 MB to 41. The second kernel then put it back to 53, so call it even — but without the cleanup it would have been 65.

The garbage test#

Here's the part I like.

Stop the board in the bootloader and check what command line is stored:

STM32MP> printenv bootargs
## Error: "bootargs" not defined

Nothing. Good — there's nothing security-relevant in writable storage any more.

Now do what an attacker would do, except lazier. Don't bother crafting a plausible command line; just put rubbish there:

STM32MP> setenv bootargs total-garbage-not-a-real-cmdline
STM32MP> boot

The board boots. Normally. All the way to a shell. And:

iex> cmd("cat /proc/cmdline")
console=ttySTM0,115200 dm-mod.waitfor=PARTUUID=... dm-mod.create="vroot,...sha256 d45a29f1..." root=/dev/dm-0 ...

The garbage is gone. The kernel threw away what the bootloader handed it and used its own.

There's an accident in there that makes it better than I planned. That boot command also saves the environment — so total-garbage-not-a-real-cmdline is now written to the SD card, permanently, and every subsequent boot ignores it too. The attacker's value is literally sitting on the disk being disregarded.

What it cost#

The escape hatch is gone. The kernel ignoring the bootloader's command line is the entire point, but that same hatch is what I used twice in one week: once to test some quoting before committing to a flash, once to rescue a board that wouldn't boot by telling it to skip verification just this once.

Neither works now. Every bad build costs a trip to the card reader.

For a product that's not a cost, it's the feature. On the bench it's a real loss, and pretending otherwise would be the kind of thing I started this series to avoid.

Where the chain ends#

Every link I am able to enforce on this board is enforced, and I have broken each one deliberately and watched it refuse. That's the claim, and I think it's the honest one.

The exception is permanent and deliberate. Anchoring the first link means burning one-way fuses, and burning them on this board would marry it forever to development keys that live unencrypted on a build server. It's a development board. That trade is absurd. ROTPK is not deployed on platform will appear in every log this board ever produces, and that's the correct outcome rather than an outstanding task.

So: the interesting engineering is done. What's left is the unglamorous half — doing all of it again on hardware that isn't mine, with keys that matter, kept somewhere better than a directory, under a written procedure with more than one person in the room.

That's a different kind of post, and I'm not sure it'll be as fun.