Breaking dm-verity on purpose: one bit in 52 MB, and the board refuses

Last post ended with dm-verity running and me refusing to call it done, because everything I'd proved was that it was switched on. Switched on and actually refusing are different claims, and only one of them is worth anything.

So: one bit.

device-mapper: verity: 179:4: data block 0 is corrupted
SQUASHFS error: Failed to read block 0x0: -5
unable to read squashfs_super_block
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(253,0)

Fifty-two megabytes of root filesystem, one byte changed from 0x70 to 0x71, and the board will not boot. The kernel re-hashed that block as it read it, compared it against the tree, and stopped.

The word that matters#

data block 0 is corrupted.

Not metadata block. Those two messages are one word apart and they mean opposite things. metadata would mean the hash tree is at the wrong offset — a build mistake, which is precisely the bug I spent yesterday afternoon on. data means the tree was fine and the filesystem didn't match it.

If you ever demo this, say which one you saw. "It refused to boot" is not evidence; the failure has to be the one you were aiming at. I've now had four different failures out of this subsystem and three of them were my own build being wrong.

How to sabotage your own firmware properly#

This turned out to be the interesting part, because two obvious approaches both produce a test that passes and proves nothing.

You cannot corrupt the filesystem before building. The build hashes the filesystem and writes the resulting root hash into the boot configuration. Corrupt it first and the build faithfully hashes the corrupted version, the numbers agree, and the board boots your sabotage without complaint. I'd already been bitten by exactly this when I was testing the kernel signing — the build kept quietly repairing my vandalism — so at least I recognised the shape of it this time.

And you cannot edit the finished firmware file either. It's an archive, and every item inside carries a cryptographic checksum. Edit one and the flashing tool rejects the whole thing — at which point you're testing the flashing tool, not the boot chain.

So the sequence has to be: build normally, pull the finished filesystem image out, flip the bit, repack — and crucially, leave the recorded root hash alone. Both images end up carrying the identical hash. The hash is the constant. The data is the variable. That's what makes it an experiment rather than an anecdote.

I checked it the paranoid way before flashing: wrote both images to simulated SD cards and compared the partitions byte by byte. One byte different, at the offset I intended, with the same hash in both boot configurations. A test that can't fail isn't a test, so the build script now refuses to produce the pack if the good image doesn't verify or if the tampered one still does.

The sequence that does work, as commands:

# 1. build normally, so the recorded root hash is the hash of a GOOD filesystem
# 2. pull the finished image out of the pack
# 3. flip exactly one byte — 0x70 -> 0x71, at offset 0 of the data area
printf '\x71' | dd of=rootfs.squashfs bs=1 seek=$DATA_OFFSET count=1 conv=notrunc
# 4. repack, and LEAVE THE RECORDED ROOT HASH ALONE

The hash is the constant, the data is the variable. That is what makes it an experiment rather than an anecdote — both images carry the identical root hash, and only one of them matches it.

Where to put the bit#

Block zero.

The filesystem's own header lives there, so it's read before anything else on every single boot. The failure is immediate and identical every time. Put your flipped bit a few megabytes in and you get a much less satisfying demo: the board boots fine and then falls over at some unpredictable moment when something happens to read that file.

It cleaned up after itself#

Both images went over the network — no SD card, no card reader, board on a different desk. A boot that never reaches a shell never marks itself as good, so on the next reset the bootloader shrugged and went back to the previous copy.

That's the third time this project has recovered from deliberate sabotage without me touching it. I keep expecting it to feel routine and it hasn't yet.

Four links, and two of them are honest about themselves#

Every link I can enforce, I've now enforced and then broken on purpose.

The two gaps are both deliberate, and I'd rather restate them every time than have someone discover them in a demo.

Link 1 will never be enforced on this board. Anchoring it means burning one-way fuses, and burning them would marry this board permanently to throwaway development keys. It's a development board; that trade is absurd. Production is a different exercise on different hardware.

And Link 4 is not tamper-proof yet. The root hash reaches the kernel on the command line, and the command line lives in the bootloader's environment — unsigned, writable, sitting on the same card as everything else. Someone who can rewrite my root filesystem can rewrite the number that judges it, with the same screwdriver, in the same sitting.

What I've genuinely bought is that the lazy attack is dead. Swap the filesystem, change nothing else, and the board stops. That's real. It is not the same as tamper-proof, and — this is the uncomfortable bit — the boot log looks exactly the same either way. Nothing on the console tells you which of those two systems you're running.

Closing it means moving the kernel image out of the root filesystem, so the hash can live somewhere that's already signed. That's the next chunk of work, and it comes with a genuine decision I haven't made: force the command line into the kernel binary itself, or build a small signed startup image that sets verity up before handing over. One is simpler, one is more flexible, and I want to think about which I'd rather maintain in three years.

Either way, the interesting part of this series is nearly over, and the boring part — doing it again with keys that matter, on hardware that isn't mine — is what's left.