Backing up the eMMC: boot partitions, rootfs, and the force_ro trap

This post exists because everything after it is destructive.

The next parts build and flash bootloaders. At some point one of them will not boot — mine didn't, in part 7 — and the difference between "an interesting afternoon" and "an expensive brick" is entirely whether you can put the factory image back.

The important part is not "run dd". It is that a naive dd of the eMMC misses the bootloader and the U-Boot environment entirely, and that one piece of the boot configuration is not block data at all, so nothing you dd will ever capture it.

First, work out what is actually on the eMMC#

Do not copy my device names. Confirm yours.

lsblk

NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
mmcblk2      179:0    0 29.2G  0 disk
|-mmcblk2p1  179:1    0  118M  0 part
`-mmcblk2p2  179:2    0 29.1G  0 part /
mmcblk2boot0 179:32   0    4M  1 disk
mmcblk2boot1 179:64   0    4M  1 disk

Cross-check against the kernel command line, so you are not guessing which partition is the root filesystem:

cat /proc/cmdline

console=ttymxc1,115200 root=/dev/mmcblk2p2 rootwait ro fsck.mode=force ...

root=/dev/mmcblk2p2 confirms p2. ro says it is mounted read-only at boot. That corroborates lsblk from a completely independent source, which is the standard worth holding yourself to before you write anything.

Identify the eMMC properly, because on another board it may be mmcblk0: it is the disk with RM=0 (non-removable), the one backing / (findmnt /), and the one with companion bootN devices.

The clue that changes the whole backup#

Here is the file that tells you the naive backup is wrong:

cat /etc/fw_env.config

# Block devices for CompuLab imx8m-plus
# emmc
/dev/mmcblk2boot0	0x3f0000 0x4000
#/dev/mmcblk2boot1	0x3f0000 0x4000   # (commented = the redundant copy)

The format is <device> <offset> <size>. So the U-Boot environment is 16 KB (0x4000) at offset 0x3f0000, inside mmcblk2boot0 — an eMMC hardware boot partition. Not in a filesystem. Not in the user area.

Two consequences, and the first one is the whole point of this post:

• A dd of /dev/mmcblk2 would miss it. The user area and the boot partitions are separate block devices. Imaging one does not touch the other.
• imx-boot is almost certainly in there too. 0x3f0000 is 4 MB − 64 KB, i.e. the environment sits at the end of the 4 MB boot partition. The natural layout is [ imx-boot from the start … ][ env at 0x3f0000 ]. You do not need to prove the exact imx-boot offset, because imaging the whole partition captures it wherever it sits.

So the backup set is four things, not one#

Items 1–3 are what you restore from. Items 4–6 cost seconds and turn a future mystery into a lookup.

Why block images and not tar. A tar of / copies files. It loses the partition table, the bootloader, the environment and the exact filesystem layout — and the board will not boot from it. dd copies raw blocks, so a restore is bit-for-bit the factory state.

The procedure#

The awkwardness here is that the stock compulab user is not in the disk group, and sudo wants a password. So ssh board "sudo dd …" hangs: there is no terminal to type the password into. Rather than fight that, grant read access once on the serial console, then pull over SSH with no sudo at all.

Step 1 — one sudo, on the DBG serial console#

sudo chmod o+r /dev/mmcblk2 /dev/mmcblk2boot0 /dev/mmcblk2boot1 && sync

Why this is safe, and why it does not contaminate the backup: device nodes under /dev live in devtmpfs, a RAM filesystem. Changing their permissions never writes a byte to the eMMC, so the image you capture is still factory-pristine. The change vanishes on reboot, or undo it with chmod o-r. The sync flushes pending writes so the image is as consistent as it can be on a live system.

Step 2 — pull the images, from your laptop#

mkdir -p ~/cl-backup && cd ~/cl-backup
BOARD=192.168.44.170

# boot partitions first — small, and they hold imx-boot + env
ssh compulab@$BOARD "dd if=/dev/mmcblk2boot0 bs=1M | gzip -1" > mmcblk2boot0.img.gz
ssh compulab@$BOARD "dd if=/dev/mmcblk2boot1 bs=1M | gzip -1" > mmcblk2boot1.img.gz

# then the whole user area
ssh compulab@$BOARD "dd if=/dev/mmcblk2 bs=4M status=progress | gzip -1" > stock-emmc-full.img.gz

Reading that pipeline, because every part of it is doing something deliberate:

• bs=4M — read in 4 MB chunks. Larger blocks mean meaningfully faster throughput on eMMC than the 512-byte default.
• status=progress — prints a live byte counter to stderr. That matters: SSH relays stderr to your terminal while stdout carries the image, so the progress text can never contaminate the data stream.
• | gzip -1 — compress on the board, before the bytes cross the network. -1 is the fastest setting; the empty regions of a 29 GB disk collapse to almost nothing anyway. Use zstd -T0 instead if the board has it.
• > file — the compressed bytes travel over SSH's stdin as a clean binary pipe.

Never add ssh -t to these commands. A pseudo-terminal translates newline bytes (\n → \r\n) and will silently corrupt the image. You will not find out until a restore fails, months later.

Step 3 — the text insurance#

# on the board
{ echo "### fw_env.config"; cat /etc/fw_env.config;
  echo "### fw_printenv";   sudo fw_printenv;
  echo "### sfdisk dump";   sudo sfdisk -d /dev/mmcblk2;
  echo "### ext_csd";       sudo mmc extcsd read /dev/mmcblk2 2>/dev/null \
                              | grep -iE 'PARTITION_CONFIG|BOOT_BUS|BOOT_PARTITION';
} > /tmp/uboot-meta.txt

# on your laptop
scp compulab@$BOARD:/tmp/uboot-meta.txt ~/cl-backup/

mmc-utils may not be installed on the stock image — that line simply no-ops if so, and the rest still works. Install it if you can; see the restore section for why that readout matters.

Verify it, or you do not have a backup#

cd ~/cl-backup
for f in *.img.gz; do gzip -t "$f" && echo "$f: archive OK"; done
ls -lh *.img.gz
shasum -a 256 *.img.gz | tee SHA256SUMS

gzip -t decompresses the entire stream in memory and checks the CRC without writing anything. If it passes, the file is neither truncated nor corrupted.

Write the checksums down somewhere outside the backup directory. Months later, when you are about to restore, you want to compare the file against a number you recorded at the time — not against another copy of the same file, which proves nothing. I nearly flashed the wrong image precisely because I trusted a filename.

Then get a second copy off the machine:

scp -r ~/cl-backup user@somewhere-else:~/cl-backup-copy

Restoring: two things that will catch you#

Restore is the discovery in reverse — every blob goes back to the device it came from. Because whole devices were imaged, you never need to know internal offsets.

EMMC=mmcblk2                      # re-confirm with lsblk on the system you restore FROM
gzip -t stock-emmc-full.img.gz    # verify before writing

# 1) user area
zcat stock-emmc-full.img.gz | sudo dd of=/dev/$EMMC bs=4M status=progress conv=fsync

# 2) boot partitions — write-protected by default
echo 0 | sudo tee /sys/block/${EMMC}boot0/force_ro
zcat mmcblk2boot0.img.gz | sudo dd of=/dev/${EMMC}boot0 bs=1M conv=fsync
echo 0 | sudo tee /sys/block/${EMMC}boot1/force_ro
zcat mmcblk2boot1.img.gz | sudo dd of=/dev/${EMMC}boot1 bs=1M conv=fsync
sync

force_ro. eMMC boot partitions come up write-protected in Linux — RO=1 in that lsblk output at the top of this post, backed by /sys/block/mmcblk2boot0/force_ro. Reading them for a backup needs nothing special, which is why step 1 only needed chmod o+r. Writing them needs the flag cleared first, and if you forget, dd fails in a way that looks like a permissions problem rather than a hardware write-protect.

conv=fsync forces the writes to physically complete before dd exits, so you are not fooled by buffering into pulling power on a half-written boot partition.

You cannot cleanly overwrite the disk you are running from. Restoring /dev/mmcblk2 while Linux is running off mmcblk2p2 is overwriting the floor you are standing on. Restore from a system that is not booted off the eMMC — external boot media, a maintenance shell, or the SoC's USB recovery mode if the board will not start at all.

The one thing dd cannot capture#

Which boot partition the eMMC presents to the boot ROM is held in the eMMC's ext_csd registers — the PARTITION_CONFIG byte — not in any sector. It is chip register state, not block data.

So if something later changes that register, restoring the data will not fix it. You re-apply it by hand:

sudo mmc bootpart enable <n> <send_ack> /dev/mmcblk2

with the values from the uboot-meta.txt you saved in step 3. In practice the factory setting usually survives everything, but this is the one knob a pure block backup cannot carry, and it is worth knowing it exists before you need it at midnight.

Next: pinning the U-Boot build environment, so that what you build is reproducible rather than whatever happened to be checked out that day.