A real mistake, and rescuing myself from it

Last post ended with a successful flash and a promise to report back honestly on whether the board actually booted. Here's the honest report: it didn't. This post is about what went wrong, how I figured out why, and the genuinely long fight to get my board back afterward. Nothing here touched anything permanent — no keys, no fuses — but it's the first time this project actually broke something, so I want to write it up properly instead of quietly editing history.

What went wrong

I reset the board expecting to see today's date in the version banner. Instead, I watched the memory training fail, completely:

DDRINFO: Cfg attempt: [ 1/6 ; ( deadbeaf / 0xdeadbeef) ]
...
DDRINFO: Cfg attempt: [ 6/6 ; ( Kingston / 0xff070010) ]
DDRINFO: Ran out all [ 6 ] cfg attempts. A non supported configuration.

Six attempts at generic memory-timing profiles, none matching, then giving up. The board couldn't get far enough in its own startup to do anything at all — not destroyed, just stuck at the very first step.

Finding the actual cause

I compared this against the very first serial-console session I ever ran on this same board, months earlier. That one went straight to the right answer, first try: Micron 4096MB, no retries, no fuss. Something about my build was different from the factory build in a way that mattered for memory training specifically.

The board's own build system has a setting — a choice between a few named variants — for exactly which memory configuration is physically populated on a given unit, since the same board design ships with a couple of different memory options. I'd built with whatever the default happened to be, and it was the wrong one for this specific unit. An easy, understandable mistake, and one I already had a written note about elsewhere in this project pointing at the right setting — I just hadn't connected it to this specific build yet. Fixed in two lines and a rebuild, confirmed by a different checksum on the resulting file.

Fixing the build was the easy part. Getting a corrected image back onto an unresponsive board was not.

The rescue: a genuinely long fight with USB

Every chip in this family has a factory-built emergency mode: plug in the right cable before power-on, and instead of trying (and failing) to boot normally, the chip just waits for someone to push a working bootloader straight into its memory over USB. There's an official tool for exactly this. I already knew about it, already had it documented as "the last resort," and had frankly hoped never to actually need it for real.

I needed it. And it did not want to cooperate.

First problem: the exact command I had written down for this — with a specific named option — turned out to simply not exist in the version of the tool I had installed. It said so plainly: that option isn't a real built-in choice. I'd apparently written that command down once, without ever actually running it for real, and it sat there unverified for months. Lesson: document what you've actually run, not what you assume should work.

Second problem, much bigger: even with the corrected, simpler command, the USB transfer itself kept failing partway through, with a generic low-level input/output error. I tried a direct cable. I tried a different adapter. I tried a cable that splits into three ports (that one didn't even see the device at all). Every single time: the same failure, or a new one just as unhelpful.

I went and reread the notes from a sibling project of mine — a different, already -working device from this same product line — specifically looking for whether anyone had ever found a better way. The honest answer: no. But there was a small gift buried in there anyway — that team had hit this exact same mistake once (the identical wrong-memory-setting bug!), and solved their USB reliability problem by plugging directly into their computer's own port instead of through their monitor's hub. Not a different tool. Just a cleaner cable path.

I tried that too. Still failed.

The actual fix: switch operating systems entirely. This exact recovery mode is known to be pickier about USB signal quality than ordinary USB traffic, and apparently pickier still under my Mac's own USB drivers specifically than under Linux's. So I set up a small Linux virtual machine — a whole separate operating system running in a window on my own computer — specifically to run the recovery tool from there instead.

My first attempt at this (a lightweight, config-file-based virtual machine tool) hit a wall immediately: the virtual machine had no visibility into the physical USB device at all, and teaching it to see one specific device meant editing a configuration format I wasn't confident I'd get right on the first try. Rather than burn more time guessing, I switched to a different virtual machine app with a normal graphical interface and a literal checkbox for "share this USB device with the virtual machine." Much faster.

Small hiccups along the way, all fixable:

• Downloaded the wrong version of the operating system image at first (built for a different kind of processor than my Mac actually has — an easy mismatch to make, immediately and clearly flagged by the VM software itself).
• Had to figure out which of several connected USB devices in the sharing menu was actually my board, since none of them were labeled obviously — got there by elimination, ruling out my monitor, my console cable, and a network adapter, leaving exactly one unlabeled entry. Checking the device's ID afterward confirmed it: NXP's own official vendor code, exactly matching what I'd expect.
• Copy-pasting long commands directly into the virtual machine's own window quietly ate characters — a real, silent data-loss bug in how that display handles pasted text. Fixed by connecting to the virtual machine over the network instead and typing into a completely normal terminal window, where copy-paste just works like it always does.
• The recovery tool isn't available as a simple install inside this Linux environment, so I had to build it from its own source code — which meant fixing three missing-dependency errors, one at a time, each with a straightforward one-line fix once I knew the actual missing piece's name.

That build finished cleanly. And — a small, satisfying detail — the build process itself printed out the tool's real list of built-in recovery scripts, which let me confirm, straight from the source code, that the option I'd mistakenly relied on months ago never actually existed. Good to have that closed out for certain.

The actual fix — and it was never really about the cable at all

With the tool built, my first real attempt still failed — but with a genuinely new, different error this time, which turned out to be the clue that cracked it. My virtual machine's own USB menu showed the board's device had quietly disconnected mid-attempt. Not a noisy transfer this time — an actual disconnect.

The real cause: my Mac has its own permission prompt for letting an app talk to a connected USB accessory, and I'd been clicking the one-time "Allow" option. That apparently doesn't survive the device resetting itself partway through this exact recovery process — so my virtual machine kept losing access right in the middle, no matter what cable or adapter I used. The actual fix was clicking "Always Allow" instead. The very next attempt worked cleanly, start to finish, on the first try.

Looking back, I'm fairly convinced this was the real problem the entire time — probably even before I ever brought a virtual machine into it. Every cable and adapter I swapped earlier may never have mattered at all. A useful, slightly humbling reminder that the obvious suspect (bad cable, bad USB hub) isn't always the real one.

Where this stands right now

The known-good backup image is now loaded and genuinely running in my board's memory, over USB, exactly as intended. The very last step — actually writing it back onto the board's storage so it survives a normal power cycle — is what I'm doing next.

The one thing I want to say clearly: at no point in any of this was any data actually lost, or any key or fuse touched. This was a stuck bootloader, not a disaster — and the reason I could stay calm through several hours of USB troubleshooting is that I had a complete, verified backup of exactly the part that broke, made carefully, months before I needed it. If there's one lesson worth pulling out of this entire post, it's that one.

An update, a few days later

I said I'd report back honestly, so here it is: as of writing this, the board still isn't recovered. But there's real progress, and one genuinely useful discovery along the way.

First, the boring good news: the rescue console itself is fine. I reset the board with nothing else touched, and it reproduced the exact same failure I'd already seen — confirming the console link was never actually the problem.

Then a new discovery, unrelated to any of the earlier cable/hub troubleshooting: there's a separate macOS security setting I hadn't known about, buried in Privacy & Security, that governs whether new external accessories are allowed to connect at all. It defaults to asking permission every time — except this time it wasn't even showing me the popup to approve. Switching that setting to always-allow (temporarily; I plan to switch it back once I'm done with this) fixed that particular symptom immediately.

Past that fix, I got further than ever before. The recovery image's first stage now loads and runs correctly every time, taking the same fast, direct path the original factory boot log always showed. It's the second stage — handing off to the full bootloader — that's now failing, consistently, at the exact same byte count, every single attempt. Not randomly. The identical number, byte for byte, three times in a row.

That consistency turned out to be useful information. It let me rule out the two obvious suspects methodically instead of just guessing and retrying: it's not a flaky USB connection (the transfer itself reports complete success every time, and the low-level device logs show nothing wrong), and it's not a corrupted backup file (I re-copied and re-verified it against the checksum I made months ago — still byte for byte correct). Whatever's actually wrong is something more specific: likely some mismatch between exactly how my recovery tool talks to this particular chip, and what the chip expects back.

I have one more concrete thing to try next — the vendor's own officially documented recovery command for this exact board, instead of the generic one I've been using — and I'm seriously considering moving this whole recovery step onto an actual Linux machine instead of a virtual one, to remove a whole layer of complexity from the picture.

Nothing about any of this touched anything permanent. The board sits exactly where it did before this session — recoverable, patiently waiting, with every original backup still intact. That patience is, again, entirely thanks to the work from four posts ago.

The actual fix — bringing in a second, completely different computer

I said I was considering moving this whole recovery step onto a real Linux machine instead of a virtual one. I did it, and it's what finally cracked this whole thing open — though not for the reason I originally expected.

I picked up a small, cheap single-board computer (a Raspberry Pi, if you're picturing the size — a whole real Linux machine on something not much bigger than a deck of cards) specifically to remove the virtual-machine layer from the picture entirely. No more "is this a genuine hardware problem or something about how the virtual machine sees USB devices" ambiguity — just a real computer, with a real USB port, talking directly to the board.

Before touching anything, a near-miss worth admitting to. I went looking for my backup file and very nearly grabbed the wrong one — a similarly-named backup that actually belonged to a different device entirely, made months earlier for a different part of this whole project. Caught it purely by checking the full file path and the date, not by the filename alone, right before it would have mattered. Lesson, worth repeating for anyone doing this kind of recovery work: when you have more than one backup lying around for more than one device, the filename alone is not enough — always confirm the exact source and the fingerprint before trusting a backup file with anything.

The new machine hit its own new failure almost immediately — the recovery tool found the board fine, then lost it again partway through, every single time, with a different low-level USB error than anything seen on my Mac. I went hunting for a known cause first: there's a real, documented class of problem on Linux where a background service meant for detecting accessibility hardware (things like braille displays) quietly grabs unrecognized USB devices to check whether they're one of its own — and that grabbing can knock a device clean off the bus at exactly the wrong moment. A completely plausible, well-precedented explanation.

It wasn't the cause here. I checked, directly — that service didn't even exist on this machine's operating system. A real, clean dead end, ruled out by evidence in about a minute rather than guessed away.

What actually got me further was something almost embarrassingly simple: which physical USB port I used. This little computer has two different kinds of USB port built in — an older, slower kind and a newer, faster kind, driven by genuinely different internal hardware. On a whim, I moved the cable from one to the other. Suddenly, real progress: the transfer got roughly halfway before anything went wrong, and — genuinely exciting to see — the board's own USB identity changed mid-process, from "I am a raw, freshly-powered-on chip waiting for a recovery push" to "I am now running a real, if temporary, copy of my own bootloader." That identity change only happens once the very first recovery stage has successfully handed off to the next one — something that had never actually happened in this entire saga until this exact moment. The front-panel light on the board even changed color to match, live, while I watched — a lucky, independent confirmation I happened to notice by chance (I'd learned what that light means from the manufacturer's own reference material a few days earlier, purely because I'd finally gotten frustrated enough with unreliable cables to go looking for any other way to check the board's state).

Then came the part that cost me the most time for the silliest reason. With real progress finally happening, I needed to actually watch the board's text console again — and I couldn't get a single character out of it, on any of several attempts, despite everything about the physical connection checking out. LED said one thing (a healthy, running bootloader); the console said absolutely nothing at all, not even an echo. A genuinely confusing contradiction between two different signals that should have agreed.

The eventual answer was almost funny in hindsight: the little program I was using to view the console apparently resets some of the wire's control signals every time it opens a fresh connection — and on this particular board, at least one of those signals happens to be wired to something that can reset the device. I may have been quietly restarting the board's memory state every single time I opened a new console window to check on it, throughout this entire saga, on more than one computer. There's a documented option to tell that program not to do this. I hadn't known it existed until I went looking specifically for "can opening a serial console reset a board," out of sheer frustration.

And then, purely by accident, I got my proof it was all still alive. Meaning to close the console program with its normal exit shortcut, I fat-fingered a plain interrupt keystroke instead — and the board answered back immediately with a live command prompt. The rescue image really had been sitting there, running, the whole time; whatever had been eating my view of it clearly wasn't the board itself. A single command confirmed I was looking at exactly the right thing — the version banner matched my verified backup, word for word, down to the exact build date.

From there, it was the same well-rehearsed procedure from months ago: tell the board to expose its tiny boot storage as an external disk one more time, write the verified backup onto it, and reboot properly — cable unplugged first, this time, so the board would boot normally instead of jumping back into recovery mode.

It worked, completely. Not just to a bootloader prompt this time — the very next boot went all the way through, kernel and all, right to a normal login. The kernel's own version banner matched a note I'd written down from the very first time I ever looked at this board, months before any of this happened — about as strong a confirmation as I could ask for that this is genuinely, bit-for-bit, the same software that was there before any of this mess started.

What actually mattered, in the end: a real, different computer removed an entire category of doubt a virtual machine could never fully rule out; the specific USB port genuinely mattered, for reasons I still don't fully understand at the hardware level; and a completely unrelated console-tool quirk had likely been sabotaging my own ability to see progress for far longer than I realized. None of the three theories I originally suspected (a bad file, a flaky cable, a broken recovery tool) turned out to be the real story at all — a good reminder that the most obvious suspects aren't always the guilty ones.

And, worth saying plainly one more time, because it was true every single day of this saga: nothing about any of it ever touched a cryptographic key or a permanent fuse. The absolute worst case, at any point, was "spend more time recovering a backup I already had" — never "lose something I can't get back." That's the whole reason it was safe to spend a long, sometimes frustrating stretch of time chasing this down calmly instead of panicking.