samienr - Console moddingElectrical Nerd + Digital ArtistZola2025-08-06T00:00:00+00:00https://samienr.com/tags/console-modding/atom.xmlHow The Nintendo 3DS Was Hacked Using a Music File2025-08-06T00:00:00+00:002025-08-06T00:00:00+00:00
Samien Rahman
https://samienr.com/blog/soundhax/<blockquote class="note">
<p class="alert-title">
<i class="icon"></i>Note</p>
<p>SoundHax was patched as of firmware version 11.4 and does not work on any Nintendo 3DS system running firmware 11.4 or higher.</p>
</blockquote>
<p><a href="https://soundhax.com">SoundHax</a> was one of the most accessible exploits to get unsigned code running on the Nintendo 3DS. It worked by abusing a bug found in the <em>Nintendo 3DS Sound</em> application with a specially-crafted <code>.m4a</code> file.</p>
<p>The exploit was special as it was the first exploit on the 3DS that was free, offline, and worked on every version of the firmware as of when it was discovered.</p>
<blockquote class="important">
<p class="alert-title">
<i class="icon"></i>Important</p>
<p>To fully appreciate this brilliant symphony of hacks, a basic understanding of memory management is recommended. <a href="https://samwho.dev/memory-allocation/">This article does a good job explaining the basics</a>.</p>
</blockquote>
<p><small>Before proceeding, I'd also highly suggest reading the writeup on <a href="https://github.com/nedwill/soundhax?tab=readme-ov-file#writeup">Nedwill's GitHub page</a></small></p>
<hr />
<h1 id="the-vulnerability">The Vulnerability</h1>
<p>For context, the M4A format uses the MP4 file format. At its core, SoundHax exploits a bug in how the <em>Nintendo 3DS Sound</em> application handles MP4 atom tags.</p>
<p>That is, when loading in an M4A file, a 256-byte buffer is allocated on the heap for song names (the max size according to the MP4 spec).
The song name is then copied from the file into the buffer using one of two functions:</p>
<ul>
<li>For ASCII strings, <code>strncpy(dst, src, 256)</code> is used, which is safe.</li>
<li>For Unicode strings, <code>memcpy()</code> is used, which <strong>takes a user-provided size</strong> instead of 256 <small>(not safe!)</small>.</li>
</ul>
<p>Unicode strings likely didn't use <code>strncpy()</code> because it stops at the first null terminating byte <small>(which is used for marking the end of an ASCII string)</small>. Unicode can have null bytes as parts of characters, which means that <code>strncpy()</code> wouldn't work well.</p>
<p>Instead, song names are simply gathered by copying over as many bytes as the file specifies, <strong>even if the number is greater than 256.</strong></p>
<p>This means that the program code is vulnerable to a <colorize><strong>buffer overflow attack.</strong></colorize>
By providing a carefully crafted Unicode title longer than 256 bytes, an attacker can write data <strong>beyond the allocated buffer and into other parts of console memory.</strong></p>
<h1 id="the-exploit">The Exploit</h1>
<p>Taking advantage of this bug, we can get arbitrary code execution on the 3DS using the following steps:</p>
<ol>
<li>Creating an M4A file with an oversized Unicode title</li>
<li>Causing a Buffer Overflow in the Heap</li>
<li>Corrupting the Heap Metadata</li>
<li>Unlink Exploitation</li>
<li>Heap Feng Shui</li>
<li>Constructing and Executing a ROP Chain</li>
<li>Abusing the GPU ('gspwn')</li>
<li>Unlimited Power!!!</li>
</ol>
<p>Anyways, here's how it works.</p>
<h2 id="1-crafting-the-malicious-file">1. Crafting the malicious File</h2>
<p>The exploit begins with the construction of a M4A file containing a specially crafted Unicode title that's intentionally oversized. Nedwill used a Python script to do this:</p>
<pre data-lang="python" style="background-color:#212733;color:#ccc9c2;" class="language-python "><code class="language-python" data-lang="python"><span style="font-style:italic;color:#5c6773;"># Start with some text for the song title: "3 nedwill 2016"
</span><span>title </span><span style="color:#f29e74;">= </span><span style="color:#bae67e;">"<</span><span style="color:#95e6cb;">\x00</span><span style="color:#bae67e;">3</span><span style="color:#95e6cb;">\x00 \x00</span><span style="color:#bae67e;">n</span><span style="color:#95e6cb;">\x00</span><span style="color:#bae67e;">e</span><span style="color:#95e6cb;">\x00</span><span style="color:#bae67e;">d</span><span style="color:#95e6cb;">\x00</span><span style="color:#bae67e;">w</span><span style="color:#95e6cb;">\x00</span><span style="color:#bae67e;">i</span><span style="color:#95e6cb;">\x00</span><span style="color:#bae67e;">l</span><span style="color:#95e6cb;">\x00</span><span style="color:#bae67e;">l</span><span style="color:#95e6cb;">\x00</span><span style="color:#bae67e;">"
</span><span style="font-style:italic;color:#5c6773;"># Add a bunch of zeros to make the title 772 bytes long
</span><span>title </span><span style="color:#f29e74;">+= </span><span style="color:#bae67e;">" </span><span style="color:#95e6cb;">\x00</span><span style="color:#bae67e;">"</span><span style="color:#f29e74;">*</span><span>((</span><span style="color:#ffcc66;">772</span><span style="color:#f29e74;">-</span><span style="color:#f28779;">len</span><span>(title)) </span><span style="color:#f29e74;">/ </span><span style="color:#ffcc66;">2</span><span>) </span><span style="font-style:italic;color:#5c6773;"># Much larger than 256 bytes!
</span></code></pre>
<p>This creates a 772 byte string for the title. But this alone isn't enough.</p>
<p>The MP4 file format needs each piece of data to be labeled with its exact size which means we also need the file metadata to tell the 3DS how long this title is. When the rest of this Python script builds the final M4A file, it calculates the length of our oversized title and embeds it as binary data into the file header, like any other MP4 file is structured.</p>
<p>The file isn't lying about anything, it genuinely contains 772 bytes of title data, properly labeled and all. It's just that 772 bytes is <em>way too many</em> bytes for a title, and it goes well over the allocated space of 256 bytes. This means that the rest of the 516 bytes end up writing over other regions of memory.</p>
<h2 id="2-triggering-the-heap-overflow">2. Triggering the Heap Overflow</h2>
<p>When the <em>Nintendo 3DS Sound</em> application encounters our special file, it will read the size field in the file metadata, sees "772 bytes of title data," and trustingly pass this value to the <code>memcpy()</code> function without questioning whether 772 bytes is a reasonable length for a song title.</p>
<p>When this happens, the 3DS:</p>
<ol>
<li>allocates the standard 256-byte buffer.</li>
<li>uses <code>memcpy()</code> with our provided length.</li>
<li>copies our oversized title, writing beyond the buffer's boundaries</li>
</ol>
<h2 id="3-heap-manipulation-and-unlink-exploitation">3. Heap Manipulation and Unlink Exploitation</h2>
<h3 id="heap-manipulation">Heap Manipulation</h3>
<p>The heap overflow allows us to manipulate the heap's metadata structures. Modern heap implementations maintain information about memory chunks, including:</p>
<ul>
<li>Chunk sizes</li>
<li>Usage status</li>
<li>Forward/backward pointers</li>
<li>Boundary tags</li>
</ul>
<p>By overflowing into these structures, we can override some of this metadata and fool the heap manager into operating using this fake data. If you're a little confused, <a href="https://samienr.com/blog/soundhax/memory#heap-overflow">take a look at this</a></p>
<h3 id="unlink-exploitation">Unlink Exploitation</h3>
<p>When the heap manager needs to free a chunk, it performs an "unlink" operation to remove the chunk from its internal data structures and possibly also merge it with adjacent free chunks.</p>
<p>Part of this operation involves updating the forward and backward pointers of a chunk. This process assumes that the chunk metadata is trustworthy. Our fake chunk exploits this trust to achieve arbitrary memory writes.</p>
<p>This typically looks something like this:</p>
<pre data-lang="C" style="background-color:#212733;color:#ccc9c2;" class="language-C "><code class="language-C" data-lang="C"><span>chunk</span><span style="color:#f29e74;">-></span><span>fd</span><span style="color:#f29e74;">-></span><span>bk </span><span style="color:#f29e74;">=</span><span> chunk</span><span style="color:#f29e74;">-></span><span>bk</span><span style="color:#ccc9c2cc;">;
</span><span>chunk</span><span style="color:#f29e74;">-></span><span>bk</span><span style="color:#f29e74;">-></span><span>fd </span><span style="color:#f29e74;">=</span><span> chunk</span><span style="color:#f29e74;">-></span><span>fd</span><span style="color:#ccc9c2cc;">;
</span></code></pre>
<p>The allocator pretty much goes "if this chunk says its next chunk is at address X, then I'll update the chunk at address X to point back to the previous chunk." Except we are in control of those addresses. That is to say, we choose what address X is, and we control what the previous chunk's address is, so we can make the allocator write an arbitrary value to an arbitrary location, A.K.A. an arbitrary write primitive.</p>
<h2 id="4-stack-manipulation-heap-feng-shui">4. Stack Manipulation | "Heap Feng Shui"</h2>
<p>With control over the unlink operation, we can write to any memory location. However, this power must be wielded strategically. The SoundHax exploit uses a technique known as <em>heap feng shui</em> to gain control over the program's execution stack.</p>
<p>The goal is to trick the heap allocator into returning a stack address when the program next requests a memory allocation. As for what this stack address is, Nedwill had to painstakingly find a particular address that's contents look like a valid heap chunk with seemingly proper heap chunk metadata values</p>
<p>We use our arbitrary write primitive to overwrite the heap's free list header with this stack address (via the unlink exploitation). At the same time, we manipulate the size field of the chunk being freed to make it appear very small, preventing it from being a viable option the next time the allocator looks for a free chunk. This sets up our fake stack "chunk" as the primary candidate for the next allocation.</p>
<p>When the program makes its next <code>malloc()</code> call, the heap allocator searches through the free list looking for a suitable block. It finds our fake chunk on the stack, validates that it has enough space and proper metadata, and then returns the stack address as if it were heap memory.</p>
<h2 id="5-rop-chain-construction-and-execution">5. ROP Chain Construction and Execution</h2>
<p>With step 4, the application writes data to what it thinks is allocated heap memory, but it's actually writing to the program's execution stack. Since the stack contains values such return addresses and the local variables of functions, we now have access to control the program's execution flow.</p>
<p>Unfortunately for us, however, we can't go ahead and just write in instructions (A.K.A. shellcode) just yet. Modern systems, such as the one on the 3DS, have some protections to prevent executing code on the stack or heap. However, we can work around this by using a technique known as Return-Oriented Programming (ROP), which chains together small pieces of existing code to perform our desired operations.</p>
<p>Think of ROP as something like the cut-up technique, where an artist cuts scraps out of a book or magazine, then rearranges these scraps to create a new text the author never intended.
<img class="
"
alt="An example of the cut-up technique"
src="Cut_up_text.webp" />
<figcaption>Imagine each one of these scraps is a piece of code.</figcaption>
</p>
<p>Each piece of code, or "gadget," in our ROP chain is a small fragment of actual program code that performs a simple operation, like loading a register or calling a function, then returns control to the next gadget in the sequence.</p>
<p>The SoundHax ROP chain performs a handful of operations. First, it copies our second-stage payload from the stack to heap memory. Next, it sets up parameters for some GPU operations. Then it actually calls these GPU functions to perform memory copying operations that would normally be forbidden (more on this in the next section). Finally, it transfers control to our newly-positioned code.</p>
<p>Each step of this process uses code that already exists in the application's memory space, making it extremely difficult for most prevention mechanisms to detect our attack.</p>
<h2 id="6-gpu-assisted-code-injection-gspwn">6. GPU-Assisted Code Injection (gspwn)</h2>
<p>I briefly mentioned GPU functions. Now let's actually talk about why we're using graphics calls and how genius this move is.</p>
<p>When gaining code execution through the ROP chain, we were still constrained by the system's memory protection mechanisms, meaning that we weren't yet able to overwrite any of the application's code segment from userland. This usually isn't allowed as that would violate a bunch of security boundaries-- user applications <em>should</em> be kept separate from system code.</p>
<p>Nedwill's solution to get around this was pretty creative: Abusing the 3DS's GPU. The GPU has special privileges to perform direct memory access (DMA) transfers between memory regions. This means that while almost nothing should be able to copy data from heap memory into the executable text segment of running applications, the GPU <em>can.</em></p>
<p>This technique, known as 'gspwn' in the 3DS homebrew community, effectively bypasses the system's code signing and memory protection by using the GPU as an unwitting accomplice in our attack. The GPU doesn't know or care that we're abusing its power, it simply performs whatever operation is requested.</p>
<h2 id="7-full-code-execution">7. Full Code Execution</h2>
<p>With our payload now residing in executable memory, the final step is to transfer control to it. Our ROP chain concludes by jumping to the newly-written code, which implements a second-stage loader that performs the final steps of the compromise.</p>
<p>The stage-2 payload disables potentially problematic system threads that might interfere with homebrew execution. It then opens and reads the <code>otherapp.bin</code> file from the SD card, which contains the <colorize>homebrew launcher.</colorize> For the uninitiated, the homebrew launcher is the gateway to running homebrew applications <small>(a.k.a. unsigned code that can be written by anyone without Nintendo's approval)</small>. The ROP chain uses the GPU one more time to copy this payload into memory and transfer control to it.</p>
<p>At this point, the system is fully compromised and running arbitrary unsigned code with the same privileges as the original Sound application. The user will now see the homebrew launcher appear on screen.</p>
<h1 id="conclusion">Conclusion</h1>
<p>Unlike many other exploits for various game consoles, SoundHax required no internet connection, specific games, hardware modifications, or any complex setup procedures. Users could simply download the appropriate M4A file for their region and firmware version, copy it to their SD card, and play it through that built-in <em>Nintendo 3DS Sound</em> application.</p>