Ever since undergraduate days, I’ve always been interested in the NES platform. I’d somehow built a mostly functional emulator (which is somewhat surprising in hindsight), and have had some toy projects around this in the past (like trying to automatically turn the renderings into 3D, https://github.com/nbirkbeck/ennes)
But, I’d never actually built a ROM or built anything that used the platform. I was looking into hobby things to do over this last Christmas break, and thought about building up a game. I know that there are good tools (like NESMaker) for building up a game, but I mostly tinker around with NES because I like the low-level stuff — so using an engine-like tool didn’t seem like something I wanted to take up right now. Also didn’t have any cool ideas of what to build.
Another project that I’ve thought about in the past, was whether I could get the NES to synthesize some audio from an actual song. Again, I’m sure there are tools to do this, but I was just interested in seeing what I could build on a one or two day deadline.
NES audio is composed of 4 signal components: 2 square waves, 1 triangle wave, and a noise channel. I remember when writing a NES emulator (back in University) that I was surprised at how the Super Mario Brothers song had used the noise channel as percussion.
So, given an input, we can do some Fourier analysis on the input (in block sizes of 1024, 2048, 4096 samples), and try to decompose the input audio signal into the best parameters from the NES signal components. Basically, I did this greedily, finding the best matching parameters (which are frequency, duty-cycle type, and volume) for each of the 4 waveforms. I guess harmonics should probably be handled better–and transients (like percussion) should likely be handled more effectively. Anway, it was fun to play around with this, and for the most part if you know how the audio is parameterized, you can easily simulate how things would sound.
But it is much more fulfilling to actually build a ROM. I noticed that there was a c-compiler (cc65) that already had support for NES, and that there was a runtime library with some crude drawing abilities. So the octave code, spits out a super inefficient c-code representation of the commands to drive the audio on the NES, and there is a small driver program to visualize some of the parameters of the audio channels.
It was fun debugging some issues in my emulator to get it to work with the ROM. Turns out the runtime library for cc65 was doing some self-modification of the code (for some tables or something) and my memory mappers weren’t handling that correctly (and ended up segfaulting). Anyway, the resulting interface looks like (the overlays are from my emulator):
The analysis code is here: https://github.com/nbirkbeck/nes-freq-audio
And there are some video examples, where hopefully you can make out the source song:
There are some cracks and pops, since the analysis code doesn’t try to maintain consistency of the waveform when switching parameters. I also tried to decompose the sound so that multiple NES machines could compose and make a more realistic rendering, but ultimately (I think due to synchronization) the result wasn’t worth posting.
