Building a Soft-Patchable Synth on Teensy 4.1 with Dynamic Audio Objects
A Synth You Can Rewire While It’s Running
The Teensy Audio Library lets you build a fixed signal graph at compile time. That’s fine for most projects. But if you want to add and remove oscillators, effects, and connections while the synth is actually playing — the way you’d repatch a modular — you need something the standard library can’t do: creating and destroying audio objects at runtime without crashing the audio engine.
That’s what Dynamic Audio Objects provide. They’ve been in development for several years in the Teensy community, and one maker’s recent proof-of-concept pulls them together into a genuinely usable soft-patchable synthesizer running on a Teensy 4.1.
The Hardware Stack
The build centres on a Teensy 4.1, running at 600 MHz with a hardware floating-point unit. It can handle fifty or more simultaneous audio objects comfortably. The 8 MB PSRAM soldered to the underside is important: it provides room for audio delay lines, large waveform tables, and the memory overhead that comes with dynamically allocated objects.
Audio I/O runs through the standard Teensy audio adaptor, which carries an SGTL5000 codec at 16-bit 44.1 kHz stereo. A 3.5-inch ILI9341 TFT with an XPT2046 resistive touch layer handles the UI. Both the display and the touch controller share an SPI bus — which matters for a specific bug covered below.
For hands-on control there are two M5Stack I²C units: the 8Angle (eight potentiometers, each paired with an RGB LED, plus a toggle switch) and the 8Encoder (eight rotary encoders with momentary switches and RGB LEDs, plus another toggle). Both communicate over Grove connectors using their STM32F030 co-processors. Together they put sixteen continuously readable controls in reach without consuming a single analog pin on the Teensy itself.
Six Screens, One Patch
The leftmost encoder cycles through six screens:
- Object placement — add new audio nodes to the canvas
- Patchcords — connect nodes together
- Edit — adjust an object’s internal settings and screen position
- MIDI settings — assign CCs to parameters with selectable mapping curves
- Delete — remove objects or connections
- Filing — save and load patches from the SD card
A handful of objects are hard-coded at compile time: the I²S input, the SGTL5000 control block, and the I²S output. They define the hardware boundaries of the patch. Movable, but not deletable. Everything placed between them is fully dynamic.
How Polyphony Works
Objects can be tagged as per-note. The first note you play uses the objects already sitting in memory. When a second note arrives, the system instantiates a fresh copy of every per-note object along with its associated patchcords.
Cloning handles the edge cases. A patchcord from a single-instance object, like a global LFO, gets duplicated so every voice shares that source. A patchcord leaving the per-note group must land on an unused mixer input. No free input means voice creation fails and you’ve hit the polyphony ceiling.
The default mixer is mixS, a new stereo mixer that can be constructed with more than eight inputs, so eight voices isn’t a hard cap. After note-off, the voice idles until the engine marks it inactive, then its objects are destroyed and memory is returned. Voices in use during patch editing persist until you load a different patch.
This is more dynamic than the typical Teensy polyphony approach, which pre-allocates a fixed voice pool at compile time. The tradeoff is a small allocation cost at note-on. At low voice counts on a 600 MHz Cortex-M7 that’s unlikely to be audible, but it’s worth profiling under heavy polyphonic load before shipping a finished instrument.
The Display Ticking Problem
There’s an audible tick whenever the screen updates. The project notes this as probably a hardware issue. It almost certainly isn’t. The Teensy audio engine fires a DMA interrupt every 128 samples — at 44.1 kHz that’s roughly one block every 2.9 ms. The ILI9341 pushes pixels over SPI, also via DMA. When a display write lands at the same moment as an audio DMA transfer, the audio block gets delayed and you hear a click.
Two standard fixes exist. First: wrap TFT pixel-push calls in AudioNoInterrupts() and AudioInterrupts() guards, or schedule all display updates to land between audio blocks rather than firing freely. Second: switch to the ILI9341_t3n fork, which adds async DMA support for the display and shrinks the contention window considerably. Neither requires touching any hardware.
MIDI CC Mapping
The plan is to make every useful parameter addressable by any CC, with mapping curves chosen per-parameter: exponential for envelope times, linear for levels, and so on. All incoming MIDI messages broadcast to every object, so a single CC can modulate multiple targets simultaneously. Input arrives via USB to a host PC or through the Teensy 4.1’s built-in USB host port, which accepts a class-compliant MIDI keyboard directly without a computer in the chain.
What’s Still Left
Full coverage of all relevant audio library objects, wavetable synthesis using samples from SD, a usable sample-playback UI, better use of the M5Stack LEDs as visual feedback, MIDI patch selection, SysEx, and documentation. Multi-timbrality is listed as a stretch goal. Structurally it’s the same problem as polyphony — just routing different MIDI channels to different per-note object groups instead of different notes to the same group.
The hardest part is already working: dynamic object lifecycle reliable enough to build and tear down voices without crashing the audio engine. Everything after that is a long but tractable implementation checklist.