TrebledJ's Pages

Relay

2024-01-20T00:00:00Z

The people walking in darkness
have seen a great light;
on those living in the land of deep darkness
a light has dawned.

– Isaiah 9:2

What is a relay? A race—passing a baton from one to the next. The passing of packets by a router or a gateway server. Or used plainly as a verb without technicalities, a relay may simply mean to receive and pass on information.¹

It's worth taking a moment to reflect: what are we relaying? As time passes, more responsibility falls on our shoulders—the responsibility to be helpful (or useful?) in society; the responsibility to mentor; the responsibility to work, to execute; the responsibility to lead.

What are we relaying? What torch are we passing on? What is our light? And what keeps us moving?

The average human works 90,000 hours. There's only so much experience, wealth, and knowledge we can accrue.

Light the torch and guide us through the dark.
Shadows, dispelled by balls of flame.
Relay to us this bright white shining spark.
Ready, for the race to carry on.
^{(Some words to go along with the climactic sections.)}

Similar to Remorse, Relay was featured as part of a challenge in the HKUST Firebird CTF 2024 competition, a competition where teams hunt for hidden pieces of text.

Footnotes

It might also (in my mind) be a play-on of "rhythmic delay". ↩︎

Digital Audio Synthesis for Dummies: Part 3

2023-05-24T00:00:00Z

Slap a timer, DMA, and DAC together, and BAM—non-blocking audio output!
— TrebledJ, 2022

Ah… embedded systems—the intersection of robust hardware and versatile software.

This is the third (and culminating) post in a series on digital audio synthesis; but the first (and only?) post touching embedded hardware. In the first post, we introduced basic concepts on audio processing. In the second post, we dived into audio synthesis and generation. In this post, we’ll discover how to effectively implement an audio synthesiser and player on an embedded device by marrying hardware (timers, DACs, DMA) and software (double buffering plus other optimisations).¹

To understand these concepts even better, we’ll look at examples on an STM32. These examples are inspired from a MIDI keyboard project I previously worked on. I'll be using an STM32F405RGT board in the examples. If you plan to follow along with your own board, make sure it's capable of timer-triggered DMA and DAC. An oscilloscope would also be handy for checking DAC output.

This post is much longer than I expected. My suggested approach of reading is to first gain a high-level understanding (possibly skipping the nitty gritty examples), then dig into the examples for details.

Timers ⏰

It turns out kitchens and embedded systems aren’t that different after all! Both perform input and output, and both have timers! Who knew?

Tick Tock

Timers in embedded systems are similar to those in the kitchen: they tick for a period of time and signal an event when finished. However, embedded timers are much fancier than kitchen timers, making them immensely useful in various applications. They can trigger repeatedly (via auto-reload), count up/down, and be used to generate rectangular (PWM) waves.

^{Timers have various applications, such as to count signals. (Source: EmbeddedTutor²)}

So what makes timers tick?

The MCU clock!

The MCU clock is the backbone of a controller. It controls the processing speed and pretty much everything!—timers, ADC, DAC, communication protocols, and whatnot. The signal itself is generated by an oscillator, typically a Quartz crystal oscillator which is capable of generating high, stable, self-sustaining frequencies.

The clock runs at a fixed frequency (168MHz on our board). By dividing against it, we can achieve lower frequencies.

^{By using different prescalers, we can scale down the frequency according to our needs. (Source: EmbeddedTutor²)}

The following diagram illustrates how the clock signal is divided on an STM. There are two divisors: the prescaler and auto-reload (aka counter period).

^{How a timer frequency is derived from the clock signal. (Diagram adapted from uPesy.³)}

Here, the clock signal is first divided by a prescaler of 2, then further "divided" by an auto-reload of 6. On every overflow (arrow shooting up), the timer triggers an interrupt. In this case, the timer runs at $\frac{1}{12}$ the speed of the clock.

These interrupts can trigger functionality such as DMA transfers (explored later) and ADC conversions.⁴ Heck, they can even be used to trigger other timers!

Further Reading:

How do microcontroller timers work?
Getting Started with STM32: Timers and Timer Interrupts
- There are more prescalers behind the scenes! (APB2)

Example: Initialising the Timer

Suppose we want to send a stream of audio output. We can use a timer with a frequency set to our desired sample rate.

We can derive the prescaler (PSC) and auto-reload (ARR) by finding integer factors that satisfy the following relationship.

$$ \text{freq}_\text{timer} = \frac{\text{freq}_\text{clock}}{(\text{PSC} + 1) \times (\text{ARR} + 1)} $$

where $\text{freq}_\text{timer}$ is the timer frequency (or specifically in our case, the sample rate), $\text{freq}_\text{clock}$ is the clock frequency.

On our STM32F405, we configured $\text{freq}_\text{clock}$ to the maximum possible speed: 168MHz. If we’re aiming for an output sample rate of 42,000Hz, we’d need to divide our clock signal by 4,000, so that we correctly get $\frac{168,000,000}{4,000} = 42,\!000$. For now, we’ll choose register values of PSC = 0 and ARR = 3999.

Why do we add $+1$ to the PSC and ARR in the relationship above?

On the STM32F4, PSC and ARR are 16-bit registers, meaning they range from 0 to 65,535.⁵ To save space and enhance program correctness, we assign meaningful behaviour to the value 0.

So in this page, when we say PSC = 0, we actually mean a prescaler divisor of 1.

Why 0 and 3999 specifically?

Other pairs of PSC and ARR can also work. We can choose any PSC and ARR which get us to our desired timer frequency. Play around and try different pairs of PSC and ARR!

Exercises for the reader:

What is the difference between different pairs, such as PSC = 0, ARR = 3999 vs. PSC = 1, ARR = 1999? (Hint: counter.)
Is there a PSC/ARR pair that is "better"?⁶

We can use STM32 CubeMX to initialise timer parameters. CubeMX allows us to generate code from these options, handling the conundrum of modifying the appropriate registers.

^{In CubeMX, we first select a timer on the left. We then enable a channel (here Channel 4) to generate PWM.⁷ We also set the prescaler and auto-reload so that our timer frequency is 42,000Hz.}

^{Some other settings in CubeMX to check.}

Remember to generate code once done.⁹ CubeMX should generate the following code in main.c:

static void MX_TIM8_Init(void)
{
    // --snip-- Initialise structs. --snip--

    htim8.Instance               = TIM8;
    htim8.Init.Prescaler         = 0;
    htim8.Init.CounterMode       = TIM_COUNTERMODE_UP;
    htim8.Init.Period            = 4000 - 1;
    htim8.Init.ClockDivision     = TIM_CLOCKDIVISION_DIV1;
    htim8.Init.RepetitionCounter = 0;
    htim8.Init.AutoReloadPreload = TIM_AUTORELOAD_PRELOAD_DISABLE;
    if (HAL_TIM_Base_Init(&htim8) != HAL_OK) {
        Error_Handler();
    }
    
    // --snip-- Initialise the clock source. --snip--
    
    if (HAL_TIM_PWM_Init(&htim8) != HAL_OK) {
        Error_Handler();
    }
    
    // --snip-- Initialise other things. --snip--
    
    HAL_TIM_MspPostInit(&htim8);
}

After initialisation, it's possible to change the timer frequency by setting the prescaler and auto-reload registers like so:

TIM8->PSC = 0;    // Prescaler: 1
TIM8->ARR = 3999; // Auto-Reload: 4000

This is useful for applications where the frequency is dynamic (e.g. playing music with a piezoelectric buzzer), but it's also useful when we're too lazy to modify the .ioc file.

Example: Playing with Timers

STM’s HAL library provides ready-made functions to interface with hardware.

HAL_TIM_Base_Start(&htim8); // Start the timer.
HAL_TIM_Base_Stop(&htim8);  // Stop the timer.

These functions are used to start/stop timers for basic timing and counting applications. Functions for more specialised modes (e.g. PWM) are available in stm32f4xx_hal_tim.h.

Digital-to-Analogue Converters (DACs) 🌉

Let's delve into our second topic today: digital-to-analogue converters (DACs).

Audio comes in several forms: sound waves, electrical voltages, and binary data.

^{Audio manifests in various forms. DACs transform our signal from the digital realm to the analogue world. (Source: Pluke, CC BY-SA 3.0, via Wikimedia Commons.)}

Since representations vastly differ, hence the need for interfaces to bridge the worlds. Between the digital and analogue realms, we have DACs and ADCs as mediators. Generally, DACs are used for output while ADCs are for input.

A Closer Look at DACs

Remember sampling? We took a continuous analogue signal and selected discrete points at regular intervals. An ADC is like a glorified sampler.

While ADCs take us from continuous to discrete, DACs (try to) take us from discrete to continuous. The shape of the resulting analogue waveform depends on the DAC implementation. Simple DACs will stagger the output at discrete levels. More complex DACs may interpolate between two discrete samples to “guess” the intermediate values. Some of these guesses will be off, but at least the signal is smoother.

^{On our STM board, signal reconstruction is staggered, like old platformer games—not that I've played any. At higher sampling rates, the staggered-ness is less apparent and the resulting curve is smoother.}

Example: Initialising the DAC

Let’s return to CubeMX to set up our DAC.

^{Enable DAC, and connect it to Timer 8 using the trigger setting. Our STM32F405 board supports two DAC output channels. This is useful if we want stereo audio output.}

^{Configure DMA settings for the DAC. We’ll cover DMA later.}

^{Enable interrupts for the DMA. These are needed to trigger DAC sends.}

Again, remember to generate code when finished.⁹ The MX_DAC_Init() function should contain the generated DAC setup code and should already be called in main().

Example: Using the DAC

On our STM32, DAC accepts samples quantised to 8 bits or 12 bits.¹⁰ We’ll go with superior resolution: 12 bits!

^{STM32 offers three different options to quantise and align DAC samples. We’ll only focus on the last option: 12-bit right aligned samples. (Source: RM0090 Reference Manual.⁸)}

For simplicity, let’s start with sending 1 DAC sample. This can be done like so:

// Start the DAC peripheral.
HAL_DAC_Start(&hdac, DAC_CHANNEL_1);

// Set the DAC value to 1024 on Channel 1, 12-bit right-aligned.
HAL_DAC_SetValue(&hdac, DAC_CHANNEL_1, DAC_ALIGN_12B_R, 1024);

This should output a voltage level of $\frac{1024}{2^{12}} = 25\%$ of the reference voltage $V_{\text{REF}}$. Once it starts, the DAC will continue sending that voltage out until we change the DAC value or call HAL_DAC_Stop().

We use DAC_CHANNEL_1 to select the first channel, and use DAC_ALIGN_12B_R to accept 12-bit right-aligned samples.

To fire a continuous stream of samples, we could use a loop and call HAL_DAC_SetValue() repeatedly. Let’s use this method to generate a simple square wave.

An aside. The default HAL_Delay() provided by STM will add 1ms to the delay time—well, at least in my version. I overrode it using a separate definition so that it sleeps the given number of ms.

void HAL_Delay(uint32_t ms)
{
    uint32_t start = HAL_GetTick();
    while ((HAL_GetTick() - start) < ms);
}

HAL_DAC_Start(&hdac, DAC_CHANNEL_1);

// Alternate between high (4095) and low (0).
uint8_t high  = 1;
while (1) {
    uint16_t sample = (high ? 4095 : 0); // max = 4095 = 2^12 - 1.
    high = !high;
    HAL_DAC_SetValue(&hdac, DAC_CHANNEL_1, DAC_ALIGN_12B_R, sample);

    // Delay for 5ms.
    HAL_Delay(5);
}

This generates a square wave with a period of 10ms, for a frequency of 100Hz.

^{Oscilloscope view of the signal. Oscilloscopes are very useful for debugging signals, especially periodic ones.}

But there are two issues with this looping method:

Using a while loop blocks the thread, meaning we block the processor from doing other things while outputting the sine wave. We may wish to poll for input or send out other forms of output (TFT/LCD, Bluetooth, etc.).
Since HAL_Delay() delays in milliseconds, it becomes impossible to generate complex waveforms at high frequencies, since that requires us to send samples at microsecond intervals.

In the next section, we’ll address these issues by combining DAC with timers and DMA.

Further Reading:

Deep Blue Embedded: STM32 DAC Tutorial

Direct Memory Access (DMA) 💉🧠

The final item on our agenda today! Direct Memory Access (DMA) may seem like three random words strung together, but it’s quite a powerful tool in the embedded programmer’s arsenal. How, you ask?

DMA enables data transfer without consuming processor resources. (Well, it consumes some resources, but mainly for setup.) This frees up the processor to do other things while DMA takes care of moving data. We could use this saved time to prepare the next set of buffers, render the GUI, etc.

DMA can be used to transfer data from memory-to-peripheral (e.g. DAC, UART TX, SPI TX), from peripheral-to-memory (e.g. ADC, UART RX), across peripherals, or across memory. In this post, we're concerned with one particular memory-to-peripheral transfer: DAC.

Further Reading:

Baeldung: How Do DMA Controllers Work?

Example: DMA with Single Buffering

We'll now try using DMA with a single buffer, see why this is problematic, and motivate the need for double buffering. If you’ve read this far, I presume you’ve followed the previous section by initialising DMA and generating code with CubeMX.

DMA introduces syncing issues. After preparing a second round of buffers, how do we know if the first round has already finished?

As with all processes which depend on a separate event, there are two approaches: polling and interrupts. In this context:

Polling: Block and wait until the first round is finished, then send.
Interrupts: Trigger an interrupt signal when transfer finishes, and start the next round inside the interrupt handler.

Which approach to choose depends on your application.

In our examples, we’ll poll to check if DMA is finished:

while (HAL_DAC_GetState(&hdac) != HAL_DAC_STATE_READY)
    ;

With DMA, we’ll first need to buffer an array of samples. Our loop will run like this:

Buffer samples.
Wait for DMA to be ready.
Start the DMA.

Do you notice a flaw in this approach? After starting DMA, we start buffering samples on the next iteration. We risk overwriting the buffer while it’s being sent.

Let’s try to implement it anyway and play a simple 440Hz sine wave.

#include <math.h>  // M_PI, sin

#define SAMPLE_RATE 42000
#define BUFFER_SIZE 1024
#define FREQUENCY   440

uint16_t buffer[BUFFER_SIZE];
uint32_t t = 0; // Time (in samples).

// Start the timer.
HAL_TIM_Base_Start(&htim8);

while (1) {
    // Prep the buffer.
    for (int i = 0; i < BUFFER_SIZE; i++, t++) {
        float val = sin(2 * M_PI * FREQUENCY * t / SAMPLE_RATE);
        buffer[i] = 2047 * val + 2047; // Scale the value from [-1, 1] to [0, 2^12-1).
    }

    // Wait for DAC to be ready, so that the buffer can be modified on the next iteration.
    while (HAL_DAC_GetState(&hdac) != HAL_DAC_STATE_READY)
        ;

    // Start the DMA.
    HAL_DAC_Start_DMA(&hdac, DAC_CHANNEL_1, (uint32_t*)buffer, BUFFER_SIZE, DAC_ALIGN_12B_R);
}

The results? As expected, pesky little artefacts invade our signal since our buffer is updated during DMA transfer. This may result in unpleasant clicks from our speaker.

^{Prep, wait, start, repeat. Artefacts distort the signal from time to time.}

But what if we prep, then start, then wait? This way, the buffer won't be overwritten; but this causes the signal to stall while prepping.¹¹

^{Prep, start, wait, repeat. The signal stalls (shown by horizontal lines) because the DAC isn’t updated while buffering.}

To resolve these issues, we'll unleash the final weapon in our arsenal.

Example: DMA with Double Buffering

We saw previously how a single buffer spectacularly fails to deal with "concurrent" buffering. With double buffering, we introduce an additional buffer. While one buffer is being displayed/streamed, the other buffer is updated. This ensures our audio can be delivered in one continuous stream.

In code, we’ll add another buffer by declaring uint16_t[2][BUFFER_SIZE] instead of uint16_t[BUFFER_SIZE]. We’ll also declare a variable curr (0 or 1) to index which buffer is currently available.

uint16_t buffers[2][BUFFER_SIZE]; // New: add a second buffer.
uint8_t curr = 0;                 // Index of current buffer.
uint32_t t   = 0;

// Start the timer.
HAL_TIM_Base_Start(&htim8);

while (1) {
    uint16_t* buffer = buffers[curr]; // Get the buffer being written.

    // --snip-- Same as before...
    // Prep the buffer.
    // Wait for DAC to be ready.
    // Start the DMA.
    // --snip--

    // Point to the other buffer, so that we
    // prepare it while the previous one
    // is being sent.
    curr = !curr;
}

Now our 440Hz sine wave is unblemished!

^{Waveform of a pure 440Hz sine tone.}

Double buffering is also used for video and displays, where each buffer stores a 2D frame instead of a 1D signal.

Example: Playing Multiple Notes with DMA and Double Buffering 🎶

With some minor changes, we can make our device generate audio for multiple notes. Let’s go ahead and play an A major chord!

// Prep the buffer.
uint16_t* buffer = buffers[curr];
for (int i = 0; i < BUFFER_SIZE; i++, t++) {
    // Compute value for each note.
    float a = sin(2 * M_PI * 440 * t / SAMPLE_RATE);
    float cs = sin(2 * M_PI * 554.37 * t / SAMPLE_RATE);
    float e = sin(2 * M_PI * 659.25 * t / SAMPLE_RATE);

    float val = (a + cs + e) / 3;  // Sum and normalise to [-1, 1].
    buffer[i] = 2047 * val + 2047; // Map [-1, 1] to [0, 2^12-1).
}

If you flash the above code and feed the output to an oscilloscope, you may find it doesn’t really work. Our signal stalls, for similar reasons as before.

Even with DMA, stalls may occur. This is usually a sign that buffering (and other processes) consume too much time. In this case, breaks in the data occur—the stream is no longer continuous, because the buffer doesn't finish prepping on time.

Optimisations 🏎

So our code is slow. How do we speed it up?

Here are a few common tricks:

Precompute constants.

Instead of computing 2 * M_PI * FREQUENCY / SAMPLE_RATE every iteration, we can precompute it before the loop, saving many arithmetic instructions.

// Precompute a factor of the 440Hz signal.
float two_pi_f_over_sr = 2 * M_PI * FREQUENCY / SAMPLE_RATE;

while (1) {
    // Prep the buffer.
    uint16_t* buffer = buffers[curr];
    for (int i = 0; i < BUFFER_SIZE; i++, t++) {
        // Use the precomputed value...
        buffer[i] = 2047 * sin(two_pi_f_over_sr * t) + 2047;
    }

    // ...
}

Wavetable synthesis.

Math functions such as sin can be computationally expensive, especially when used a lot. By caching the waveform in a lookup table, we can speed up the process of computing samples.
Increase the buffer size.

By increasing the buffer size, we spend less overhead switching between tasks.
Decrease the sample rate.

If all else fails, we can decrease the load by compromising the sample rate, say from 42000Hz to 21000Hz. With a buffer size of 1024, that means we’ve gone from a constraint of $\frac{1,024}{42,000} = 24.4$ms to $\frac{1,024}{21,000} = 48.8$ms per buffer.

To avoid complicating things, I lowered the sample rate to 21000Hz. This means changing the auto-reload register to 7999, so that our timer frequency is $$\frac{168,000,000}{(0 + 1) \times (7,999 + 1)} = 21,\!000\text{Hz.}$$

TIM8->ARR = 7999;

After all this hassle, we get a beautiful chord.

^{A nifty waveform of an A major chord (440Hz + 554.37Hz + 659.25Hz).}

Recap 🔁

By utilising both hardware and software, we reap the benefits of parallel processing while implementing an efficient, robust audio application. On the hardware side, we explored:

Timers, which are an useful and inexpensive way to trigger actions at regular intervals.
DACs, which enable us to communicate with a speaker by translating digital samples into analogue signals.
DMA, which enables data transfer with minimal processor resources. This way, we can process other things while streaming audio.

In software, we explored:

Double buffering, a software technique for buffering data to achieve continuous or faster output.
Various optimisations, which enable us to squeeze more processing into our tiny board.

When combined, we save processing resources, which can possibly be spent on additional features.

In case you want to go further, here are some other things to explore:

Generating stereo audio. We’ve generated audio for Channel 1. What about stereo audio for Channel 2? If you’re using reverb effects and wish for a fuller stereo sound, you’ll need an extra pair of buffers (and more processing!).
Streaming via UART (+ DMA).
Using SIMD instructions to buffer two (or more?) samples at a time.
- Other assembly-level bit-hacking tricks.
RTOS for multitasking.
Other boards or hardware with specialised audio features.

Hope you enjoyed this series of posts! Leave a comment if you like to see more or have any feedback!

Full Code

The complete code for DMA with double buffering has been uploaded as a GitHub Gist. It hasn't been fully optimised yet. I'll leave that as an exercise for the reader.

Footnotes

Each of these components (especially hardware) deserve their own post to be properly introduced; but for the sake of keeping this post short, I’ll only introduce them briefly and link to other resources for further perusal. ↩︎
Timer/Counter in Embedded System ↩︎ ↩︎
How do microcontroller timers work? – A decent article on timers. Diagrams are in French though. ↩︎
The extent of timer events depends on hardware support. Timers can do a lot on ST boards. For other brands, you may need to check the datasheet or reference manual. ↩︎
Some other timers have 32-bit ARR registers. But eh, we can achieve a lot with just 16-bit ones. ↩︎
What is the difference between pairs of prescaler/auto-reload, such as PSC = 0, ARR = 3999 vs. PSC = 1, ARR = 1999?
Indeed, given a fixed clock frequency, the same timer frequency will be generated (since the divisor is the same: 2000). However, the difference lies in the counter. Recall each step of auto-reload equals a step of the counter.
The counter is used in calculating the on-time (or duty cycle). By using a higher ARR, we gain a higher resolution in the counter, which allows us to say, control servos with finer granularity. Thus, a lower prescaler is often preferred.
Of course, different vendors may implement timers differently or have different features attached to timer peripherals. Other considerations may come into play, depending on the vendor and your application. ↩︎
We chose Timer 8 (with Channel 4) because it's an advanced control timer (a beefy boi!), capable of a lot, though probably overkill for our simple examples. The timer and channel you use depends on your STM board and model. If you’re following along with this post, make sure to choose a timer which has DMA generation. When in doubt, refer to the reference manual.⁸ ↩︎
STM's Official Reference Manual for F405/F415, F407/F417, F427/F437, F429/F439 boards. Definitely something to refer to if you’re working on one of those boards. ↩︎ ↩︎
In CubeMX, you can generate code by choosing the Project > Generate Code menu option. When coding, keep in mind that only code between USER CODE BEGIN and USER CODE END comments will be preserved by ST's code generator. ↩︎ ↩︎
There are pros to using 8-bit or 12-bit DAC. 8-bit conversion is faster, whereas 12-bit offers higher resolution. To slightly complicate things, the 12-bit DAC option on our STM32 can be aligned either left or right. That is, we can choose whether our data takes up the first 12 bits or last 12 bits on a 16-bit (2-byte) space. Alignment exists to save you a shift operation, which depends on your application. ↩︎
Not sure if stall is the right word. Let me know if there's a better one. ↩︎

Midnight Enigma

2023-04-13T00:00:00Z

Midnight Enigma is composed for a musical challenge constraining pitches within two octaves, with the bonus theme of game music.¹ Pitch and range constraints aren't that unusual. When writing for any instrument or voice, it's important to keep in mind its range and register. You probably don't want to see (let alone hear) a bass singer singing an A5 note.

Though being limited to two octaves bordered on draconian, I made do. I had to cut some corners melodically, since imitation was more difficult and flamboyant melodies couldn't be expressed. Nonetheless, I focused on exploring other aspects of music: the rhythm, extended chords, microtones, and whatnot. I also decided to have fun with the Phrygian mode a bit more.² I started playing with this mode in Remorse (without knowing about it 🫢), and still find it very interesting to use. (Perhaps this will become the topic of a blog post?)

I decided to go with a band/minimalist/electronic fusion with a mix of drums, Reichian phasing, and electronic-inspired delay. I took (mental) inspiration from open world games such as Minecraft, where a huge number of possibilities lie at your fingertips and consequences aren't drastic.

A majority of the piece was composed during midnight hours in the days before the challenge deadline, hence the title. (Yes, I messed up my sleep schedule then.)

On another note, this was my first time using MIDI vocals. They sound better than I expected; might play with them in future pieces. :)

Enjoy~

Footnotes

The octaves I chose were G2-G3 + G4-G5. Pitches can be anywhere inside these ranges, but not between (G3-G4) or outside. ↩︎
The Phrygian mode is characterised by the flat-2nd. As Midnight Enigma is primarily in C minor, the flat-2nd would be the D-flat. ↩︎

Déjà Vu – Cycle of Power

2023-04-09T00:00:00Z

And in the days of those kings the God of heaven will set up a kingdom that shall never be destroyed, nor shall the kingdom be left to another people. It shall break in pieces all these kingdoms and bring them to an end, and it shall stand forever…

— Daniel 2:44 (NIV)

Organisations, institutions, nations, governments, civilisations. Whether big or small, strong or weak, aggressive or meek, each eventually fades and another rises to take its place. Though on the outside, they seem different; in the inside, they are underscored by a common denominator: the human nature of selfishness and stupidity (or put bluntly, sin).

Does this mean we should dissolve into anarchy? Well, no. Organisation, with enough trust, is better than disorganisation. Proper care should be taken to manage resources, especially on a nation-wide level.

What has been is what will be, and what has been done is what will be done, and there is nothing new under the sun.

— Ecclesiastes 1:9 (NIV)

"Gibberish, ChatGPT is new under the sun!" But if we take into account all history and the timeline of events, there’s really not much new. Did you know before computers were invented, a computer used to be a job title? It wouldn’t be surprising if, later on, an engineer or artist refers to a computer program.

My point lies in the cycle of power and time. Developments, alliances, wars, laws, technologies, and disasters become mere motifs in the grand scheme of things. Standards of living, science, technology, and health have all improved over the past millennia, but nations still rage like a ceaseless tempest. Conflicts arise, almost inevitably, as if they were meant to be. Such is human nature.

Déjà Vu – Cycle of Power is my impression of all the above. The musical ideas in this piece were not without inspiration. Some notable inspirations were Fauré's melodic tune in Libera Me, Holst's militaristic rhythm in Mars, and John Adams’ minimalist works.

This piece is scored for orchestra, with repeated sections to capture the essence of a cycle. Throughout this timeless journey, civilisations rise and fall. They come in different shapes and sizes, from the grand to the subdued, from the pacifists to the militarists. A fleeting escape is made into a lucid dreamlike passage where flourishing arpeggios elevate us from reality; yet order lurks in the background. The piece culminates in an oddly familiar setting as history seemingly repeats itself.

Each motif and section comes to an end, their place taken by a new motif—and ultimately silence. But there’s a motif which doesn’t end—which can’t be captured on sheets of music.¹

Although the world we live in may seem like an inescapable matrix², each of us has the opportunity to learn, to teach, to experience different perspectives, and to share the hope of something unseen—something more.

On this Easter Sunday (and well… all Easter Sundays), we commemorate the resurrection of Jesus Christ, remembering the glorious occasion of His power—and victory—over death, sin, and evil.

Footnotes

How do you apply dimension reduction on the infinite without discarding an ounce of its beauty? Good luck trying to use PCA or whatever dimensionality reduction algorithm is popular at the time of reading. I mean, I suppose it's still possible to focus on some aspects and reduce accordingly. Maybe for a later time. ↩︎
Alluding to the movie. Go watch it if you haven’t already. _{cough - You know who you are.} ↩︎

Digital Audio Synthesis for Dummies: Part 2

2023-03-09T00:00:00Z

This is the second post in a series of posts on Digital Audio Processing. Similar to the previous post, this post stems from a lil’ MIDI keyboard project I worked on last semester and is an attempt to share the knowledge I've gained with others. This post will dive into the wonderful world of audio synthesis and introduce two important synthesis techniques: additive synthesis and wavetable synthesis.

Audio Synthesis 🎶

Where do audio signals come from? Our signal might be…

recorded. Sound waves are picked up by special hardware (e.g. a microphone) and translated to a digital signal through an ADC.
loaded from a file. There are many audio formats out there, but the most common ones are .wav and .mp3. The .wav format is simple: just store the samples as-is. Other formats compress audio to achieve smaller file sizes (which in turn, means faster upload/download speeds).
synthesised. We generate audio out of thin air (or rather, code and electronics).

I’ll mainly focus on synthesis. We’ll start by finding out how to generate a single tone, then learn how to generate multiple tones simultaneously.

Buffering 📦

A naive approach to generate audio might be:

Process one sample
Feed it to the DAC/speaker

But there are several issues with this: function call overhead may impact performance, and we have little room left to do other things. For the sound to play smoothly while sampling at 44100Hz, each sample needs to be delivered within $\frac{1}{44100}$ s = $22.6$ µs.

A better approach is to use a buffer and work in batches. The buffer will hold onto our samples before feeding it to the speaker.

Process $N$ samples and store them in a buffer
Feed all $N$ samples to the DAC/speaker

We'll focus more on step 1 (processing) for now. We'll cover step 2 (output) in the next post.

In a previous post, we discussed quantisation and how different representations (such as integers and floats) are suited for different tasks. Integers are discrete numbers, while floats are (imprecise) real numbers. Since we're concerned with audio processing, we'll be using floats and quantising from -1 to 1.

In C/C++, we can generate a sine tone like so:

#define SAMPLE_RATE 44100  // Number of samples per second.
#define BUFFER_SIZE 1024   // Length of the buffer.

// Define an array for storing samples.
float buffer[BUFFER_SIZE]; // Buffer of samples to populate, each ranging from -1 to 1.

/**
 * Generate samples of a sine wave and store them in a buffer.
 * @param freq  Frequency of the sine wave, in Hz.
 */
void generate_samples(float freq) {
    // Populate the buffer with a sine tone with frequency `freq`.
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = sin(2 * PI * freq * i / SAMPLE_RATE);
    }
}

And that’s it—we’ve just whooshed pure sine tone goodness from nothing! Granted, there are some flaws with this method (it could be more efficient, and the signal clicks when repeated); but hey, it demonstrates synthesis.

Note on Buffers: Usually, the buffer size is medium-sized power of 2 (e.g. 512, 1024, 2048, 4096...). This enhances cache loads and processing speed (dividing by a power of 2 is super easy for processors!).¹

The Fourier Theorem 📊

One fundamental theorem in signal processing is the Fourier Theorem, which relates to the composition of signals. It can be summarised into:

Any periodic signal can be broken down into a sum of sine waves.

We can express this mathematically as $$ f(x) = a_0\sin(f_0x + b_0) + a_1\sin(f_1x + b_1) + \cdots + a_n\sin(f_nx + b_n) $$ where $a_i$, $f_i$, and $b_i$ are the amplitude, frequency, and phase of each constituent sine wave.

The Fourier Theorem and Fourier Transform are ubiquitous in modern day technology. It is the basis for many audio processing techniques such as filtering, equalisation, and noise cancellation. By manipulating the individual sine waves that make up a sound, we can alter its characteristics and create new sounds. The Fourier Transform is also a key component in compression, such as the JPG image format.

What’s cool about this theorem is that we can apply it the other way: any periodic signal can be generated by adding sine waves. This lays the groundwork for additive synthesis and generating audio with multiple pitches (e.g. a chord).

Additive Synthesis ➕

The principle of additive synthesis is pretty straightforward: signals can be combined by adding samples along time.

^{Example of additive synthesis. The first and second signal show pure sine tones at 440Hz ($s_1$) and 660Hz ($s_2$). The third signal adds the two signals ($s_1 + s_2$). The fourth signal scales the third signal down to fit within $[-1, 1]$ ($(s_1 + s_2) / 2$). (Source Code)}

To sound another pitch, we simply add a second sine wave to the buffer.

#define SAMPLE_RATE 44100
#define BUFFER_SIZE 1024

float buffer[BUFFER_SIZE];

/**
 * Generate samples of two sine waves played together and store them in a buffer.
 * @param freq  Frequency of the first sine wave, in Hz.
 * @param freq2 Frequency of the second sine wave, in Hz.
 */
void generate_samples2(float freq, float freq2) {
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = 0.5f * sin(2 * PI * freq * i / SAMPLE_RATE);
        buffer[i] += 0.5f * sin(2 * PI * freq2 * i / SAMPLE_RATE);
    }
}

Again, the code above populates the buffer with 1024 samples of audio. But this time, we introduced a second frequency freq2 and added a second sample to the buffer. We also made sure to scale the resulting sample back down to the $[-1, 1]$ range by multiplying each sample by 0.5.

We can see additive synthesis in action with some help from Audacity.

Let’s start off with one tone.
- Generate a 440Hz tone (Generate > Tone… > Sine).
- Play it. You’ll hear a pure tone.
Now let’s add another tone.
- Make a new track (Tracks > Add New > Mono Track).
- Generate an 880Hz tone in the new track. Same method as above.
- Play it to hear a beautiful sounding octave.

During playback, Audacity will combine the samples from both tracks by summing them and play the summed signal.

You can try layering other frequencies (554Hz, 659Hz) to play a nifty A Major chord.

Fundamental Frequency

A side note. When combining two frequencies with additive synthesis, something subtle happens. The ground shifts and our feet fumble! The fundamental frequency implicitly changes!

DIY Example

Fire up Audacity.
Generate a 400Hz sine tone. (Generate > Tone...)
Generate a 402Hz sine tone on a different track.
Play the audio and observe. How many times does the audio peak per second?

It peaks twice per second. (That is, we implicitly added a 2Hz signal beneath!)

Specifically, the fundamental changes to the greatest common divisor (GCD) of the two frequencies.

More notes:

When we play a note and its 5th (in Just Temperament), say 440Hz and 660Hz, our fundamental is 220Hz.
With octaves, the fundamental frequency is just the frequency of the lower note.
This also leads to some interesting phenomena.
- When we play two super-low-register notes a semitone apart, we get funny, dissonant pulses emanating from our keyboard or piano.
- This may also lead to unpleasant buzzes when mixing synths, possibly due to frequency modulation on top of a steady tone.

Wavetable Synthesis 🌊

In previous code blocks, we computed samples using sin(). But what if we wanted to compute something more complex? Do we really just reverse the Fourier theorem and apply additive synthesis on a bunch of sine signals? It turns out there's a better way.

A more efficient approach is to interpolate over pre-generated values, sacrificing a bit of memory for faster runtime performance. This is known as wavetable synthesis or table-lookup synthesis. The idea is to pre-generate one cycle of the wave (e.g. a sine) and store it in a lookup table. Then when generating samples for our audio, we would look up the pre-generated samples and derive intermediate values if necessary (via interpolation).

This is akin to preparing a cheat sheet for an exam, but you're only allowed to bring one sheet of paper—space is precious. You decide to only include the most crucial equations, key points, and references. Then when taking the exam you refer to the cheat sheet for ideas, connect the dots, and combine them with your thoughts to form an answer.

^{Example of wavetable synthesis. The blue dots (above) show a pre-generated wavetable of length 32. The red dots (below) are samples of a 8Hz sine wave sampled at 100Hz, generated by interpolating on the wavetable. (Source Code)}

Wavetable synthesis can be implemented in C++ like so:

#define SINE_WAVETABLE_SIZE 256
#define SAMPLE_RATE 44100  // Number of samples per second (of the target waveform).
#define BUFFER_SIZE 1024

// The pre-generated wavetable. It should capture one cycle of the desired waveform (in this case, a sine).
float sine_wavetable[SINE_WAVETABLE_SIZE] = { /* pre-generated values ... */ };

// The phase indicates how far along the wavetable we are.
// We're concerned with two components: the integer and decimal.
// The integer part indicates the index of left sample along the wavetable, modulus the size.
// The decimal part indicates the fraction between the left and right samples.
float phase = 0;

float buffer[BUFFER_SIZE];

/**
 * Generate samples of a sine wave by interpolating a wavetable.
 * @param freq  Frequency of the sine wave, in Hz.
 */
float get_next_sample(float freq) {
    size_t idx = size_t(phase); // Integer part.
    float frac = phase - idx;   // Decimal part.

    // Get the pre-generated sample to the LEFT of the current sample.
    float samp0 = sine_wavetable[idx];

    // Get the pre-generated sample to the RIGHT of the current sample.
    idx = (idx + 1) % SINE_WAVETABLE_SIZE;
    float samp1 = sine_wavetable[idx];

    // Interpolate between the left and right samples to get the current sample.
    float inter = (samp0 + (samp1 - samp0) * frac);

    // Advance the phase to prepare for the next sample.
    phase += SINE_WAVETABLE_SIZE * freq / SAMPLE_RATE;

    return inter;
}

void generate_samples_w(float freq) {
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = get_next_sample(freq); // We've offloaded the calculations to `get_next_sample`.
    }
}

For a sine wave, we don't gain much in terms of performance. But when it comes to generating complex waveforms, wavetable synthesis rocks!²

Wavetable synthesis is commonly used by MIDI to generate sounds. Each instrument has its own soundfont, which is a collection of wavetables of different pitches. This unifies the synthesis approach for all instruments, as some may be simple to generate (e.g. clarinet) while others are more complex.

Additive synthesis and wavetable synthesis serve two very different purposes!

Additive synthesis aims to combine multiple waveforms, of any shape and size (e.g. playing chords, or combining guitar and voice tracks).
Wavetable synthesis aims to generate a specific waveform (in a fast manner).

Besides this software approach, we can also leverage hardware to speed up processing. But this is a matter for the next post.

Exercise for the reader:

What variables affect the speed at which we iterate through the pre-generated wavetable?
What happens if we try to generate a waveform with frequency equal to half the sample rate? Or with frequency equal to the sample rate itself?

Recap 🔁

Audio generation is pretty fun once we dive deep, as are its applications: toys, electronic instruments, virtual instruments, digital synths, speakers, hearing aids, and whatnot. As before, I hope we communicated on the same wavelength and the information on this post did not experience aliasing. 😏

In the next post, we'll dive even deeper into audio synthesis (particularly in embedded systems) and engineer a simple tone generator.

To recap…

Audio samples may come from several sources. It may be recorded, loaded from a file, or synthesised.
We can synthesise musical pitches by buffering samples and feeding them to hardware.
According to the Fourier Theorem, all signals can be broken into a summation of sine waves.
To combine audio signals, we can apply additive synthesis.
- This also allows us to play multiple pitches simultaneously (chords).
We can generate complex waveforms by using wavetable synthesis, which trades memory for speed by sampling pre-generated signals.

Further Reading:

Waveforms (Sine, Square, Triangle, Sawtooth)
- Most audio synthesis tutorials cover simple waveforms; but as internet content is saturated here, I'll just drop a couple links. I couldn't find an article I like that introduces these waveforms in all their glory. If you know of better articles, let me know.
- Perfect Circuit (conceptual, high-level)
- Electronics Tutorial (geared towards electronics; would be a nice read to prepare for the next post)
Beginner’s Guide: Everything you need to know about synthesis in music production – Introduces more forms of audio synthesis, geared towards music production.

Footnotes

Boy, do I have a lot to say about buffers. Why is the buffer size important? Small buffers may reduce the efficacy of batching operations (which is the primary purpose of buffers). Large buffers may block the processor too much, making it sluggish to respond to new input. Choosing an appropriate buffer size also depends on your sampling rate. With a buffer size of 1024 sampling at 44100Hz, we would need to generate our samples every $\frac{1024}{44.1\text{kHz}} \approx 23.2$ ms. On a single processor, this means we have less than 23.2 ms to perform other tasks (e.g. handle UI, events, etc.). ↩︎
Guess what? There are more ways to optimise wavetable synthesis—so it'll rock even more! See the open source LEAF library for an example of optimised wavetable synthesis in C. ↩︎

Amorama

2023-02-14T00:00:00Z

Amorama was composed for a fun little musical challenge on the theme of love ❤️ with the constraint of using the rarely-sighted 13/16 metre. Often, music is made for more balanced, basic metres such as 4/4, 3/4, and 6/8. For some reason, we seem to prefer small subdivisions of 2 or 3. We can still apply this principle with 13/16 though, and it turns out that's what most contestants did. :)

I chose to blend Cuban rhythms, some sweet little tunes, and—since it's February—various love themes (see if you recognise any!). The piece is scored for violin, piano, double bass, castanets (the clacky things), and drums. The violin and piano are responsible for the melody, while the bass, castanets, and drums primarily generate rhythm and provide the swaying dance line.

As encouraged by Spotify advertisements, go and dance with abandon!

Happy Valentine's Day, God bless, and enjoy~ ❤️

Remorse

2023-01-27T00:00:00Z

A reflection of the past, composed and mixed during my 7-day covid quarantine.

Time ebbs past,
As anguish holds fast.
Without you I’m filled with remorse,
For you are my one driving force.

As the season turns,
My heart still churns.
I’ll leave my sins and remorse,
Heading on a different course.

The purpose of composing this piece is threefold: it served as an expression of faith and emotions, a challenge for a Capture-the-Flag (CTF) competition, and an enjoyable way to pass quarantine. There wasn't any priority to a particular purpose; all of them seemed to develop together.

This piece was composed for the HKUST Firebird 2023 Internal CTF. Such competitions are designed to challenge players with cybersecurity know-how. Once players identify and exploit a vulnerability, they are rewarded with a flag (a piece of text), which awards points to the player when submitted. Occasionally, some challenges deviate from the norm and test players in other areas. In this case, my challenge tested players in analysing music and patterns.

When composing this piece, I aimed to compose something listenable and motivic. I decided to keep constraints flexible within limits. If a music is too constrained, it sounds choked, inevitable, or unimaginative. Some music ciphers out there encode letters into pitches and duration. This was a bit too far for my liking, as it becomes painstakingly difficult to find a pleasurable tune.

As hinted by the title, I first translated the flag into Morse using an online converter with the extended Morse character set. This allowed for some punctuation such as (, -, _, and funky non-ASCII characters such as é.

With plaintext, we need to encode at least 50 distinct letters. With Morse, we just need to encode 3 instead: ., -, and space (as a word separator). So things are relatively simple.

I toyed around with a few ideas of encoding these three characters. Eventually I ended up with this mapping:

. → "Note On" in upper stave (treble clef)
- → "Note On" in lower stave (bass clef)
space → "Note On" in both staves

This lent the music well to a contrapuntal form, with the occasional grace-note or glissando to pack characters in a way that preserves the melodic contour. After composing the sheet music, I ended up mixing it with Reaper for some extra flair and charged resonance, making the night seem younger.

Enjoy the result!

Seaside Garden

2023-01-01T00:00:00Z

Seaside Garden is composed for HKUST's Call for Scores for the School Anthem.¹ I present my rationale below, at the expense of being cheesy.

Rationale

My journey with HKUST started three and a half years ago. This was a period of becoming, exploration, and development. To me, the university is like a garden: cultivating and nurturing flora and fauna, a peaceful sanctuary, a place of life and beauty. Situated near the sea, HKUST stands firm as waves ceaselessly crash against its foundation.

But what's a garden without birds? A garden is a place where birds can flourish and grow before departing and exploring the outside world. There are countless kinds of birds: wisened owls, agile peregrines, shrewd crows, and hardworking weavers among others. The garden and birds live in symbiosis. The garden provides nutrition and shelter; in exchange, birds spread seeds and pollinate flowers. It is well-known that birds digest fruit from one end and discharge (plus fertilise!) seeds from the other end, planting these gifts to who-knows-where. Without birds, the world becomes a very dark and dismal place.

HKUST's iconic sundial goes by many names. In Cantonese, it's colloquially known as the turkey (literally, fire chicken). Some university groups such as the Red Bird and Firebird teams name themselves after the sculpture. Indeed, it is difficult not to feel a fiery sensation when looking at the sundial.² The name it was officially given was "Circle of Time", a timeless tribute to humanity and a reflection of Earth's endless cycle. Although the two HKUST campuses are separated by a border, the two are united by such a monument.

Further references include the purple orchid, storms, and other subtle things. The purple orchid—aka the Bauhinia, Hong Kong's national flower—was included in view of HKUST's commitment towards the development of Hong Kong. Storms, on the other hand, are an inevitable part of life in Hong Kong and neighbouring ports.

Composed in the final week of the year 2022 and released on New Year's Day 2023, it is my wish that this song instils vision and harmony in learning and teaching throughout the HKUST community.

Lyrics

I'm no lyricist, but for completeness and posterity here are my jank lyrics.

Verse 1
Seedlings sprout forth
As purple orchids bloom.
And birds start the morning
As they sing a joyful tune.

Pre-Chorus 1
Oh the sun rises and falls,
In an endless Circle of Time.
And clouds can't hide the rays that shine from above.

Chorus 1
But though storms and strife may rage all night,
Our journey carries on.
Oh, the seaside garden,
Our community.

Through storms and strife we'll find our way
As one in unity.
In the seaside garden,
Our community.

Verse 2
Dandelions
Blow far, far away.
And birds leave the treetops
As they fly off on their way.

Pre-Chorus 2
The stars rise and fall
In an endless Circle of Time.
And clouds can't hide our voice and fiery call

Chorus 2
And through the books, the toil,
The sleepless nights,
Our journey carries on.
Though storms and strife may rage all night,
Our path is not forgone
Let vision guide all our hearts
Through vivid dreams and youthful hope.
In the seaside garden,
Our community.³

Chorus 3⁴
And through the times, the trials,
The countless miles,
Our venture knows no end.
Wind and water carry far
The gifts that we present.
Let virtues latch on our wings
With steadfast bonds and harmony.
Oh seaside garden,
Our family.³

Outro
From the seaside garden,
We fly!
We fly!

Footnotes

It didn't get accepted, sadly. But that's okay. ↩︎
Up to interpretation, of course. Perhaps it's a fiery love. A fiery joy. A fiery excitement. It could just as well be a fiery hatred, anger, or diarrhoea. ↩︎
May interchange between "community" or "family". ↩︎ ↩︎
May repeat Chorus 2 instead, or sing Chorus 3 twice. Ad lib. ↩︎

STM32 MIDI Keyboard

2022-12-20T00:00:00Z

This project was made for a course on embedded systems and is published online.

Synopsis

The STM32 MIDI Keyboard was a project aimed to practice embedded systems design while also have fun developing a tactile music application. Here are some features found on the keyboard:

2+ octaves (29) piano keys to flexibly play various melodies
Supports multi-press, so that we aren't stuck with boring one-note tunes and can play chords
Volume control, so that we don’t disturb our neighbours
Metronome, in case the user can’t keep track of tempo
TFT display and menu selection
Record and playback music (supports multiple channels!)
Load/store MIDI in flash memory, in case power runs out
Send MIDI signals through Serial USB (UART)
- Receiving is handled by a script (receive.py)
Extra fanciful features:
- Transpose: shift pitches up/down on the diatonic scale
- Auto-Chord: quality-of-life function to play octaves, triads, and open triads by just pressing the root key
- Instruments: supports playback of sine, triangle, square, and sawtooth signals

Development

I was lucky to work with another member of the Robotics Team at HKUST. She did the electrical work of designing and soldering the PCB. Both of us weren't familiar with mechanical design, so we ended up sticking things onto plywood and screwing things up together. But hey, at least it's a minimum viable product.

Although the course provided us with a multi-functional board (STM32F103VET6), we ended up using with a different board (STM32F405). The F4 series comes with more processing power, flash memory, and RAM. These are important considerations when it comes to a real-time music system. Audio buffering needs to be fast and steady, and sufficient memory is required for storage and buffering.

We used C++20 for the project.¹ Although most objects and peripherals were singletons, classes (and templates!) proved useful, especially for containers (such as a fixed-size vector). C++ also allowed us to have static reflection for enums, thanks to magic_enum. The alternative would've been X-macros or hard-coding, both less maintainable options.

Most MIDI keyboards out there don't produce sound by themselves, and sometimes I'd rather just noodle around without having to set up any computer software. So one of our goals was to have the keyboard produce sound. This was pretty straightforward. Slap a timer, DMA, and DAC together, and BAM—non-blocking audio output!

We had more trouble with the speakers. They truly annoyed the heck out of us. Earlier on, we used a pair of small woofers. These were connected to an amplifier (because the voltage delivered by the board's DAC was not enough). The audio output generated when playing one tone (e.g. 440Hz sine) was fine. However for some unknown reason, playing multiple tones (e.g. 440Hz sine + 660Hz sine) was catastrophic. I asked a question on dsp.se, where some users suggested it being a psychoacoustic issue. Thankfully, that was not the case. Eventually we solved the issue by changing to different speakers, the kind used by end-users.

Concluding Remarks

It was quite relaxing and nice to have a (semi-?)personal project developing and designing an embedded application from head to tail. The instructors for the course were really helpful as well.

If you're interested in trying something similar, here are some things we planned but didn't incorporate into our project:

On/off switch
Rechargeable battery
MIDI-USB output + Real-time MIDI input into software
LED backlight under the keyboard?
Stereo audio (L/R)
Reverb/Chorus settings
Note velocity! (Our buttons weren't responsive enough—even though we used the Yamaha ones.)
Better storage (currently we only load/store MIDI for one piece)
Explore and use FluidSynth
More instrument options

I've also published some follow-up tutorials on digital audio synthesis, in case you're interested in getting your feet wet with audio processing:

Footnotes

I was excited to try C++20 modules—which gcc-arm v11 supports!—but CMake doesn't support modules yet. The issue is still ongoing as I write. I think they're trying, but I guess nobody likes dealing with compilation order? C'mon CMake! It's been three years already! ↩︎

MuseScore Navigation Plugins

2022-09-10T00:00:00Z

MuseScore, a music notation desktop application, allows mini-extensions through its QML Plugins. MuseScore is built with the Qt framework and leverages the QML ecosystem for rapid prototyping and user-developed plugins.

QML is a fun language to work with. Generally, the UI is coded in a declarative style and the logic is coded in JavaScript. This, of course, has the downside of losing the static type-checking of C++; so I would often be in the situation where I need to reload and fix things several times before getting it functioning properly.

But the freedom and “I don’t care whether your types are correct” attitude of JS were quite nostalgic. Having played with QML/JS before, I decided I wanted to write some MuseScore plugins for fun ~~and also because I felt a need for them~~.

In this post, I’ll introduce my first set of MuseScore plugins and give a brief developer’s account of them. These plugins aren’t really related to music. Rather, they’re inspired by VSCode plugins and features which I find useful. In a way, the plugins below make MuseScore more of a developer’s second home.

todo-list

An update has been published for MuseScore 4.1+. Please refer to the new release.

View the project on: MuseScore / GitHub.

In software development, to-dos are commonly added into source code as reminders to the poor developers toiling away writing code. To-dos come in many forms: bugs to be fixed, issues to be resolved, feature requests, etc.

Sometimes when composing, I find myself lost in a sea of to-dos. During a composing session, I might drop a to-do note to follow-up on it next time.

Here are some examples of to-dos I might encounter while composing:

TODO: Explore different chord progressions for this transition.
TODO: More brass to this section.
FIXME: Playback sounds wonky.
TODO: Revise counterpoint.
TODO: Add bowing articulation to strings.

To allow for different to-do styles and text, I provided several settings for the user to modify. These are listed in on the GitHub readme.

Developer’s Note

This plugin is inspired by the todo-tree plugin in VSCode, which searches files in the current workspace for TODOs/FIXMEs and lists them in a tree view. I toyed with idea of replicating this on MuseScore—listing to-dos on all open scores—but eventually decided to embrace the KISS principle and just list to-dos for the current score.

When starting, I took reference of jeetee’s annotation plugin. I noticed jeetee used Qt Quick Controls 1.0 instead of 2.0 used in some other plugins. Apparently, QML had made some drastic changes to the styling of controls (buttons, checkboxes, etc.). In 1.0, controls used the native style (e.g. Apple’s aqua style for macs). On the other hand, 2.0 controls require developers to customise styling; this may sound great for design flexibility, but in my experience it’s annoying to get it working with both light and dark themes.

^{Qt Quick Controls 1.0 vs 2.0. The latter comes with barely any default and takes more effort to properly set up.}

Implementation-wise, the todo-list plugin aims to be self-contained and simple. Self-contained, meaning that everything is in a single .qml file, so that the user doesn’t need to bother with structure too much. Simple, meaning that it doesn’t store too much metadata. The plugin only stores the configuration options mentioned above. I avoid storing data such as measure and staff—which are alike the x and y position in a score—because things get messy when the stored to-do is displaced, e.g. when a user inserts a bunch of measures or removes an instrument.

Even though I’ve used Qt before, I was still surprised with how easy it was to set up a form dialog to configure user settings. Just slap together a Dialog, GridLayout, several Labels and TextFields, code the logic, and violà—we have our settings dialog.

History

View the project on: MuseScore / GitHub.

This plugin keeps track of where your cursor has been so that you can easily jump back and forth between different points in time (effectively making you a time traveller!).

When programming in an IDE, it is sometimes necessary to jump to a different part of code, then jump back to where you were before. For example, you might want to check the implementation of a certain function.

Similarly, when editing a score one might wish to visit a section, perhaps copy something, then return to their previous position.

The History module comprises three separate plugins:

Go Back,
Go Forward, and
the UI.

The first two are action plugins—plugins without any visual component, and just run—whereas the third contains a simple UI. The need for a UI makes the module slightly inflexible for reasons explained below.

Developer’s Note

There are some limitations to this module, the root cause being the limited MuseScore plugin API.

For the plugins to work properly, the UI plugin must be activated at all times. This plugin is responsible for recording cursor positions, since MuseScore does not provide a way for plugins to record score/cursor information in the background. We could start a subprocess, keylog user input, and interpret it to determine where the cursor currently is… but this is of course out of the question, it’s much too tedious and not worth the effort.

Another drawback is that plugins can’t jump across scores. This is very convenient in IDEs when you have several files open and want to quickly check/copy something from a different file. In MuseScore however, this simply isn’t possible (as far as I could see). So for now, we’ll have to be content with keeping cursor history isolated within each score.

Bookmarks

View the project on: MuseScore / GitHub.

Marks down a section, saving it so that you can easily return to it later without the pain of scrolling.

This is really useful for large scores, where scrolling from front to middle to end can be pain. An existing built-in way to jump around sections is to use rehearsal marks plus MuseScore's timeline. However, I find this insufficient since a rehearsal mark only carries horizontal positioning info (measures), but not vertical positioning info (staffs).

The Bookmarks module comprises four plugins:

Go to Previous Bookmark,
Go to Next Bookmark,
Toggle Bookmark at Cursor, and
Clear All Bookmarks.

All of these are action plugins, so in my mind they’re fairly simple and straightforward.

Developer’s Note

This module was inspired by the VSCode bookmarks plugin. In VSCode, bookmarks allowed you to jump between bookmarks across files and long stretches of code. Sadly alike History, Bookmarks doesn’t jump across files.

Something else to note—and this applies to all my plugins here: currently, I use a rather hacky method to jump to the selected notes:

cmd("note-input")
cmd("note-input")

We call the command twice to toggle between the two different modes (default and note-input).

However, when jumping to a specific measure + staff, MuseScore will scroll to the correct measure, but not the staff. Logic-wise, nothing is affected since the correct measure + staff are selected. The problem is just a UI issue which the MuseScore Plugins API doesn’t have a solution for. 😢

For History and Bookmark, I decided to use separate JS files to hold all the logic. This promotes code reuse. For example, both History: Go Back and History: Go Forward use the same underlying function to iterate across the score.

I tried commenting my code cleanly, in case I need to come back to it later; I’m quite forgetful.

JavaScript’s lack of type checking really irks me. I’m toying with the idea of using TypeScript and compiling the file to JavaScript for testing. I’ve already set up a MakeFile for packaging (zipping) the files anyway, so might as well add a rule that compiles .ts into .js and gain type safety.

Epilogue

If you’ve read this far, then you’re probably aware that these plugins aren’t perfect. But most of these issues can't be trivially fixed due to limitations with the Plugin API.

In my development roadmap, I’ve planned several new (music-related!) plugins. As MuseScore 4 is coming out, I’ll also need to plan the maintenance of the above plugins.

If you’re interested in using the plugins or contributing, you can check the MuseScore or GitHub links below.

To-Do List: MuseScore / GitHub.

History + Bookmarks: MuseScore / GitHub.

Thanks for reading, and happy musing!

Space Penguin

2022-08-26T00:00:00Z

Dedicated to everyone feeling stressed, down, lost, or under the weather.

As we all know, penguins are great swimmers. Sadly however they can't fly (or so we think!). But as the "Learn to Fly" game series teaches us, with enough dedication, penguins can indeed fly. And with each attempt they reach higher altitudes, even if simply done using a tiny booster rocket.

Hopefully this little piece may be a booster rocket for you.

Joyride in D

2022-01-20T00:00:00Z

This upbeat track is an offshoot of Morning Rush, spawned from a melody in the latter's interlude. Couldn't resist going ham on the drums.

Unlike Breath of Life and Morning Rush, this piece doesn't have any particular meaning imbued into it, just a fun little composition. I enjoyed writing the keyboard solo (although in its current state, it's not exactly playable), as well as the guitar countermelody at 2:50. Also writing for drums wasn't as boring as I thought.

This is the third of my three 2021 compositions, uploaded to mark the start of 2022. Enjoy.

Morning Rush

2022-01-19T00:00:00Z

Do you feel the exhilaration of the morning air? The excitement of stepping out of bed? Chomping down breakfast?

This upbeat track is an offshoot of The Breath of Life, rendering its minimalistic elements with a modern instrumentation and at a faster pace. In particular, it combines the 1564 chord progression with the 332 rhythmic pattern in the rhythm guitar.

This is the second of my three 2021 compositions, uploaded to mark the start of 2022. Enjoy.

The Breath of Life

2022-01-18T00:00:00Z

Slow. Growing. Contemplating. Lush. Carefree. Hopeful. Yearning. This piece seeks to paint a diverse set of feelings. Listen for the string ensemble easing in and out of the void, the solemn moments of silence, the lightheartedness of the syncopated pizzicato, the relentless cycle of tension and relief.

In some sections, I aimed for a more contemplative and reflective atmosphere. One key inspirations was Steve Reich's Music for 18 Musicians (an hour-long piece!) where Reich demonstrates the idea of "pulsing", where a note is repeated rapidly at regular intervals with an increase then decrease in dynamics. The overall effect is quite serene, making the soundscape ebb in and out.

To capture this atmosphere of serenity in The Breath of Life but maintain a slow and steady tempo, I decided for the lower strings (cello + double bass) to play long breves. At the same time, the music would "breathe in" and "breathe out" both dynamically and sometimes harmonically. As the lower strings play a third, the measure tenses up a bit. Examples of these are at 2:24 and 8:00.

This is the first of my three 2021 compositions, uploaded to mark the start of 2022. Enjoy.

Reticence

2021-08-21T00:00:00Z

Something about the past few years has been very poignant—deep, memorable, touching yet almost hurting; somewhat close yet also far; cold and distant yet warm inside.

Reticence was composed as a final project for an introductory course to music composition. Originally, I wanted the piece to be faster and more brisk (akin to a stressed state of mind). But after some reflection (and on further advice by instructors), I settled with a slower tempo. Following further modifications, the final result attempts to capture a held back, cold yet comforting, and ultimately poignant mood.

The piece was performed by Galison Lau and Ka Lap Wong in the final concert of the course. The SoundCloud upload is a software rendition.