UMass CTF 2021 Postmortem

This was my favorite challenge.

Writeups:

• Janet v1.1 REPL Sandbox Bypass (by Aneesh Dogra)
• UMass CTF 2021 - replme [pwn]​ (by TheGoonies)
• UMassCTF'21 replme writeup (by Chai Yi Chen)

Older versions of the Janet programming language had a vulnerable NaN boxing implementation. An attacker capable of running arbitrary Janet code could write specific bytes to a float64 object, resulting in type confusion.

The challenge presented itself as an online REPL for Janet¹ that blacklisted functions dealing with the filesystem. The blacklisting was done by overwriting the functions with one that would error (i.e. (defn os/shell [&] (error ...). The intended solution was to use the NaN boxing vulnerability to create a cfunction object that pointed at the original implementation.

The reason there was a replme2 is that I didn't blacklist all of the relevant functions. At least one team realized that they could call (slurp "flag.txt"). This is something that playtesting by the rest of the team likely would have caught, but I'll save that for a later section. Sunday morning, I released a fixed version. Some teams solved it instantly, indicating that they had either found a different unintended solution, or that they had solved replme the intended way.

A Python bytecode crackme. The challenge was titled "babushka" because the program is designed like a matryoshka doll. There is one entry point to a long chain of functions, each one unpacking and calling the next in the chain.

Each of these functions does some check on the input. All of them, except for the fourteenth, have some check that involves a decoy flag, and some way of combining the results of the checks further down. They are combined such that only the output of one of the 500 functions is used.

The intended solution was to write a script to extract all of the functions, and look at the code for combining function outputs to see which mattered. You could then manually reverse engineer that function to get the flag.

The way the input is checked is about the same across all functions, so in theory, one could write a script to extract the value the input is checked against from all of the functions, and then try all 500 to see which one worked.

I programmatically generated and obfuscated the script that was given to teams. This challenge was fairly unremarkable, besides there being no writeups, and some people giving me flak because it didn't work on their version of Python.

Oh well. I had a lot of fun making it.

Writeups:

• chains (by Chris Greene)
• UMassCTF 2021 - Chains [Reversing]​ (by Scavenger Security)
• Chains (by Arijeet Mondal)

This was an "optimizeme" challenge, one where the flag is generated by an inefficient algorithm. To get the flag in a reasonable amount of time, one would need to either patch the algorithm or reimplement it.

Here, the characters of the flag were encoded as very large numbers, and the program would, for each character, generate the entire Collatz sequence to see where that number occurs. There were some pretty clever solutions, but the one I used when I was playtesting was to memoize the Collatz function.

Oh, and to make it less reasonable for someone to let the program run for the duration of the competition, I compiled for AARCH64. I apologize to the handful of people who were unable to get an ARM environment set up.²

This was my second favorite challenge, even though a lot went wrong with it.

This was a networked maze game written in Rust, where the server would spit out the flag after 500 mazes had been solved. I spent a lot of time on the packet format and the renderer.

Communications between the client and the server were encrypted. The algorithm was AES-128-CBC with the key STRINGS NOT HERE.³ I made things slightly easier by using OpenSSL for the AES implementation, and slightly harder by linking statically. If I'd linked dynamically, it would have been trivial to run the client with ltrace. Static linking should've helped somewhat, since the OpenSSL .a includes a ton of informative strings, but no one came close to a solution until I'd re-released the client binary with symbols.

And then, in the last 30 minutes or so of the competition, two people had scripts, but the server was crashing around 200 mazes in. I couldn't figure out the bug in that short period of time, so I hastily re-deployed with a server that would spit out the flag after 50 solves. It was stressful.

Small lesson: networked games, where someone has to reimplement the protocol, should be somewhat tolerant to errors. In this case, the server panicked and closed the connection if there was anything wrong with the header, which I'll admit is bullshit.

The idea of leaking memory through a visual medium is neat to me, so I made a shitty BMP utility that encodes text into the least-significant bits of the image data. Source code wasn't made available during the competition, as I find leaking memory blindly to be more interesting.

This seemed to make it a very difficult challenge. Only one team solved it, which surprised me. BMP is simple enough that there are only so many things one could have tried. Anyway, the intended solution was to overwrite the "data start" pointer in some BMP file so that the tool would leak the heap data containing the flag.

Writeups:

• War and Pieces (by legallybearded)
• War and Pieces (by ugotjelly)

This wasn't one I had originally planned. I was in the grocery store one day and saw a bag of toy army soldiers for $1.99. I bought them because that was one of my favorite things to play with when I was younger. My mother, upon seeing them, asked if I had gotten them for the CTF, which gave me the idea.

The pieces came in 6 poses and 2 colors. I encoded the flag as hexadecimal, and assigned each combination of pose/color/orientation to a hexadecimal nibble. In retrospect, the challenge would've been more interesting had I treated the flag as a natural number, and written it in base-12 using the piece alphabet.

The flag was supposed to be UMASS{lil_t0y_s0lj4s}, but I made a mistake when lining up the pieces: in the script I had made to help me line up the toys for the picture, I had accidentally used the same color for two different digits. Someone was kind enough to point it out, so we accepted the mangled flag (UMASS{lfl_t0v_s0lj4s}). This wasn't a great decision, though, since you wouldn't realistically be able to pull that out of list of guesses. I should have just done the picture over again.

I'm hoping that next time, I won't be working on challenges in the week before the competition. Here, I had only playtested the first few steps because I was so short on time, hence this mistake falling through the cracks.

Writeups:

• suckless2 writeup (by renia256)

I have little to say. Someone asked if I could recycle some of my challenges from last year, but I had already posted a writeup. I ended up modifying the challenge in a way that would render my exploit useless, and releasing that as a challenge. It was meant to be easy.

Writeups:

• UMassCTF'21: easteregg (by Łukasz Szymański)
• easteregg (by Th0m4sK)
• Easteregg UMASS CTF 21 (by Hacklabor)
• Engenharia reversa - UMass CTF'21 - 'Easteregg' Writeup (by zapzap kkkkkkkkkkkkkkkkkkkkk '-')

Like warandpieces, this was not a challenge I had originally planned to make. One of the problem sets for the introduction to computer systems class this semester involved reverse engineering an ELF binary. In an effort to appeal to students who actually attend UMass, I was asked to make an easy reverse engineering challenge in the same vain.

I took that class two years ago, and I still had the programs I'd written for it, so I took one of them (a text adventure game), added an "easter egg" that you would need to reverse engineer the executable to find, and uploaded it.

This was one of the most solved challenges.

Next year, no more compiled Haskell.

I spent a lot of time on a cool sine scroller with one of my favorite mod files playing in the background.

And then I made the mistake of writing the key check algorithm in Haskell. This is a reminder that there's a fine line between "possible" and "reasonable". No one solved this.

When I'm done with school in two weeks, I'll give this my all, and give it a proper write-up.

This might be seem obvious, but please humor me.

I made an effort to go through all of my challenges to see that they were reasonable, and I suspect this was the case for the rest of the team as well. It was better than nothing, but we could have done better in a couple of ways. I'll use three challenges as examples: 'replme' and 'warandpieces', challenges I had created, and 'pikcha', made by Steven.

The original "replme" was solvable by one function that wasn't blacklisted. I think a less-experienced player would have come across it, and we do have a handful of people on our team who are less familiar with the "pwn" category.

When playtesting "warandpieces", I only went through the first line or so. The error doesn't show up until much later. If I had either solved it in full, or had someone else playtest it, I would have caught the error.

"pikcha" included the answer in the session. LOL. I think if I had playtested it before we went live, I would have caught it.

These issues could have been caught if we playtested our peer's challenges.

This is what I have proposed to do next time: for every challenge, we choose two other people to playtest it. That way, there are three (if you include the challenge creator) pairs of eyes on the challenge. The only work on challenges happening in the last month should be fixing issues that come up during playtesting. Playtesters should also be tracking how much time they spend on each challenge.

Often, a CTF will have an easy "entry" challenge that doesn't require any solving. The flag might be given in the challenge description. In our case, it was hidden somewhere in the communication platform we used. As an idea, it looks good on paper, but actually implementing the challenge was one of the biggest mistakes we made. We had nearly a hundred people spamming some variation of "/flag" or "!flag" into the chat, and a handful of people opening support tickets because they thought the flag was supposed to be obtained by talking to us.

A few minutes in, someone made the following joke:

<Youssef>: UMASS{WowIReallySuck...}
<ralpmeTS>: !flag
<Youssef>: I actually hate myself...
<Youssef>: I found the flag.

Besides 'UMASS{flag}', which is what we had in the message describing the flag format, 'UMASS{WowIReallySuck…}' was the second most popular incorrect answer for this challenge.

It was a mess. No more challenges like this.⁴

When challenges went down, or were modified, announcements went through our communication platform. Not everyone saw those announcements, and some teams had an unfair advantage by getting both broken and fixed versions of the same challenge.

Next year, announcements will occur on the CTF platform as well. And hopefully, with our commitment to playtesting, we will not need to fix any challenges after going live, but we will ensure the old versions are still available if we do.

Some people seemed to still be looking for teams in the week leading up to the competition, so I put together a little form where people would enter their experience, timezone, etc.

It was a good idea, but I think only about one team came about from this, since a lot of people either put bad contact information or flaked.

Next year, the matchmaking form will be available at the time we announce the competition, and the matchmaking process will be partially automated.

—