iCTF 2023: RemoteC4

There are 52435435 combinations to defuse this C4. Can you make it in time?

TL;DR

SystemFunction033 is used to perform RC4 Encryption. RC4 Key is derived using the Middle-square method, and rotates during half-time.

Understanding Program Behaviour

Decompile RemoteC4.exe using Ghidra and locate the main function:

Immediately, we notice the use of SystemFunction033 Windows API exported from advapi32.dll:

A simple Google search shows that SystemFunction033 is used to perform RC4 encryption:

We know that SystemFunction033 takes in 2 arguments: data & key, and local_580 is the exported address of SystemFunction033. Renaming these variables in Ghidra will result in the following:

Locating Encrypted Data

There's also this bunch of data (presumably our encrypted flag) located in the stack:

Tracing User Input

Based on some recognizable terminal texts available, we can easily trace our user input:

There are 2 loops that iterates for 32 times. While this does not provide any useful information, we know that both loops are the same length as our input, and that the inner loop is associated with our input in some kind of way:

Deriving RC4 Key

Continuing from line 94, this is where the ustring struct from SystemFunction033 comes into play. For each ustring struct (data & key), we need to assign 2 values/attributes to it, which are DWORD Length and unsigned char* Buffer respectively.

With the information from above, we now know that data[0] is our data length (well, because 0x20 is 32 in decimal notation), and key[0] is the length of RC4 key:

On line 96, our initial key of 52435435 is multiplied by itself, resulting in a 16 character key = 2749474843639225, which make sense because key[0] is 0x10 (16 in decimal):

Diving Deeper

Right after SystemFunction033 is invoked, the result is copied from the data buffer to another array acStack1216 based on the loop counter:

This intermediate array is used at the end of execution for comparison with local_500, which is where all the bunch of encrypted data is located:

Continue tracing the usage of key by renaming variables. At the end of the outer loop, notice that our encryption key initial_key is updated with strtol(key_var4), and this only applies if the loop counter is 0xf (15 in decimal), which is the halfway point of our flag. This means that the RC4 key is rotated for the second half portion of the flag:

On line 107, 8 characters are copied from _Source to key_var4. After that, the 9th character of key_var4 at index local_30 = 8 is assigned with a null terminator \0 to denote end of character. From line 113 - 117, the initial_key variable is then overwritten with these 8 characters that will multiply by itself on the next loop interval to form our new encryption key.

While we have no idea what lStack740 is, we know that _Source is somewhat associated with key_var3, which is the length of our old encryption key.

Based on strncpy function alone, we can deduce that the program is trying to extract 8 characters from our 16-length old encryption key. This is known as the Middle-square method.

The 8 characters derived from our old encryption key 2749474843639225 will be 47484363.

We can now repeat all the steps above to calculate our 2nd encryption key. 47484363 multiply by itself = 2254764729515769