Anti-Analysis Tricks

You open a binary in Ghidra and there is almost nothing there. A tiny bit of code, a huge blob of high-entropy data, an import table with three functions instead of forty. The program is packed: the real code is compressed or encrypted on disk and only gets unpacked into memory when it runs. Packing is the first wall authors put up to slow you down, and malware uses it constantly to dodge both analysts and antivirus signatures.

Spot it before you waste time. High entropy across a section is the tell, because compressed and encrypted data looks like noise while normal code has structure. A stub that unpacks then jumps to the freshly written region is the other giveaway.

Unpack the easy ones, dump the hard ones

UPX is the most common packer and it unpacks itself. upx -d packed.exe reverses it cleanly, and a shocking amount of real-world packed malware is just UPX that the author never bothered to modify. Try it first, always.

When the packer is custom or commercial, you do not reverse the packer. You let the program unpack itself, then grab the result from memory. Run it under a debugger, set a breakpoint at the moment the unpacking stub finishes and jumps to the original entry point, and dump the process memory to disk. The unpacked code is sitting right there once the stub has done its work. Tools like Scylla rebuild the import table on the dumped image so the result actually runs and disassembles.

The trick is finding the jump to the real code. The stub writes decompressed bytes into a region, then transfers control with a jmp to an address that was empty at load time. Break on writes to that region or on the tail jump, and the instruction right after it is the original entry point.

Anti-debug, and getting past it

Packing hides the code. Anti-debug stops you from watching it run. On Linux the classic is ptrace: a process can only be traced once, so the program calls ptrace(PTRACE_TRACEME, 0, 0, 0) on itself. If that call fails, a debugger already has it, and the program exits or lies about the result.

$catch syscall ptrace

Set the return to 0 and the check passes. You can also patch the call out entirely with the NOP technique from the patching lesson, or preload a library that stubs ptrace to always return success. Any of the three works; the register override is fastest when you just want to get past it once.

Timing checks are the other common family. The program reads the clock with rdtsc or clock_gettime before and after a block, and if too much wall time passed it assumes you were single-stepping and bails. You defeat these by not single-stepping through the timed region: set a breakpoint after it and let it run at full speed, so the delta stays small. If the check is inline, patch the comparison so the "too slow" branch never fires.

The mindset that saves you: every anti-analysis trick is a check, and every check is code you can find and neutralize. The packer has to unpack to run, so the real code is always reachable in memory. The anti-debug call has to return a value, so you can force that value. Nothing here is magic. It is one more branch to flip, and you already know how to flip branches.