Debugging

Finding text in a running process

What a debugger physically does to a process, and why hidden text is nearly always text in a structure you have not identified yet.

"Extracting hidden text" makes the task sound like cryptography. It almost never is. Text in a running program is sitting in plain view in a heap allocation, and the only reason you cannot find it is that you are searching for the wrong encoding, in the wrong region, or as a bare character sequence when it is actually a field inside a structure.

Getting good at this means understanding what a debugger is actually doing, which is worth knowing regardless of why you started.

Scope: this describes generic operating-system debugging facilities, the sort you use on software you wrote or are authorised to inspect. It is not about circumventing a specific product's protections, and nothing here is aimed at that.

A debugger is a process with special privileges over another process

There is no magic layer. A debugger is an ordinary program that has asked the kernel for a specific relationship with a target, and the kernel then grants it four capabilities:

  1. Suspend and resume the target's threads.
  2. Read and write the target's memory, across the address-space boundary that normally makes that impossible — ReadProcessMemory and WriteProcessMemory on Windows, process_vm_readv or /proc/<pid>/mem on Linux.
  3. Read and write registers for any thread, including the instruction pointer.
  4. Receive debug events — the target stops and the debugger is notified on exceptions, module loads, thread creation and exit.

Everything a debugger does is built from those four. Which is why the same techniques transfer between WinDbg, gdb, lldb and a language runtime's own debugger: the tools differ, the primitives do not.

Breakpoints are a patched byte

A software breakpoint is not a flag the CPU knows about. The debugger reads the byte at the target address, saves it, and writes 0xCC in its place — the one-byte int3 instruction on x86. When execution reaches it, the CPU raises a breakpoint exception, the kernel routes it to the attached debugger, and the target stops.

To continue, the debugger has to undo its own trick: restore the original byte, move the instruction pointer back one (it advanced past the int3), single-step the real instruction, then re-patch 0xCC and resume. All of that happens behind a "continue" command.

Three consequences fall out of this that explain otherwise baffling behaviour:

  • Breakpoints modify the target's code. Anything reading its own bytes sees the patch. This is why self-checking code and debuggers interact badly, and why a checksum over a code page changes the moment you set a breakpoint in it.
  • You cannot set one in read-only memory without the debugger first making the page writable, which it does silently.
  • They cost nothing when not hit. The patched instruction only exists at that address; the rest of the program runs at full speed. This is the opposite of the intuition that breakpoints slow the program down.

Hardware breakpoints are the other kind, and are much more useful than they get credit for. x86 provides four debug registers (DR0–DR3) plus control bits in DR7, and each can be set to fire on execution, on write, or on read/write of an address. They do not modify memory, they are limited to four at a time, and critically they can watch data. "Break when anything writes to this address" is the single most effective technique for the question "what set this field to garbage", and most developers never use it because the UI hides it.

Single-stepping uses a third mechanism: the trap flag in the flags register, which makes the CPU raise a debug exception after every instruction.

Symbols are why addresses have names

Without symbols a stack frame is a number. With them it is OrderService::Validate + 0x2c, with parameter names, local variable names, structure layouts and line numbers.

On Windows the symbol information lives in a separate PDB file rather than in the binary. That separation is the source of most symbol problems in practice:

  • Public versus private PDBs. A public PDB, of the kind vendors ship for operating system binaries, has function names and little else. A private one has locals, types and line numbers. Same file extension, very different experience.
  • They must match exactly. A PDB is bound to its binary by a GUID and an age stamp. Rebuild the binary and last week's PDB is silently useless — which is what a debugger means when it reports a symbol mismatch rather than simply loading it anyway.
  • Symbol servers solve distribution: a path of the form srv*<cache>*<url> downloads and caches PDBs on demand, keyed by that GUID.
  • Optimised builds lie a little. Inlining removes frames, and the compiler reorders code so line attribution jumps around. A release build with correct symbols still shows a stack that does not match the source structure, and this is expected rather than broken.

The practical rule that follows: archive the PDBs for every build you ship. A crash dump from production without matching symbols is a list of hexadecimal addresses, and there is no way to recover the mapping afterwards.

Searching a process for a string

Now the actual task. It has three parts, and only one of them is searching.

Get the encoding right. This is where most attempts fail immediately. Windows APIs and .NET use UTF-16 little-endian, so the string "Order" is stored as 4F 00 72 00 64 00 65 00 72 00 — an ASCII search for those five bytes finds nothing. Elsewhere, and in most modern file formats and network protocols, it is UTF-8. WinDbg exposes both forms as separate commands: the memory search takes an ASCII or a Unicode mode, and da and du dump ASCII and Unicode text respectively. In gdb the find command takes an explicit byte pattern, and in lldb it is memory find. Search for both encodings before concluding the string is absent.

Search the right regions. A process address space is mostly not interesting. Executable images, mapped files and reserved-but-uncommitted ranges dwarf the heap. Enumerate the regions first — !address in WinDbg, /proc/<pid>/maps on Linux — and restrict the search to committed, private, readable-writable memory. That is where allocated data lives. Searching the whole space instead is slow and returns thousands of hits from string tables in loaded DLLs.

Interpret the hits. You will get many. Most are the same literal in a resource section, a format string, or a stale copy in freed-but-not-decommitted memory. The address alone tells you very little.

The structure is the point

Here is the part that turns a fishing expedition into an investigation. Text in a program is almost never a bare run of characters. It is a field inside an object, and the object has a layout.

  • A C++ std::string or std::wstring in every mainstream implementation uses small-string optimisation: short contents are stored inline in the object itself, longer ones on the heap with the object holding a pointer, a size and a capacity. So the same logical string is in two completely different places depending on its length, which is why a search finds some of your strings and not others.
  • A COM BSTR stores a four-byte length before the address the pointer refers to, and is null-terminated as well, so it can be passed to code expecting either convention.
  • A .NET String is a managed object with a header and a type pointer ahead of the character data, and a length field. Finding the characters is easy; finding the object that owns them means stepping backwards past the header.
  • A Rust String or a Go string is a pointer plus a length with no terminator at all, so the bytes after the content are unrelated memory and a null-terminated dump will read past the end and print garbage.

Once you know which of these you are looking at, you stop searching and start walking. In WinDbg, dt applies a symbol-defined type to an address and prints the fields; the managed extension does the equivalent for CLR objects, including enumerating every string on the heap by type. In gdb, casting the address to the right type does the same job.

From an address to the code responsible

Finding the text is usually not the question. The question is which code produced it, or which code corrupted it.

The technique is the data breakpoint. Having located the address, set a hardware write watchpoint on it and continue. The next write stops the process with a full stack trace of whoever did it. This answers "what set this to null", "which component is overwriting my buffer" and "who is mutating this shared field after initialisation" directly, rather than by inference.

Its limitation is that the address must be stable. Heap addresses change between runs, and a garbage collector may move managed objects, so the watchpoint has to be set during the run you care about. For addresses that move, breaking on the allocation site or on the setter instead is the workaround.

The general lesson

Nothing in a process's own address space is hidden from a debugger with the right privileges. Encryption at rest and encryption in transit both end at the point where the program needs the plaintext to do anything with it, and at that point it is a buffer like any other.

What actually obstructs you is not concealment but ignorance of layout. The text is there. It is in an encoding you did not search for, in a region you excluded, at an offset inside a structure you have not identified, or it existed briefly and the allocation has been reused. Every one of those is a fact you can establish, and establishing them is what separates debugging from guessing.

If you want to get better at this quickly, use the facilities on your own software while it is working. Attach to a healthy process, find a string you know is there, walk back to the object that owns it, and set a watchpoint on it. Doing that once, deliberately, makes the next production crash dump a great deal less intimidating — and it pairs well with reading the stack trace properly when the dump arrives.