anti-debugging-techniques
SKILL: Anti-Debugging Techniques — Detection & Bypass Playbook
AI LOAD INSTRUCTION: Expert anti-debug techniques across Linux and Windows. Covers ptrace, PEB flags, NtQueryInformationProcess, timing attacks, signal-based detection, TLS callbacks, VEH tricks, and all corresponding bypass methods. Base models often miss the distinction between user-mode and kernel-mode detection and the correct patching strategy for each.
0. RELATED ROUTING
- code-obfuscation-deobfuscation when the binary also uses control flow flattening, VM protection, or string encryption
- vm-and-bytecode-reverse when the anti-debug sits inside a custom VM dispatcher
- symbolic-execution-tools when you want to symbolically skip anti-debug checks entirely
Advanced Reference
Also load ANTI_DEBUG_MATRIX.md when you need:
- Complete cross-reference matrix of technique × OS × detection method × bypass method
- Per-technique reliability ratings and false-positive notes
- Tool compatibility chart (GDB, x64dbg, WinDbg, Frida, ScyllaHide)
Quick bypass picks
| Detection Class | First Bypass | Backup |
|---|---|---|
| ptrace-based (Linux) | LD_PRELOAD hook ptrace() → return 0 |
Kernel module to hide tracer |
| PEB.BeingDebugged (Windows) | Patch PEB byte at fs:[0x30]+0x2 |
ScyllaHide auto-patch |
| Timing check (rdtsc) | Conditional BP after rdtsc, fix registers | Frida hook rdtsc return |
| IsDebuggerPresent | NOP the call / hook return 0 | x64dbg built-in hide |
| INT 2D / UD2 exception | Set VEH to handle gracefully | TitanHide driver |
1. LINUX ANTI-DEBUG TECHNIQUES
1.1 ptrace(PTRACE_TRACEME)
The classic self-attach: a process calls ptrace(PTRACE_TRACEME, 0, 0, 0). If a debugger is already attached, the call fails (returns -1).
if (ptrace(PTRACE_TRACEME, 0, 0, 0) == -1) {
exit(1); // debugger detected
}
Bypass methods:
| Method | How |
|---|---|
LD_PRELOAD shim |
Compile shared lib: long ptrace(int r, ...) { return 0; } and set LD_PRELOAD |
| Binary patch | NOP the ptrace call or patch return value check |
| GDB catch | catch syscall ptrace → modify $rax to 0 on return |
| Kernel module | Hook sys_ptrace to allow multiple tracers |
1.2 /proc/self/status — TracerPid
FILE *f = fopen("/proc/self/status", "r");
// parse TracerPid: if non-zero → debugger attached
Bypass: Mount a FUSE filesystem over /proc/self, or LD_PRELOAD hook fopen/fread to filter TracerPid to 0.
1.3 Timing Checks (rdtsc / clock_gettime)
Measures elapsed time between two points; debugger single-stepping causes noticeable delay.
rdtsc
mov ebx, eax ; save low 32 bits
; ... protected code ...
rdtsc
sub eax, ebx
cmp eax, 0x1000 ; threshold
ja debugger_detected
Bypass: Set hardware breakpoint after second rdtsc, modify eax to pass the comparison. Or use Frida to replace the timing function.
1.4 Signal-Based Detection (SIGTRAP)
volatile int caught = 0;
void handler(int sig) { caught = 1; }
signal(SIGTRAP, handler);
raise(SIGTRAP);
if (!caught) exit(1); // debugger swallowed the signal
When a debugger is attached, SIGTRAP is consumed by the debugger rather than delivered to the handler. Bypass: In GDB, use handle SIGTRAP nostop pass to forward the signal.
1.5 /proc/self/maps & LD_PRELOAD Detection
Checks for injected libraries or memory regions characteristic of debuggers/instrumentation.
FILE *f = fopen("/proc/self/maps", "r");
while (fgets(buf, sizeof(buf), f)) {
if (strstr(buf, "frida") || strstr(buf, "LD_PRELOAD"))
exit(1);
}
Bypass: Hook fopen("/proc/self/maps") to return a filtered version, or rename Frida's agent library.
1.6 Environment Variable Checks
Some protections check for LD_PRELOAD, LINES, COLUMNS (set by GDB's terminal), or debugger-specific env vars.
Bypass: Unset suspicious env vars before launch, or hook getenv().
2. WINDOWS ANTI-DEBUG TECHNIQUES
2.1 IsDebuggerPresent / CheckRemoteDebuggerPresent
if (IsDebuggerPresent()) ExitProcess(1);
BOOL debugged = FALSE;
CheckRemoteDebuggerPresent(GetCurrentProcess(), &debugged);
if (debugged) ExitProcess(1);
Bypass: Hook kernel32!IsDebuggerPresent to return 0, or patch PEB directly.
2.2 PEB Flags
| Field | Offset (x64) | Debugged Value | Normal Value |
|---|---|---|---|
BeingDebugged |
PEB+0x02 |
1 | 0 |
NtGlobalFlag |
PEB+0xBC |
0x70 (FLG_HEAP_*) |
0 |
ProcessHeap.Flags |
Heap+0x40 | 0x40000062 |
0x00000002 |
ProcessHeap.ForceFlags |
Heap+0x44 | 0x40000060 |
0 |
mov rax, gs:[0x60] ; PEB
movzx eax, byte [rax+0x02] ; BeingDebugged
test eax, eax
jnz debugger_detected
Bypass: Zero all four fields. ScyllaHide does this automatically.
2.3 NtQueryInformationProcess
| InfoClass | Value | Debugged Return |
|---|---|---|
ProcessDebugPort |
0x07 | Non-zero port |
ProcessDebugObjectHandle |
0x1E | Valid handle |
ProcessDebugFlags |
0x1F | 0 (inverted!) |
Bypass: Hook ntdll!NtQueryInformationProcess to return clean values per info class.
2.4 Hardware Breakpoint Detection
CONTEXT ctx;
ctx.ContextFlags = CONTEXT_DEBUG_REGISTERS;
GetThreadContext(GetCurrentThread(), &ctx);
if (ctx.Dr0 || ctx.Dr1 || ctx.Dr2 || ctx.Dr3)
ExitProcess(1);
Bypass: Hook GetThreadContext to zero DR0–DR3, or use NtSetInformationThread(ThreadHideFromDebugger) preemptively (ironically, the anti-debug technique itself).
2.5 INT 2D / INT 3 / UD2 Exception Tricks
INT 2D is the kernel debug service interrupt. Without a debugger, it raises STATUS_BREAKPOINT; with a debugger, behavior differs (byte skipping).
xor eax, eax
int 2dh
nop ; debugger may skip this byte
; ... divergent execution path ...
Bypass: Handle in VEH or patch the interrupt instruction.
2.6 TLS Callbacks
TLS callbacks execute before main() / WinMain(). Anti-debug checks placed here run before the debugger's initial break.
Bypass: In x64dbg, set "Break on TLS Callbacks" option. In WinDbg, use sxe ld to break on module load.
2.7 NtSetInformationThread(ThreadHideFromDebugger)
NtSetInformationThread(GetCurrentThread(), ThreadHideFromDebugger, NULL, 0);
After this call, the thread becomes invisible to the debugger — breakpoints and single-stepping stop working silently.
Bypass: Hook NtSetInformationThread to NOP when ThreadInfoClass == 0x11.
2.8 VEH-Based Detection
Registers a Vectored Exception Handler that checks EXCEPTION_RECORD for debugger-specific behavior (single-step flag, guard page violations with debugger semantics).
Bypass: Understand the VEH logic and ensure the exception chain behaves identically to non-debugged execution.
3. ADVANCED MULTI-LAYER TECHNIQUES
3.1 Self-Debugging (fork + ptrace)
The process forks a child that attaches to the parent via ptrace. If an external debugger is already attached, the child's ptrace fails.
pid_t child = fork();
if (child == 0) {
if (ptrace(PTRACE_ATTACH, getppid(), 0, 0) == -1)
kill(getppid(), SIGKILL);
else
ptrace(PTRACE_DETACH, getppid(), 0, 0);
_exit(0);
}
wait(NULL);
Bypass: Patch the fork() return or kill/detach the watchdog child.
3.2 Multi-Process Debugging Detection
Parent and child cooperatively check each other's debug state, creating a mutual-watch pattern.
Bypass: Attach to both processes (GDB follow-fork-mode, or two debugger instances).
3.3 Timing-Based with Multiple Checkpoints
Distributes timing checks across multiple functions, comparing cumulative drift. Single patches fail because the total still exceeds threshold.
Bypass: Frida Interceptor.replace all timing sources (rdtsc, clock_gettime, QueryPerformanceCounter) to return controlled values.
3.4 Nanomite / INT3 Patching
Original conditional jumps are replaced with INT3 (0xCC). A parent debugger process handles each INT3, evaluates the condition, and sets the child's EIP accordingly.
Bypass: Reconstruct the original jump table by tracing all INT3 handlers, then patch the binary.
4. COUNTERMEASURE TOOLS
| Tool | Platform | Capability |
|---|---|---|
| ScyllaHide | Windows (x64dbg/IDA/OllyDbg) | Auto-patches PEB, hooks NtQuery*, hides threads, fixes timing |
| TitanHide | Windows (kernel driver) | Kernel-level hiding for all user-mode checks |
| Frida | Cross-platform | Script-based hooking of any function, timing spoofing |
| LD_PRELOAD shims | Linux | Replace ptrace, getenv, fopen at load time |
| GDB scripts | Linux | catch syscall, conditional BP, register fixup |
| Qiling | Cross-platform | Full-system emulation, bypass all hardware checks |
5. SYSTEMATIC BYPASS METHODOLOGY
Step 1: Static analysis — identify anti-debug calls
└─ Search for: ptrace, IsDebuggerPresent, NtQuery, rdtsc,
GetTickCount, SIGTRAP, INT 2D, TLS directory entries
Step 2: Classify each check
├─ API-based → hook or patch the call
├─ Flag-based → patch PEB/proc fields
├─ Timing-based → spoof time source
├─ Exception-based → forward/handle exception correctly
└─ Multi-process → handle both processes
Step 3: Apply bypass (order matters)
1. Load ScyllaHide / set LD_PRELOAD (covers 80% of checks)
2. Handle TLS callbacks (break before main)
3. Patch remaining custom checks (Frida or binary patch)
4. Verify: run with breakpoints, confirm no premature exit
Step 4: Validate bypass completeness
└─ Set BP on ExitProcess/exit/_exit — if hit unexpectedly,
a check was missed → trace back from exit call
6. DECISION TREE
Binary exits/crashes under debugger?
│
├─ Crashes immediately before main?
│ └─ TLS callback anti-debug
│ └─ Enable TLS callback breaking in debugger
│
├─ Crashes at startup?
│ ├─ Linux: check for ptrace(TRACEME)
│ │ └─ LD_PRELOAD hook or NOP patch
│ └─ Windows: check IsDebuggerPresent / PEB
│ └─ ScyllaHide or manual PEB patch
│
├─ Crashes after some execution?
│ ├─ Consistent crash point → API-based check
│ │ ├─ NtQueryInformationProcess → hook return values
│ │ ├─ /proc/self/status → filter TracerPid
│ │ └─ Hardware BP detection → hook GetThreadContext
│ │
│ ├─ Variable crash point → timing-based check
│ │ └─ Hook rdtsc / QueryPerformanceCounter
│ │
│ └─ Crash on breakpoint hit → exception-based check
│ ├─ INT 2D / INT 3 trick → handle in VEH
│ └─ SIGTRAP handler → GDB: handle SIGTRAP pass
│
├─ Debugger loses control silently?
│ └─ ThreadHideFromDebugger
│ └─ Hook NtSetInformationThread
│
├─ Child process detects and kills parent?
│ └─ Self-debugging (fork+ptrace)
│ └─ Patch fork() or handle both processes
│
└─ All basic bypasses applied but still detected?
└─ Multi-layer / custom checks
├─ Use Frida for comprehensive API hooking
├─ Full emulation with Qiling
└─ Trace all calls to exit/abort to find remaining checks
7. CTF & REAL-WORLD PATTERNS
Common CTF Anti-Debug Patterns
| Pattern | Frequency | Quick Bypass |
|---|---|---|
Single ptrace(TRACEME) |
Very common | LD_PRELOAD one-liner |
IsDebuggerPresent + NtGlobalFlag |
Common | ScyllaHide |
| rdtsc timing in loop | Moderate | Patch comparison threshold |
| signal(SIGTRAP) + raise | Moderate | GDB signal forwarding |
| fork + ptrace watchdog | Rare but tricky | Kill child or patch fork |
| Nanomite INT3 replacement | Rare (advanced) | Reconstruct jump table |
Real-World Protections
| Protector | Primary Anti-Debug | Recommended Tool |
|---|---|---|
| VMProtect | PEB + timing + driver-level | TitanHide + ScyllaHide |
| Themida | Multi-layer PEB + SEH + timing | ScyllaHide + manual patches |
| Enigma Protector | IsDebuggerPresent + CRC checks | x64dbg + ScyllaHide |
| UPX (custom) | Usually none (just packing) | Standard unpack |
| Custom (malware) | Varies widely | Frida + Qiling for analysis |
8. QUICK REFERENCE — BYPASS CHEAT SHEET
Linux One-Liners
# LD_PRELOAD anti-ptrace
echo 'long ptrace(int r, ...) { return 0; }' > /tmp/ap.c
gcc -shared -o /tmp/ap.so /tmp/ap.c
LD_PRELOAD=/tmp/ap.so ./target
# GDB: catch and bypass ptrace
(gdb) catch syscall ptrace
(gdb) commands
> set $rax = 0
> continue
> end
Frida Anti-Debug Bypass (Cross-Platform)
// Hook IsDebuggerPresent (Windows)
Interceptor.replace(
Module.getExportByName('kernel32.dll', 'IsDebuggerPresent'),
new NativeCallback(() => 0, 'int', [])
);
// Hook ptrace (Linux)
Interceptor.replace(
Module.getExportByName(null, 'ptrace'),
new NativeCallback(() => 0, 'long', ['int', 'int', 'pointer', 'pointer'])
);
// Timing spoof
Interceptor.attach(Module.getExportByName(null, 'clock_gettime'), {
onLeave(retval) {
// manipulate timespec to hide debugger delay
}
});
x64dbg ScyllaHide Quick Setup
- Plugins → ScyllaHide → Options
- Check: PEB BeingDebugged, NtGlobalFlag, HeapFlags
- Check: NtQueryInformationProcess (all classes)
- Check: NtSetInformationThread (HideFromDebugger)
- Check: GetTickCount, QueryPerformanceCounter
- Apply → restart debugging session