Classic Vulnerability: Writing beyond buffer bounds can overwrite adjacent memory, including return addresses, allowing attackers to hijack control flow.
; Vulnerable code pattern
section .bss
buffer resb 64 ; 64-byte buffer
section .text
vulnerable_func:
; No bounds checking!
lea rdi, [buffer]
call gets ; NEVER use gets()
ret ; May jump to attacker-controlled address
Stack Smashing Attack
A stack smashing attack overwrites the return address to hijack control flow. Let's visualize exactly what happens:
Stack smashing attack: buffer overflow overwrites saved RBP and return address to hijack control flow
Stack Layout During Function Call:
High Memory
┌───────────────────┐
│ Return Address │ ← Attacker's target!
├───────────────────┤ Overwrite this to redirect execution
│ Saved RBP │
├───────────────────┤
│ Local Buffer[63] │
│ ... │ Buffer overflow direction ↑
│ Local Buffer[0] │ ← Input starts here, overflows upward
└───────────────────┘
Low Memory (RSP)
Overflow Attack:
1. Attacker inputs 64+ bytes via gets()
2. Input overflows buffer, overwrites saved RBP
3. Keeps writing, overwrites return address
4. When function returns, RIP = attacker-controlled!
; Exploitable function
vulnerable:
push rbp
mov rbp, rsp
sub rsp, 64 ; 64-byte buffer
lea rdi, [rbp-64] ; Buffer address
call gets ; Reads unlimited input!
leave ; mov rsp, rbp; pop rbp
ret ; pop rip - DANGER: may be overwritten!
; Attack payload (conceptual)
; [64 bytes padding][8 bytes fake RBP][8 bytes shellcode_addr]
Classic Attack Flow: Attacker sends 80 bytes: 64 bytes trash + 8 bytes fake RBP + 8 bytes pointing to shellcode. When ret executes, the CPU jumps to shellcode instead of the real caller.
Protection Mechanisms
Stack Canaries
; GCC -fstack-protector inserts canary
function_with_canary:
push rbp
mov rbp, rsp
sub rsp, 80
; Canary placed between buffer and saved RBP
mov rax, fs:[0x28] ; Load canary from TLS
mov [rbp-8], rax ; Store on stack
; ... function body ...
; Check canary before return
mov rax, [rbp-8]
xor rax, fs:[0x28]
jnz .stack_smash_detect ; Abort if modified
leave
ret
.stack_smash_detect:
call __stack_chk_fail
Address Space Layout Randomization (ASLR)
ASLR randomizes where code and data are loaded in memory each time a program runs. This breaks exploits that hardcode addresses.
ASLR in action: stack, heap, and library base addresses change with each program execution
# Observe ASLR in action
$ cat /proc/sys/kernel/randomize_va_space
2 # 0=off, 1=stack/mmap, 2=full (heap too)
# Run program twice, check stack address
$ ./test_aslr
Stack address: 0x7fff2a3b4c80
$ ./test_aslr
Stack address: 0x7fffde123a50 # Different each time!
# Check library base addresses
$ ldd /bin/ls | head -2
linux-vdso.so.1 => (0x00007ffd12345000) # Randomized
libc.so.6 => /lib/x86_64.../libc.so.6 (0x00007f8abc123000)
ASLR Level
Randomized Regions
Entropy (bits)
Level 1
Stack, shared libraries, mmap
~28 bits
Level 2
+ Heap (brk)
~28 bits
PIE Binary
+ Main executable
~28 bits
Bypassing ASLR: Attackers use information leaks (format string bugs, buffer over-reads) to discover actual addresses at runtime, then calculate offsets to known functions.
Data Execution Prevention (DEP/NX)
DEP (Windows) / NX bit (Linux) marks memory pages as non-executable. Even if an attacker gets shellcode into memory, the CPU refuses to execute it.
DEP/NX bit enforcement: W^X policy ensures memory pages are either writable or executable, never both
Page Table Entry NX Bit:
┌─────────────────────────────────────────────┐
Bit 63: NX (No eXecute)
0 = Page is executable (code)
1 = Page is NOT executable (data/stack)
└─────────────────────────────────────────────┘
W^X (Write XOR Execute) Policy:
- A page can be Writable OR Executable, never both
- Stack: RW- (read, write, no execute)
- Code: R-X (read, execute, no write)
- Heap: RW- (read, write, no execute)
# Check memory protections on a process
$ cat /proc/$(pidof victim)/maps
00400000-00401000 r-xp /victim # Code: executable
00600000-00601000 rw-p /victim # Data: writable, not exec
7ffffffde000-7ffffffff000 rw-p [stack] # Stack: not executable
# Attempt to execute shellcode on NX stack
$ ./shellcode_test
Segmentation fault (core dumped) # CPU refused!
Legacy Note: The NX bit requires CPU support (most CPUs since ~2004) and 64-bit mode (or PAE in 32-bit). Check with grep nx /proc/cpuinfo.
Return-Oriented Programming (ROP)
ROP bypasses NX by reusing existing executable code snippets called "gadgets." Instead of injecting shellcode, attackers chain together instruction sequences that end in ret.
A gadget is any sequence of instructions ending in RET:
; Example gadgets found in libc:
pop rdi; ret ← Set RDI (first argument)
pop rsi; ret ← Set RSI (second argument)
pop rdx; ret ← Set RDX (third argument)
mov [rdi], rax; ret ← Write memory
xchg eax, esp; ret ← Stack pivot
Building a ROP Chain
Goal: Call execve("/bin/sh", NULL, NULL)
Stack Layout (attacker-controlled):
┌───────────────────────┐
│ pop rdi; ret │ RSP → First gadget
├───────────────────────┤
│ addr of "/bin/sh" │ ← Popped into RDI
├───────────────────────┤
│ pop rsi; ret │ Next gadget
├───────────────────────┤
│ 0 (NULL) │ ← Popped into RSI (argv)
├───────────────────────┤
│ pop rdx; ret │ Next gadget
├───────────────────────┤
│ 0 (NULL) │ ← Popped into RDX (envp)
├───────────────────────┤
│ syscall; ret │ Execute syscall 59 (execve)
└───────────────────────┘
Execution Flow:
1. ret pops "pop rdi; ret" address into RIP
2. pop rdi loads "/bin/sh" address into RDI
3. ret pops next gadget address...
4. Eventually syscall executes execve("/bin/sh", NULL, NULL)
5. Attacker gets shell!
Finding Gadgets
# Use ROPgadget tool
$ ROPgadget --binary /lib/x86_64-linux-gnu/libc.so.6 | grep "pop rdi"
0x000000000002155f : pop rdi ; ret
0x00000000000215a0 : pop rdi ; pop rbp ; ret
# Or use ropper
$ ropper -f /lib/x86_64-linux-gnu/libc.so.6 --search "pop rdi"
[INFO] Searching for gadgets: pop rdi
0x000000000002155f: pop rdi; ret;
Defense: Control Flow Integrity (CFI) and Shadow Stacks detect and prevent ROP by validating return addresses match legitimate call sites.
Secure Assembly Practices
Writing secure assembly requires constant vigilance. Here are essential practices to protect your code:
Secure assembly practices: bounds checking, sensitive data zeroing, constant-time comparisons, and stack canaries
