Phase 11: 64-Bit Long Mode

CPU Detection

Before switching to 64-bit mode, we must verify the CPU supports it. Not all x86 processors have long mode—older 32-bit-only CPUs don't. The CPUID instruction tells us what features are available.

CPUID Instruction

CPUID is x86's feature discovery mechanism. You put a "leaf" number in EAX, execute CPUID, and get feature flags back in EAX/EBX/ECX/EDX:

Checking Support

Here's the complete sequence to verify long mode is available:

; Check for long mode support
check_long_mode:
    ; Check if CPUID is supported
    pushfd
    pop eax
    mov ecx, eax
    xor eax, 1 << 21    ; Flip ID bit
    push eax
    popfd
    pushfd
    pop eax
    push ecx
    popfd
    cmp eax, ecx
    je .no_cpuid
    
    ; Check for extended CPUID
    mov eax, 0x80000000
    cpuid
    cmp eax, 0x80000001
    jb .no_long_mode
    
    ; Check for long mode
    mov eax, 0x80000001
    cpuid
    test edx, 1 << 29   ; Long mode bit
    jz .no_long_mode
    
    mov eax, 1          ; Long mode supported
    ret
    
.no_cpuid:
.no_long_mode:
    xor eax, eax        ; Long mode not supported
    ret

Detection steps explained:

1. Check CPUID support - Toggle EFLAGS.ID bit. If it toggles, CPUID exists
2. Check extended leaves - Call CPUID with EAX=0x80000000 to see if 0x80000001 is supported
3. Check long mode bit - Call CPUID with EAX=0x80000001, check EDX bit 29

64-Bit GDT

Long mode still requires a GDT, but it's much simpler. Most segment fields are ignored—the CPU uses a flat memory model. We only need to set a few specific bits.

GDT Structure

64-Bit GDT Entry (8 bytes):

Bit:   63                                                    0
       ┌────┬───┬───┬───┬────┬─────────────────────────────────┐
       │ G  │ D │ L │AVL│Seg │         (Ignored in 64-bit)     │
       │    │ B │   │   │Lim │                                 │
       ├────┼───┼───┼───┼────┼───┬───┬───┬───┬────────────────┤
Byte:  │  7 │   │   │   │  6 │ P │DPL│ S │Type│       5-0      │
       └────┴───┴───┴───┴────┴───┴───┴───┴───┴────────────────┘

For 64-bit code segment:
  L  = 1  (Long mode)
  D  = 0  (Must be 0 when L=1)
  P  = 1  (Present)
  S  = 1  (Code/data, not system)
Type = 0xA (Execute/Read)

Result: The code segment is just a few bits set!

; 64-bit GDT
section .rodata
gdt64:
    dq 0                            ; Null descriptor
.code: equ $ - gdt64
    dq (1 << 43) | (1 << 44) | (1 << 47) | (1 << 53)  ; Code segment
.data: equ $ - gdt64
    dq (1 << 44) | (1 << 47)        ; Data segment
.pointer:
    dw $ - gdt64 - 1                ; GDT limit
    dq gdt64                        ; GDT base

Breaking down the magic numbers:

• (1 << 43) - Executable bit (this is a code segment)
• (1 << 44) - Descriptor type: code/data (not system)
• (1 << 47) - Present bit
• (1 << 53) - Long mode bit (L flag)

Long Mode Segments

In 64-bit mode, segmentation is mostly disabled:

4-Level Paging

Long mode requires paging—you can't run without it. And it uses 4-level paging (instead of 32-bit's 2-level) to address the huge 48-bit virtual address space.

PML4 Table

4-Level Page Table Structure:

Virtual Address (48 bits used):
┌────────┬────────┬────────┬────────┬────────────────────┐
│        │  PML4  │  PDPT  │   PD   │   PT   │  Offset   │
│ Sign   │  Index │  Index │  Index │  Index │           │
│ Extend │   9b   │   9b   │   9b   │   9b   │    12b    │
│ 16 bits│        │        │        │        │           │
└────────┴────────┴────────┴────────┴────────┴───────────┘
  63-48     47-39    38-30    29-21    20-12      11-0

Page Table Walk (for 4KB pages):

    CR3 ────► ┌──────────┐
              │   PML4   │ (512 entries × 8 bytes = 4KB)
              │   Table  │
              └────┬─────┘
                   │ [PML4 index]
                   ▼
              ┌──────────┐
              │   PDPT   │ (Page Directory Pointer Table)
              │  Table   │
              └────┬─────┘
                   │ [PDPT index]
                   ▼
              ┌──────────┐
              │    PD    │ (Page Directory)
              │   Table  │
              └────┬─────┘
                   │ [PD index]
                   ▼
              ┌──────────┐
              │   Page   │ (Page Table)
              │   Table  │
              └────┬─────┘
                   │ [PT index]
                   ▼
              ┌──────────┐
              │ Physical │ + Offset (12 bits)
              │   Page   │
              └──────────┘

The code below sets up minimal paging for the first 1GB using 2MB huge pages (skips the PT level):

; Set up 4-level paging for long mode
section .bss
align 4096
p4_table:
    resb 4096
p3_table:
    resb 4096
p2_table:
    resb 4096

section .text
setup_page_tables:
    ; Map P4 -> P3
    mov eax, p3_table
    or eax, 0b11        ; Present + Writable
    mov [p4_table], eax
    
    ; Map P3 -> P2
    mov eax, p2_table
    or eax, 0b11
    mov [p3_table], eax
    
    ; Map P2 entries (2MB pages)
    mov ecx, 0          ; Counter
.map_p2:
    mov eax, 0x200000   ; 2MB
    mul ecx
    or eax, 0b10000011  ; Present + Writable + Huge
    mov [p2_table + ecx * 8], eax
    inc ecx
    cmp ecx, 512        ; Map 1GB
    jne .map_p2
    
    ret

Table Hierarchy

Each level of the page table hierarchy covers progressively smaller regions:

; 2MB huge page entry
mov eax, 0x200000       ; Physical address
or eax, 0b10000011      ; Present + Writable + Huge
mov [p2_table + ecx*8], eax

Identity Mapping

During the transition to long mode, we need identity mapping—where virtual address equals physical address. Why? Because when we enable paging, the instruction pointer suddenly becomes a virtual address. If that virtual address isn't mapped to where the code actually is, we crash.

Identity Mapping for Boot:

Physical:  0x00000000 ─────────────────────► 0x40000000 (1GB)
              │                                   │
              │        1:1 Mapping                │
              │                                   │
Virtual:   0x00000000 ─────────────────────► 0x40000000 (1GB)

After boot, you might want a higher-half kernel:

Physical:     0x00000000 ──────────────────► 0x40000000
                   │
                   │ Mapped to TWO places:
                   │
Virtual:   0x00000000 (identity, temporary)
        0xFFFF800000000000 (higher-half kernel)

Mode Transition

The transition from 32-bit protected mode to 64-bit long mode requires a specific sequence. You can't just flip a switch—several CPU features must be enabled in order.

Transition Sequence:

┌─────────────────────────────────────────────────────────────────┐
│  32-Bit Protected Mode                                          │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ 1. Disable paging (if enabled)        CR0.PG = 0         │  │
│  │ 2. Load PML4 address into CR3         CR3 = p4_table     │  │
│  │ 3. Enable PAE                         CR4.PAE = 1        │  │
│  │ 4. Enable Long Mode in EFER           EFER.LME = 1       │  │
│  │ 5. Enable paging                      CR0.PG = 1         │  │
│  └───────────────────────────────────────────────────────────┘  │
│                         │                                       │
│                         ▼                                       │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ 6. Far jump to 64-bit code segment    jmp gdt64.code:...  │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                                                                   
┌─────────────────────────────────────────────────────────────────┐
│  64-Bit Long Mode                                               │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ 7. Reload segment registers with 64-bit selectors         │  │
│  │ 8. Set up 64-bit stack pointer                            │  │
│  │ 9. Call 64-bit kernel main                                │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Enable PAE

PAE (Physical Address Extension) is required for long mode. It changes the page table format from 32-bit entries to 64-bit entries, which is the basis for 4-level paging:

; Enable PAE (required for long mode)
mov eax, cr4
or eax, 1 << 5      ; Set PAE bit (bit 5)
mov cr4, eax

Enable Long Mode

Long mode is enabled through the EFER (Extended Feature Enable Register) MSR. This is a model-specific register accessed with RDMSR/WRMSR:

; Transition to long mode
enable_long_mode:
    ; Disable paging first
    mov eax, cr0
    and eax, ~(1 << 31)
    mov cr0, eax
    
    ; Load P4 table address
    mov eax, p4_table
    mov cr3, eax
    
    ; Enable PAE
    mov eax, cr4
    or eax, 1 << 5      ; PAE bit
    mov cr4, eax
    
    ; Enable long mode in EFER MSR
    mov ecx, 0xC0000080 ; EFER MSR
    rdmsr
    or eax, 1 << 8      ; LM bit
    wrmsr
    
    ; Enable paging
    mov eax, cr0
    or eax, 1 << 31     ; PG bit
    mov cr0, eax
    
    ret

Jump to 64-Bit

After enabling paging with long mode active, we're technically in "compatibility mode" until we load a 64-bit code segment. A far jump loads the new GDT and switches to true 64-bit mode:

; Jump to 64-bit code
    lgdt [gdt64.pointer]
    jmp gdt64.code:long_mode_start

[BITS 64]
long_mode_start:
    ; Reload segment registers
    mov ax, gdt64.data
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax
    
    ; Set up 64-bit stack
    mov rsp, stack_top
    
    ; Clear screen and print success
    mov rax, 0x2F4B2F4F2F20
    mov qword [0xB8000], rax
    
    ; Call 64-bit kernel main
    extern kernel_main_64
    call kernel_main_64
    
    ; Halt
    hlt

64-Bit Kernel Code

With 64-bit mode active, you'll write kernel code in 64-bit C. But there are important differences from 32-bit code.

Calling Convention

The System V AMD64 ABI (used on Linux, BSD, macOS) passes arguments in registers, not on the stack:

System V AMD64 Calling Convention:

Arguments (left to right):
  Integer/Pointer:  RDI, RSI, RDX, RCX, R8, R9
  Floating-point:   XMM0-XMM7
  Additional args:  On stack (right-to-left)

Return value:
  Integer/Pointer:  RAX (and RDX for 128-bit)
  Floating-point:   XMM0

Caller-saved (volatile):    RAX, RCX, RDX, RSI, RDI, R8-R11
Callee-saved (preserved):   RBX, RBP, R12-R15

Example: my_func(a, b, c, d, e, f, g)
                │  │  │  │  │  │  └── On stack
                │  │  │  │  │  └───── R9
                │  │  │  │  └──────── R8
                │  │  │  └─────────── RCX
                │  │  └──────────────  RDX
                │  └─────────────────  RSI
                └────────────────────  RDI

/* 64-bit kernel entry point */
// System V AMD64 ABI calling convention:
// Arguments: RDI, RSI, RDX, RCX, R8, R9
// Return: RAX
// Caller-saved: RAX, RCX, RDX, RSI, RDI, R8, R9, R10, R11
// Callee-saved: RBX, RBP, R12, R13, R14, R15

void kernel_main_64(void) {
    // Initialize 64-bit IDT
    idt64_init();
    
    // Initialize 64-bit memory manager
    pmm64_init();
    vmm64_init();
    
    // Initialize interrupts
    pic_init();
    
    // Print welcome message
    terminal_write("Welcome to 64-bit mode!\n");
    
    // Main kernel loop
    while (1) {
        __asm__("hlt");
    }
}

Porting Considerations

When converting 32-bit kernel code to 64-bit, watch for these common issues:

/* Portable types for 64-bit */
#include 

// Page table entry - always 64-bit in long mode
typedef uint64_t pte_t;

// Physical addresses can be 52-bit
typedef uint64_t phys_addr_t;

// Virtual addresses are 64-bit (48 used)
typedef uintptr_t virt_addr_t;

// Use size_t for sizes and counts
size_t num_pages = mem_size / PAGE_SIZE;

/* 64-bit IDT entry structure */
typedef struct {
    uint16_t offset_low;      // Bits 0-15
    uint16_t selector;        // Code segment selector
    uint8_t  ist;             // Interrupt Stack Table
    uint8_t  type_attr;       // Type and attributes
    uint16_t offset_mid;      // Bits 16-31
    uint32_t offset_high;     // Bits 32-63  ← NEW!
    uint32_t reserved;        // Must be 0    ← NEW!
} __attribute__((packed)) idt64_entry_t;

What You Can Build

Complete 64-Bit Boot Sequence

; boot64.asm - Complete boot to 64-bit mode
[BITS 32]
section .text
global _start

_start:
    ; We're in 32-bit protected mode from bootloader
    
    ; 1. Check long mode support
    call check_long_mode
    test eax, eax
    jz .no_long_mode
    
    ; 2. Set up 4-level page tables
    call setup_page_tables
    
    ; 3. Enable PAE
    mov eax, cr4
    or eax, 1 << 5
    mov cr4, eax
    
    ; 4. Load PML4 into CR3
    mov eax, p4_table
    mov cr3, eax
    
    ; 5. Enable Long Mode in EFER
    mov ecx, 0xC0000080
    rdmsr
    or eax, 1 << 8
    wrmsr
    
    ; 6. Enable paging (enters compatibility mode)
    mov eax, cr0
    or eax, 1 << 31
    mov cr0, eax
    
    ; 7. Load 64-bit GDT and far jump
    lgdt [gdt64.pointer]
    jmp gdt64.code:long_mode_entry

.no_long_mode:
    ; Print error and halt
    mov dword [0xB8000], 0x4F524F45  ; "ER"
    mov dword [0xB8004], 0x4F3A4F52  ; "R:"
    hlt

[BITS 64]
long_mode_entry:
    ; Reload segment registers
    mov ax, gdt64.data
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax
    
    ; Set up stack
    mov rsp, stack_top
    
    ; Call C kernel
    extern kernel_main
    call kernel_main
    
    ; Halt
    cli
    hlt

section .bss
align 16
stack_bottom:
    resb 16384
stack_top:

Exercises

; Map same physical memory to high address
mov rax, p3_table
or rax, 0b11
mov [p4_table + 256 * 8], rax  ; PML4[256] = high half

// Set NX bit on data page entry
pte |= (1ULL << 63);  // No execute

// Setup SYSCALL
wrmsr(MSR_STAR, ((uint64_t)KERNEL_CS << 32) | 
                ((uint64_t)USER_CS << 48));
wrmsr(MSR_LSTAR, (uint64_t)syscall_handler);
wrmsr(MSR_FMASK, 0x200);  // Clear IF

Next Steps

Congratulations! Your kernel now runs in 64-bit mode with access to modern CPU features. But we're still booting via legacy BIOS. In Phase 12, we'll modernize the boot process with UEFI.

BIOS vs UEFI Boot:

Legacy BIOS (Current):           UEFI (Next Phase):
┌─────────────────────┐          ┌─────────────────────────┐
│  16-bit Real Mode   │          │  64-bit from start!     │
│  512-byte bootloader│          │  Full FAT32 filesystem  │
│  MBR partitioning   │          │  GPT partitioning       │
│  No graphics        │          │  GOP framebuffer        │
│  Manual A20 gate    │          │  Modern memory map      │
└─────────────────────┘          └─────────────────────────┘

Key Takeaways

• CPUID before transition - Always verify long mode support before attempting switch
• PAE is mandatory - Long mode requires PAE-style page tables
• 4-level paging - PML4 → PDPT → PD → PT for 48-bit addresses
• Identity mapping during switch - EIP must be valid as virtual address
• Far jump activates 64-bit - Loading CS with 64-bit segment completes transition
• Register-based calling - Arguments in RDI, RSI, RDX, RCX, R8, R9