ARM Assembly Part 18: Performance Profiling & Micro-Optimization

PMCR_EL0 — control: E=enable all, C=cycle reset, P=counter reset, N=num counters, LC=64-bit cycles
PMCNTENSET_EL0 — counter enable set: bit 31 = cycle counter, bits 0–30 = event counters
PMCNTENCLR_EL0 — counter enable clear (complement of above)
PMCCNTR_EL0 — 64-bit CPU cycle counter
PMEVCNTR<n>_EL0 — event counter n (n = 0 to PMCR_EL0.N-1)
PMEVTYPER<n>_EL0 — event type for counter n (lower 16 bits = event code)
PMUSERENR_EL0 — user-mode enable: EN=1 allows EL0 reads of all PMU registers
PMINTENSET_EL1 — PMU interrupt enable (for overflow interrupts)

RAW (Read After Write) — Data dependency hazard
Most common. Instruction N writes a register that instruction N+1 reads. On in-order pipelines this stalls until the result is available. On OOO cores the scheduler may find other independent instructions to fill the slots.
Fix: Reorder instructions to introduce independent work between producer and consumer. Use LDP/STP to hide load latency.

WAR (Write After Read) — Anti-dependency
N reads a register that N-1 writes. In OOO with register renaming this is transparent. In explicitly scheduled code, can surface as stalls on in-order micro-ops.
Fix: Use a different destination register.

WAW (Write After Write) — Output dependency
Two instructions write the same register; the second must follow the first. Register renaming eliminates this in hardware.
Fix: Use different destination registers when possible.

Structural hazard
Two instructions need the same execution unit in the same cycle (e.g., two FP divides; only one FDIV unit). Compiler/assembler scheduling must respect issue slots.
Fix: Interleave non-competing instructions (e.g., integer ops between FP divides).

Control hazard (branch misprediction)
Speculative execution fetches the wrong path; misprediction flushes the OOO window and refetches. Cortex-A72 mispredict penalty ≈ 15 cycles; Neoverse N1 ≈ 10 cycles.
Fix: CSEL/CSINC for short conditionals. Restructure loops to be loop-carried-dependency–free. Sort data to increase branch predictability.

Unrolling reduces loop overhead (SUBS+B.NE = 2 instructions per element) and exposes more independent instructions to the OOO scheduler. Unroll factor of 4–8 typically optimal; beyond that, register pressure limits gains.

PRFM PLDL1KEEP, [x0, #256] — hint to hardware prefetcher to load cache line at address (x0+256) into L1-D. Use when the hardware stride prefetcher can't detect the pattern (e.g., pointer-chasing). Prefetch distance = latency × bandwidth. Over-prefetching pollutes the cache.

Replace short if/else with CSEL Xd, Xn, Xm, <cond>. No branch = no misprediction = no 10–15 cycle flush. Works when both values are already computed (no side effects in the branch body). The compiler does this automatically with -O2 for simple ternaries; inline asm for complex cases.

Overlap iteration N's computation with iteration N+1's load. Classic pattern: LDR Xn, [ptr, #STRIDE] at the bottom of iteration N starts the load early; the data is consumed at the top of iteration N+1, hiding the L1D load latency (4 cycles on Cortex-A55). Requires one loop prolog load and one epilog.

Profiling a Redis-like Cache on Neoverse V1

When a cloud provider migrated their Redis-compatible in-memory cache from Intel Xeon (Ice Lake) to AWS Graviton3 (Neoverse V1), initial benchmarks showed 15% slower throughput on ARM — surprising given Graviton3's competitive IPC. The profiling investigation uncovered three issues:

1. Branch misprediction spike: perf stat -e branch-misses showed 8.3% miss rate on the hash table lookup path (vs 2.1% on Intel). Root cause: the hash function used CRC32C — Intel had a hardware CRC instruction, but the ARM build fell back to a software polynomial table with unpredictable branches. Fix: switched to __crc32cd() ARM CRC intrinsic (ARMv8.1-A), reducing branch misses to 1.8%.
2. L1D cache miss on atomic operations: PMU counter L1D_CACHE_REFILL was 3× higher than expected. The LDXR/STXR atomic increment in the per-shard counter was causing excessive cacheline bouncing across Graviton3's 64 cores (each core has its own L1). Fix: switched to per-CPU counters with periodic aggregation, reducing L1D refills by 60%.
3. Memory bandwidth ceiling: perf stat -e armv8_pmuv3/bus_access/ showed the workload was saturating DDR bandwidth during bulk SCAN operations. Fix: added PRFM PLDL1KEEP prefetch hints 256 bytes ahead in the key iteration loop, improving SCAN throughput by 22%.

Result: After all three fixes, Graviton3 throughput was 31% higher than the original Intel baseline — at 40% lower cost per hour. None of these optimizations would have been found without PMU-guided profiling.

From Hardware Logic Analyzers to PMU Counters

In the early ARM days (ARM2, 1986), performance profiling meant connecting a logic analyzer to the external bus and counting clock edges manually. The first on-chip performance monitoring appeared in ARM9E (late 1990s) with a simple cycle counter. ARMv6 introduced the PMU specification with 4 configurable event counters. ARMv7 standardized the interface (CP15 access, event codes), and ARMv8 moved everything to System Registers (MSR/MRS) with up to 31 counters and architecture-defined event codes. Modern Neoverse cores (V1, N2) support 6+ configurable counters with Arm's Statistical Profiling Extension (SPE) — which records individual instruction events (latency, data source, branch outcome) into a memory buffer, enabling perf record -e arm_spe// for instruction-level profiling without sampling bias.

PMU Cycle Counting from User Space

Write a C program that measures the cycle cost of a simple operation:

1. Enable PMU user-space access (use the pmu_init_el1() pattern from this article, or on Linux, use perf_event_open() syscall with PERF_COUNT_HW_CPU_CYCLES)
2. Read PMCCNTR_EL0 before and after a tight loop of 1000 NOP instructions
3. Run 100 iterations and record the minimum cycle count
4. Calculate cycles-per-NOP (should be close to 0.25 on a 4-wide core, or pipeline fill overhead)

Challenge: Add ISB serialization before each counter read and explain why the minimum is ~4 cycles higher than without ISB.

Cache Miss Profiling with perf

Create two C programs with identical computation but different memory access patterns:

1. Sequential access: Iterate through a 4 MB array linearly (arr[i] += 1)
2. Random access: Use a pre-shuffled index array to access the same 4 MB array randomly
3. Profile both with: perf stat -e L1-dcache-load-misses,L1-dcache-loads,LLC-load-misses,LLC-loads
4. Calculate the L1D miss rate and LLC miss rate for each pattern

Expected result: Sequential access should have <1% L1D miss rate (hardware prefetcher effective). Random access should have 30-50% L1D miss rate on a 32KB L1D. The difference in wall-clock time should be 5-10× — demonstrating that memory access pattern dominates performance.

Branch Elimination Benchmark

Measure the impact of branch misprediction and demonstrate CSEL elimination:

1. Write a function that clamps values using an if/else: if (x > max) x = max; else if (x < min) x = min;
2. Write the same function using inline asm with CMP + CSEL (no branches)
3. Profile both with: perf stat -e branches,branch-misses,cycles,instructions
4. Test with: (a) sorted input (predictable branches), (b) random input (unpredictable)

Expected: With random input, the branching version shows 25-40% branch miss rate on the clamp path. CSEL version shows 0 branch misses for clamping (only the loop branch remains). The IPC difference reveals the misprediction penalty.