Undefined Behavior (UB)

Level 13 — Unsafe Rust & FFI Operations or program execution states that violate the language specification's core memory and execution rules, allowing the compiler to optimize under assumptions that result in completely unpredictable program execution, memory corruption, or silent data corruption.

1. Prerequisites

• References and Borrowing (&, &mut) — Rules regarding reference validity, lifetimes, and aliasing XOR mutability.

2. Term Category

Rust Safety Boundary (compiler invariants & unsound code execution): Undefined Behavior (UB) is a fundamental concept in systems programming languages like Rust, C, and C++. In Rust, the compiler optimizes code under the strict assumption that UB can never occur. If an unsafe block executes code that triggers UB, the compiler's optimizations become invalid, leading to unpredictable program crashes, silent data corruption, time-travel compiler optimization bugs, or security vulnerabilities.

3. Explanation

(1) Design Motivation — "Why did we design this?"

In high-level garbage-collected languages like JavaScript, Python, or Java, invalid operations (such as accessing an out-of-bounds array element or dereferencing null) throw runtime exceptions (TypeError, ArrayIndexOutOfBoundsException). The language specification guarantees that even buggy code fails safely without corrupting unrelated program memory.

In systems languages like Rust, compiler optimization pipelines (LLVM) achieve zero-cost abstractions by making strict, mathematical assumptions about machine code:

1. "A standard reference &T will never be null or point to uninitialized memory."
2. "A mutable reference &mut T is guaranteed to be the sole exclusive pointer to that memory location."
3. "Data races between threads never happen."

If a developer breaks one of these assumptions inside an unsafe block:

• The compiler does NOT promise to throw a clean runtime panic or error.
• The compiler optimizer may silently delete code paths, reorder memory instructions incorrectly, cause time-traveling logic bugs (where code executed before the UB line behaves erratically), or corrupt adjacent memory variables.

Safe Rust guarantees that safe code can NEVER cause Undefined Behavior. The entire purpose of Rust's ownership system, lifetime tracking, and borrow checker is to ensure that valid Rust code remains 100% UB-free at compile time.

(2) What Specifically Causes Undefined Behavior in Rust?

The Rust Language Reference explicitly defines what operations constitute Undefined Behavior:

1. Dereferencing null, dangling, or unaligned pointers.
2. Violating aliasing rules: Creating aliased &mut T references or mutating data behind a shared &T reference without interior mutability.
3. Data races: Accessing a memory location concurrently from two threads where at least one access is a non-atomic write.
4. Producing invalid values: Creating an bool that is not 0 or 1, an enum variant with an invalid discriminant tag, a char outside Unicode range, or an uninitialized reference.
5. Executing code compiled with target features the CPU doesn't support.
6. Unwinding across an FFI boundary (e.g., panicking inside an extern "C" function).

(3) Reality Metaphor

Imagine a High-Speed Maglev Train on Automated Magnetic Tracks:

• Safe Rust is the automated maglev guidance computer: it enforces speed limits, keeps trains on designated tracks, and prevents collisions.
• Undefined Behavior (UB) is like manually overriding the guidance system inside the cab (unsafe block) and throwing a steel beam directly onto the magnetic track while traveling at 300 mph.
  • The train doesn't just display an "Error Code: Steel Beam Detected" screen (it doesn't throw a clean exception).
  • The physics of the magnetic tracks are completely disrupted: the train derails, smashes into signals 2 miles down the track, and destroys surrounding infrastructure (unpredictable memory corruption / time-travel optimization bugs).

(4) Code Examples

Short Snippet (Classic Undefined Behavior: Creating Aliased &mut)

fn main() {
    let mut num: i32 = 42;

    // Creating two raw pointers to `num` is 100% SAFE
    let ptr1: *mut i32 = &mut num as *mut i32;
    let ptr2: *mut i32 = ptr1;

    unsafe {
        // ❌ UNDEFINED BEHAVIOR (UB)!
        // Converting raw pointers into simultaneous `&mut i32` references violates Rust's aliasing rules!
        let ref1: &mut i32 = &mut *ptr1;
        let ref2: &mut i32 = &mut *ptr2;

        *ref1 = 10;
        *ref2 = 20; // LLVM compiler optimizations assume ref1 and ref2 cannot alias!
        println!("ref1: {}, ref2: {}", ref1, ref2);
    }
}

Fuller Example (Catching Undefined Behavior with Miri)

/// A buggy custom uninitialized memory buffer demonstration.
/// Miri (Rust's official UB interpreter) will flag this immediately!
fn create_invalid_bool() -> bool {
    unsafe {
        // ❌ UNDEFINED BEHAVIOR (UB)!
        // `bool` in Rust MUST be represented in memory as byte 0x00 (false) or 0x01 (true).
        // Byte 0x05 is an INVALID VALUE representation for bool!
        let invalid_byte: u8 = 5;
        let b: bool = std::mem::transmute(invalid_byte);
        b
    }
}

fn main() {
    // Run this file with `cargo miri run` to catch the UB!
    // In standard release compilation, this produces unpredictable pattern evaluation!
    let invalid = create_invalid_bool();
    
    // The compiler optimizer assumes `invalid` is either true or false.
    // An invalid bool representation can cause both branches to execute or neither!
    if invalid == true {
        println!("It is true!");
    } else if invalid == false {
        println!("It is false!");
    }
}

4. Common Mistakes & Pitfalls

Mistake 1: Believing UB Always Causes an Immediate Crash

The mistake: Testing code with unsafe blocks and assuming that because cargo run didn't crash or segfault, the code has no UB.

Why it's wrong: Undefined Behavior does not guarantee a crash. Code with UB may appear to work fine during local testing, but crash or silently corrupt production databases under a different compiler version, optimization level (-O3), or CPU target architecture.

Incorrect:

// "Works on my machine during testing!"
unsafe {
    let ptr: *const i32 = std::ptr::null();
    // ❌ Still UNDEFINED BEHAVIOR, even if it happens not to crash in debug mode!
}

Fix:

// Always validate unsafe code using Miri (`cargo miri run`) and sanitizers
unsafe {
    let ptr: *const i32 = std::ptr::null();
    if let Some(r) = ptr.as_ref() {
        println!("{}", r);
    }
}

Mistake 2: Confusing Panic with Undefined Behavior

The mistake: Believing that panic!("out of bounds") or Option::unwrap() on None is Undefined Behavior.

Why it's wrong: Panicking is 100% SAFE behavior. A panic safely unwinds the stack (or aborts the process cleanly), cleans up resources, and prevents memory corruption. UB, by contrast, bypasses safety checks and causes unhandled memory corruption.

Incorrect:

// Thinking this panic is unsafe:
fn get_elem(arr: &[i32], idx: usize) -> i32 {
    arr[idx] // Safe! Panics cleanly on out-of-bounds index.
}

Fix:

// Out-of-bounds indexing in Safe Rust is 100% UB-free because it panics safely.
// `get_unchecked` in Unsafe Rust bypasses bounds checking and causes UB!
unsafe fn get_elem_unchecked(arr: &[i32], idx: usize) -> i32 {
    unsafe { *arr.as_ptr().add(idx) } // CALLER MUST PREVENT UB!
}

Mistake 3: Creating Uninitialized Memory via mem::uninitialized

The mistake: Using std::mem::uninitialized() to create uninitialized memory buffers for custom data structures.

Why it's wrong: std::mem::uninitialized() was deprecated and replaced with std::mem::MaybeUninit<T> because instantiating types (especially types containing references or enums) with uninitialized bytes is instant UB.

Incorrect:

unsafe {
    // ❌ UNDEFINED BEHAVIOR! Instant UB for types with invalid bit patterns
    let x: String = std::mem::uninitialized(); 
}

Fix:

use std::mem::MaybeUninit;

// Correct: `MaybeUninit<T>` explicitly informs the compiler that memory is uninitialized
let mut x: MaybeUninit<String> = MaybeUninit::uninitialized();
// Write initialized data safely before calling `.assume_init()`

5. Practice Exercises

Exercise 1: Sound Safe Abstraction for Uninitialized Stack Buffers (MaybeUninit)

Scenario: In embedded network drivers and high-throughput packet processing engines, pre-allocating or zero-initializing large I/O buffers (e.g., 1024-byte packet RX buffers) before passing them to DMA (Direct Memory Access) hardware incurs unnecessary runtime performance overhead. However, creating uninitialized memory in Rust can easily lead to instant Undefined Behavior (UB) if uninitialized bytes are read as initialized types or if raw references are formed prematurely.

Implement a type-safe #![no_std] stack buffer structure UninitPacketBuffer<const CAP: usize> using core::mem::MaybeUninit<u8>. Provide methods:

1. new() -> Self to construct an uninitialized buffer safely without UB.
2. raw_mut_ptr(&mut self) -> *mut u8 to expose raw uninitialized storage to hardware I/O fill routines.
3. set_initialized(&mut self, len: usize) -> Result<&[u8], BufferError> to safely mark len bytes as initialized and return an initialized byte slice (&[u8]), returning an error if len > CAP.
4. reset(&mut self) to reset the initialized length counter safely.

Write unit tests with assert_eq! and assert! verifying buffer creation, partial writes, bounds validation, and safe slice retrieval without UB.

#![no_std]

use core::mem::MaybeUninit;

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum BufferError {
    CapacityExceeded,
}

/// A zero-allocation packet receiver buffer using `MaybeUninit<u8>` to prevent UB.
pub struct UninitPacketBuffer<const CAP: usize> {
    storage: [MaybeUninit<u8>; CAP],
    initialized_len: usize,
}

impl<const CAP: usize> UninitPacketBuffer<CAP> {
    /// Constructs a buffer with uninitialized memory without triggering UB.
    pub fn new() -> Self {
        Self {
            // SAFETY: An array of `MaybeUninit<u8>` in an uninitialized state is valid
            // and does NOT constitute UB, unlike `mem::uninitialized()`.
            storage: unsafe { MaybeUninit::uninit().assume_init() },
            initialized_len: 0,
        }
    }

    /// Provides a raw mutable pointer slice for hardware DMA or driver writes.
    pub fn raw_mut_ptr(&mut self) -> *mut u8 {
        self.storage.as_mut_ptr() as *mut u8
    }

    /// Safely marks `len` bytes as initialized after data has been written into storage,
    /// returning a reference to the initialized slice.
    pub fn set_initialized(&mut self, len: usize) -> Result<&[u8], BufferError> {
        if len > CAP {
            return Err(BufferError::CapacityExceeded);
        }
        self.initialized_len = len;

        unsafe {
            // SAFETY:
            // 1. Caller/driver guaranteed that `len` bytes have been populated with initialized data.
            // 2. `self.storage` remains valid for the lifetime of `&self`.
            // 3. Pointer casting from `*const MaybeUninit<u8>` to `*const u8` is valid because
            //    `MaybeUninit<T>` has the exact same layout and alignment as `T`.
            let ptr = self.storage.as_ptr() as *const u8;
            Ok(core::slice::from_raw_parts(ptr, len))
        }
    }

    /// Returns the slice of initialized data.
    pub fn initialized_slice(&self) -> &[u8] {
        unsafe {
            let ptr = self.storage.as_ptr() as *const u8;
            core::slice::from_raw_parts(ptr, self.initialized_len)
        }
    }

    /// Resets the initialized length counter without invalidating memory.
    pub fn reset(&mut self) {
        self.initialized_len = 0;
    }

    /// Returns current initialized byte length.
    pub const fn len(&self) -> usize {
        self.initialized_len
    }

    /// Checks if buffer is empty.
    pub const fn is_empty(&self) -> bool {
        self.initialized_len == 0
    }

    /// Returns maximum capacity.
    pub const fn capacity(&self) -> usize {
        CAP
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_uninit_buffer_initialization_and_reads() {
        let mut buf: UninitPacketBuffer<128> = UninitPacketBuffer::new();
        assert_eq!(buf.capacity(), 128);
        assert_eq!(buf.len(), 0);
        assert!(buf.is_empty());

        // Simulate I/O fill writing into raw pointer
        let raw_ptr = buf.raw_mut_ptr();
        let payload = b"GET /api/v1/telemetry HTTP/1.1";
        unsafe {
            core::ptr::copy_nonoverlapping(payload.as_ptr(), raw_ptr, payload.len());
        }

        // Safely set initialized count
        let slice = buf.set_initialized(payload.len()).unwrap();
        assert_eq!(slice, payload);
        assert_eq!(buf.len(), payload.len());
        assert_eq!(buf.initialized_slice(), payload);
    }

    #[test]
    fn test_uninit_buffer_bounds_check() {
        let mut buf: UninitPacketBuffer<16> = UninitPacketBuffer::new();
        assert_eq!(buf.set_initialized(32), Err(BufferError::CapacityExceeded));
        assert_eq!(buf.len(), 0);
    }

    #[test]
    fn test_uninit_buffer_reset() {
        let mut buf: UninitPacketBuffer<64> = UninitPacketBuffer::new();
        let ptr = buf.raw_mut_ptr();
        unsafe {
            *ptr = 0xAA;
            *ptr.add(1) = 0xBB;
        }
        buf.set_initialized(2).unwrap();
        assert_eq!(buf.initialized_slice(), &[0xAA, 0xBB]);

        buf.reset();
        assert!(buf.is_empty());
        assert_eq!(buf.initialized_slice(), &[]);
    }
}

Exercise 2: Sound Non-Overlapping In-Place Slice Mutation (Eliminating Aliasing UB)

Scenario: In real-time signal processing and cryptographic buffer transformation pipelines, developers often perform in-place byte XOR or audio gain scaling using raw pointers for speed. A naive implementation that takes two overlapping raw mutable pointers and constructs simultaneous &mut [u8] slices triggers severe Aliasing Undefined Behavior (UB) under LLVM's noalias optimization rules.

Implement a sound, UB-free #![no_std] function sound_in_place_xor(data: &mut [u8], key: &[u8]) and a raw pointer slice splitter safe_split_mut<T>(slice: &mut [T], mid: usize) -> Option<(&mut [T], &mut [T])>:

1. sound_in_place_xor must mutate data in-place by XORing bytes with key cyclically using raw pointer arithmetic or safe iterators without creating aliased mutable references.
2. safe_split_mut must safely split a mutable slice into two disjoint mutable sub-slices (&mut [T], &mut [T]) using raw pointers (core::slice::from_raw_parts_mut), validating mid <= slice.len() to guarantee zero memory overlap and eliminate aliasing UB.

Write comprehensive unit tests with assert_eq! verifying decryption round-trips, slice splitting, disjoint mutability, and edge cases.

#![no_std]

/// In-place byte XOR mask operation without creating aliased &mut references.
pub fn sound_in_place_xor(data: &mut [u8], key: &[u8]) {
    if key.is_empty() || data.is_empty() {
        return;
    }

    let len = data.len();
    let key_len = key.len();
    let data_ptr = data.as_mut_ptr();
    let key_ptr = key.as_ptr();

    unsafe {
        // SAFETY:
        // 1. `data_ptr` and `key_ptr` are derived from valid slices `data` and `key`.
        // 2. Iteration stays within bounds `0..len` and `0..key_len`.
        // 3. We read values using raw pointers and write back to single `data_ptr`,
        //    creating NO simultaneous overlapping `&mut` references.
        for i in 0..len {
            let key_byte = *key_ptr.add(i % key_len);
            let data_byte_ptr = data_ptr.add(i);
            *data_byte_ptr = *data_byte_ptr ^ key_byte;
        }
    }
}

/// Safely splits a mutable slice into two non-overlapping mutable sub-slices using raw pointers.
pub fn safe_split_mut<T>(slice: &mut [T], mid: usize) -> Option<(&mut [T], &mut [T])> {
    if mid > slice.len() {
        return None;
    }

    let len = slice.len();
    let ptr = slice.as_mut_ptr();

    unsafe {
        // SAFETY:
        // 1. `ptr` points to valid initialized elements of `slice`.
        // 2. `mid <= len`, so `ptr.add(mid)` is within or at the boundary of allocation.
        // 3. Memory regions `[ptr, ptr + mid)` and `[ptr + mid, ptr + len)` are non-overlapping (disjoint).
        //    This strictly obeys Rust's aliasing XOR mutability rule (no two &mut point to same element).
        let left = core::slice::from_raw_parts_mut(ptr, mid);
        let right = core::slice::from_raw_parts_mut(ptr.add(mid), len - mid);
        Some((left, right))
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_sound_xor_roundtrip() {
        let mut message = *b"Embedded Security Core 2026";
        let key = b"SECRET_KEY_123";

        // Encrypt in-place
        sound_in_place_xor(&mut message, key);
        assert_ne!(&message, b"Embedded Security Core 2026");

        // Decrypt in-place (XOR again)
        sound_in_place_xor(&mut message, key);
        assert_eq!(&message, b"Embedded Security Core 2026");
    }

    #[test]
    fn test_safe_split_mut_disjoint_mutation() {
        let mut numbers = [10, 20, 30, 40, 50, 60];
        let (left, right) = safe_split_mut(&mut numbers, 3).unwrap();

        assert_eq!(left, &[10, 20, 30]);
        assert_eq!(right, &[40, 50, 60]);

        // Mutate both non-overlapping halves independently
        left[0] = 100;
        right[2] = 600;

        assert_eq!(numbers, [100, 20, 30, 40, 50, 600]);
    }

    #[test]
    fn test_safe_split_mut_out_of_bounds() {
        let mut data = [1, 2, 3];
        assert!(safe_split_mut(&mut data, 4).is_none());
    }
}

Exercise 3: Preventing Volatile Memory & Interrupt Data Race UB (UnsafeCell & Volatile I/O)

Scenario: In bare-metal microcontrollers, Hardware I/O Registers (MMIO) and interrupt status flags are updated asynchronously by external hardware peripheral hardware (e.g., UART RX interrupt or Timer hardware). Standard Rust compiler optimizations assume that memory locations do not spontaneously change value between reads, leading to LLVM optimizing out read loops (e.g., collapsing while reg.read() == 0 {} into an infinite loop or deleting repeated reads). Furthermore, accessing shared memory from an Interrupt Service Routine (ISR) without volatile or atomic abstractions causes severe Data Race UB.

Implement a #![no_std] hardware volatile register wrapper VolatileRegister<T: Copy>:

1. Use core::cell::UnsafeCell<T> as the underlying storage to allow interior mutability behind shared references (&VolatileRegister<T>).
2. Provide pub const fn new(val: T) -> Self for static register initialization.
3. Provide pub fn read_volatile(&self) -> T using core::ptr::read_volatile to force the compiler to issue actual hardware read instructions every time without optimizing them away.
4. Provide pub fn write_volatile(&self, val: T) using core::ptr::write_volatile to force hardware MMIO write instructions.
5. Provide pub fn set_bits(&self, mask: u32, enable: bool) for boolean flag updates when T = u32.

Include unit tests with assert_eq! verifying interior mutability, volatile reads and writes, and bit manipulation.

#![no_std]

use core::cell::UnsafeCell;

/// Type-safe hardware volatile memory register wrapper for MMIO & interrupt status flags.
/// Uses `UnsafeCell` for interior mutability and volatile pointer operations to prevent compiler optimization bugs & UB.
#[repr(transparent)]
pub struct VolatileRegister<T: Copy> {
    value: UnsafeCell<T>,
}

// SAFETY: VolatileRegister can be shared across thread/interrupt boundaries if T is Copy,
// provided accesses use volatile instructions and target architecture supports aligned single-word accesses.
unsafe impl<T: Copy + Send> Sync for VolatileRegister<T> {}

impl<T: Copy> VolatileRegister<T> {
    /// Creates a static volatile register with an initial value.
    pub const fn new(val: T) -> Self {
        Self {
            value: UnsafeCell::new(val),
        }
    }

    /// Reads register value using volatile load instruction.
    /// Prevents LLVM from caching values in registers or optimizing away polling loops.
    pub fn read_volatile(&self) -> T {
        unsafe {
            // SAFETY: `self.value.get()` returns a valid raw pointer `*mut T`.
            // Pointer is aligned, non-null, and points to initialized type T.
            core::ptr::read_volatile(self.value.get())
        }
    }

    /// Writes value into register using volatile store instruction.
    pub fn write_volatile(&self, val: T) {
        unsafe {
            // SAFETY: `self.value.get()` returns a valid mutable raw pointer `*mut T`.
            core::ptr::write_volatile(self.value.get(), val);
        }
    }
}

impl VolatileRegister<u32> {
    /// Atomically bitwise-updates bitmask flags on u32 register.
    pub fn modify(&self, f: impl FnOnce(u32) -> u32) {
        let current = self.read_volatile();
        let updated = f(current);
        self.write_volatile(updated);
    }

    /// Sets or clears specific bits specified by bitmask.
    pub fn set_bits(&self, mask: u32, enable: bool) {
        self.modify(|val| {
            if enable {
                val | mask
            } else {
                val & !mask
            }
        });
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_volatile_register_read_write() {
        let reg = VolatileRegister::new(0x00u32);
        assert_eq!(reg.read_volatile(), 0x00);

        reg.write_volatile(0xDEADBEEF);
        assert_eq!(reg.read_volatile(), 0xDEADBEEF);
    }

    #[test]
    fn test_volatile_register_bit_operations() {
        const RX_ENABLE: u32 = 1 << 0;
        const TX_ENABLE: u32 = 1 << 1;
        const INT_ENABLE: u32 = 1 << 4;

        let ctrl_reg = VolatileRegister::new(0);

        ctrl_reg.set_bits(RX_ENABLE | TX_ENABLE, true);
        assert_eq!(ctrl_reg.read_volatile(), 0b00011);

        ctrl_reg.set_bits(INT_ENABLE, true);
        assert_eq!(ctrl_reg.read_volatile(), 0b10011);

        ctrl_reg.set_bits(RX_ENABLE, false);
        assert_eq!(ctrl_reg.read_volatile(), 0b10010);
    }

    #[test]
    fn test_interior_mutability_shared_reference() {
        let reg = VolatileRegister::new(100u32);
        let shared_ref = &reg;

        // Mutate through shared reference (&VolatileRegister) without needing `&mut`
        shared_ref.write_volatile(250);
        assert_eq!(reg.read_volatile(), 250);
    }
}

6. Related Terms

• unsafe Block — The language scope where developers assume responsibility for preventing UB.
• Raw Pointers (*const T, *mut T) — Unchecked pointers whose misuse frequently causes UB.
• unsafe fn — Functions whose caller preconditions prevent UB.
• FFI (Foreign Function Interface) — Language boundaries where UB can occur across C/Rust boundaries.
• union — Related concept: union.
• unsafe trait / unsafe impl — Related concept: unsafe trait / unsafe impl.
• Miri (Undefined Behavior Detector) — Related concept: Miri (UB Detector).

7. Key Takeaways

• Undefined Behavior (UB) occurs when code violates core language memory and execution rules inside unsafe blocks.
• Common causes of UB include dereferencing null/dangling pointers, aliasing &mut T, data races, and creating invalid type bit patterns.
• Safe Rust code is guaranteed to be 100% free of Undefined Behavior.
• A program with UB may not crash immediately; it can result in silent data corruption or unpredictable optimization bugs.
• Always validate unsafe code using cargo miri run.