Optimizing Bare-Metal Transformer Inference with AXIOM

Practical Summary of AXIOM

AXIOM is a specialized, bootable no_std kernel written in Rust, designed specifically as an inference substrate rather than a general-purpose operating system. It addresses the performance bottlenecks inherent in running transformer models on standard Linux environments, where generic memory management and scheduler preemption often disrupt inference-critical tasks. By implementing inference-specific primitives—such as tensor-native memory allocation, layer-boundary scheduling, and double-buffered weight streaming—AXIOM aims to provide a stable, high-performance environment for memory-constrained inference, particularly for 7B-class models on hardware with limited RAM. The project is currently in an experimental phase, focusing on bare-metal execution to eliminate virtualization overhead and OS-level jitter.

Prerequisites and Build Environment

To utilize AXIOM, developers must work within a no_std Rust environment. The project requires the Rust nightly toolchain and the x86_64-unknown-none target. Essential components include rust-src and the bootimage crate for creating the bootable kernel. The build process assumes the use of specific target features, including +avx2, +fma, and +sse2, which must be enabled in the kernel to support efficient tensor operations. Users should ensure their development environment is configured to handle these specific architectural requirements before attempting to compile the kernel or run the provided QEMU-based testing scripts.

Documented Workflow and Setup

The workflow for AXIOM involves preparing model weights, building the kernel, and executing the inference task within a controlled environment. Users must first pack their model weights into a format compatible with the AXIOM runtime using the provided Python tools. The following sequence illustrates the typical setup flow:

rustup toolchain install nightly
rustup component add rust-src --toolchain nightly
cargo install bootimage
python tools/pack_weights.py --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --output axiom_weights.img --quant q4
cargo +nightly run --release

This process initializes the kernel, loads the weights, and executes the inference loop, which emits telemetry data regarding layer timing and throughput.

Core Architectural Primitives

AXIOM replaces standard Linux abstractions with three primary mechanisms. First, the Tensor-Native Allocator utilizes a pre-reserved, physically contiguous memory region divided into three pools: KVCachePool, WeightPool, and ActivationPool. This eliminates allocator churn during the hot path. Second, the LayerLock Scheduler suppresses timer-based preemption during the execution of a single transformer layer, ensuring cache residency is maintained. Finally, Double-Buffered Weight Streaming allows the system to overlap I/O and compute by loading the next layer's weights into a secondary buffer while the current layer is being processed, effectively hiding the latency of weight retrieval from storage.

Interpreting Performance Telemetry

Performance in AXIOM is measured through per-layer timing telemetry. The project's documentation highlights a significant reduction in streaming overhead, moving from second-scale delays to approximately 42 microseconds per layer after prefetch-path corrections. Current benchmarks indicate that once streaming overhead is minimized, the primary bottleneck shifts to compute-intensive operations, specifically FFN projection and down-projection. Users should monitor these metrics to identify whether their specific model configuration is I/O-bound or compute-bound. It is recommended to report median values over repeated trials rather than single samples, as VM-based execution can introduce material noise into the timing results.

Limitations and Current Status

AXIOM is currently an experimental project with several documented limitations. Benchmarks are performed on KVM with emulated virtio-blk storage, which does not fully replicate the performance characteristics of bare-metal NVMe. Furthermore, the current runtime includes a simplified output-head implementation, and the AVX2 intrinsic path has shown regressions compared to the scalar LUT path in some environments. The project intentionally excludes general userspace, networking, and complex process models, focusing strictly on the inference task. Users should be aware that these limitations are intentional design choices to maintain a minimal, predictable execution environment for research purposes.

Intended Use Cases

AXIOM is intended for developers and researchers working on memory-constrained inference, specifically scenarios where the model size exceeds available RAM and full residency is impossible. It is not designed for general-purpose computing or standard application deployment. The project is most useful for those investigating the impact of OS-level control over memory layout, I/O pipelines, and scheduling on transformer performance. By providing a substrate where these factors can be tuned at the kernel level, AXIOM serves as a platform for evaluating the feasibility of running large models on constrained hardware, such as 4 GB class systems, through optimized streaming and compute overlap.

Further Information

For detailed technical specifications, repository structure, and the latest updates on the project's progress, users should refer to the official documentation at https://github.com/Kanchisaw03/axiom. The repository contains the full source code, including the scheduler, memory management, and runtime subsystems, as well as tools for weight packing and benchmarking. As the project moves toward bare-metal NVMe evaluation, the documentation will likely be updated to reflect new findings regarding cache behavior and storage performance under realistic, memory-constrained conditions.