Technical Guide: Optimizing GGUF Quantization for GitNeural with Shoehorn

Practical Overview & Architecture

Shoehorn is a specialized tool designed to quantize a BF16 GGUF model with an imatrix so it fits exactly into your available VRAM, then run it with llama.cpp. Preset quantizations like Q4_K_M and Q5_K_S ignore your specific hardware. They force you to either leave hundreds of megabytes of quality unused or discover at load time that the model does not fit after all. Shoehorn starts from the exact memory you actually have, subtracts what inference itself will need including the KV cache and compute buffers, and solves a per-tensor mixed-precision assignment whose total size lands within a rounding error of the remainder. Every spare megabyte goes where the importance matrix indicates it buys the most model quality.

The quantizer is implemented entirely from scratch in Rust with no llama.cpp code linked directly in the core engine. The resulting output files are standard GGUF v3 binaries that any standard llama.cpp build or downstream consumer loads directly. Underlying math mirrors ggml's object formulations, combining importance matrices from calibration text with multi-grid candidate exploration. This ensures compatibility with the standard decoder while leveraging an advanced knapsack solver to maximize utilization. Peak memory consumption during processing remains extremely low because the BF16 source file is mmap'd and never fully materialized in RAM during the optimization sweep.

Prerequisites & Installation & Setup

Before running Shoehorn, your system requires an installed version of llama.cpp to supply the underlying inference backend and support imatrix generation tasks. For macOS users on Apple Silicon, prebuilt binaries and dependencies can be installed via package managers. For instance, running

brew install notactuallytreyanastasio/shoehorn/shoehorn

cargo install --path .

On Linux environments targeting NVIDIA GPUs, you must build llama.cpp from source with CUDA enabled or acquire a matching release binary and place it on your system PATH. The VRAM probe reads the first NVIDIA device's free memory automatically through NVML, which ships with the standard driver requiring no extra installation. AMD setups fall back to

rocm-smi

Documented Implementation Workflow

The most streamlined way to execute the pipeline is via the automated fit command. Running

shoehorn fit unsloth/Qwen3-4B-GGUF --serve

shoehorn ui

If you prefer performing the steps manually by hand, the documented workflow starts by generating an importance matrix over calibration text using

llama-imatrix -m model-bf16.gguf -f calibration.txt -o model.imatrix -ngl 99

shoehorn quantize -m model-bf16.gguf -i model.imatrix --ctx 8192 -o fitted.gguf

shoehorn run -m fitted.gguf --ctx 8192

shoehorn plan

Known Limitations, Tradeoffs & Error Scenarios

While Shoehorn automates precise VRAM fitting, several limitations and hardware constraints remain documented. Intel GPUs are not probed automatically by the VRAM discovery tool, requiring users on those platforms to manually pass explicit budget flags such as

--budget

-i

Additional constraints apply to specific tensor types during optimization. Norms, biases, and one-dimensional structures remain strictly unquantized at F32 values to protect numerical stability, perfectly matching standard llama.cpp conventions. Token embeddings and output weights are strictly floored at 4-bit formats like IQ4_XS because embedding matrices lack imatrix tracking data, making weighted mean squared error metrics unreliable for language model heads. Users experiencing memory pressure must also account for KV cache growth scaling directly with context length, which reduces the absolute space available for weight allocations unless lower-precision KV types like

--kv q8_0

Who Should Use It & Production Fit

Shoehorn is built specifically for machine learning engineers, local AI hobbyists, and developers who deploy large language models on edge hardware or memory-constrained GPUs. If you routinely experience out-of-memory errors when loading standard fixed-quantization GGUF files like Q4_K_M or find yourself wasting hundreds of megabytes of available VRAM because the next preset tier is too large, this tool provides an exact mathematical solution. By tailoring the bit allocation of every individual tensor to match the specific remaining hardware capacity after accounting for compute buffers and active context windows, it extracts maximum possible intelligence from constrained hardware profiles.

The project integrates smoothly into existing local inference architectures since its outputs are fully standard GGUF v3 files compatible with any downstream tool capable of reading llama.cpp models. Production environments leveraging automated local model pipelines can utilize CLI primitives like

shoehorn fit

shoehorn eval