Benchmarking Local AI and ML Performance with ai-ml-gpu-bench

Practical Summary and Purpose

The ai-ml-gpu-bench suite is a specialized tool designed to measure the performance of consumer-grade hardware when executing common artificial intelligence and machine learning workloads. By providing a standardized testing environment, it allows users to evaluate both CPU and GPU capabilities through two primary benchmarks: Ollama-based large language model (LLM) inference and XGBoost training and inference on the HIGGS dataset. The suite is orchestrated via a single configuration file and a runner script, ensuring that benchmarks are reproducible and comparable across different systems. Results are presented through an automatically generated Jupyter notebook and can be contributed to a public dashboard to help build a broader reference dataset for hardware performance.

Prerequisites and System Requirements

To utilize the suite effectively, your system must meet specific software requirements. The tool requires Python ≥ 3.13 as the runtime environment and uv 0.8.x as the package manager for dependency management and environment isolation. For GPU-accelerated benchmarks, CUDA ≥ 12.x is necessary, along with appropriate NVIDIA drivers. If you intend to perform LLM benchmarks, you must have Ollama installed and running locally at http://localhost:11434. While these components are essential for full functionality, the suite is designed to be flexible; for instance, if a GPU is not detected or selected in the configuration, the tool will automatically skip GPU-specific tests, allowing users with CPU-only systems to still participate in the benchmarking process.

Environment Setup and Configuration

Getting started involves cloning the repository and allowing the tool to manage its own dependencies. Once the repository is cloned, the uv package manager handles the installation of the required Python environment. The core of the benchmarking process is the ai_bench_suite.yaml file, which serves as the central configuration hub. In this file, users can define their machine's hardware details, such as the CPU and GPU models, to ensure accurate reporting. Users can also customize the benchmark scope by commenting out specific LLM models they do not wish to test. It is recommended to keep the default configuration to contribute to the standardized reference results available at https://ai-ml-gpu-bench.streamlit.app.

Documented Execution Workflow

The execution of the benchmark is designed to be straightforward, requiring only a single command to initiate the process. The runner script, run_suite.py, reads the configuration file and orchestrates the tests. During a standard run, the tool generates a unique run_id, executes the selected benchmarks, and records the performance metrics into CSV files. If you need to pull missing Ollama models automatically, you can use the --autopull flag. The following command serves as the primary entry point for running the full suite:

git clone https://github.com/albedan/ai-ml-gpu-bench
cd ai-ml-gpu-bench
uv run run_suite.py

The tool also supports specific flags for targeted testing, such as --suite to run only Ollama or XGBoost tests, and --fast to limit Ollama benchmarks to a subset of smaller, faster models.

Interpreting Results and Privacy

Upon completion of the benchmarks, the suite automatically executes and opens a Jupyter notebook, bench_results_analysis_altair.ipynb, in your browser. This notebook provides an immediate visual analysis of your system's performance, highlighting your results against existing reference data. Regarding data privacy, the suite offers a mechanism to share results to help grow the public reference base. If enabled, the generated CSV files are encrypted using an RSA 4096-bit scheme before being uploaded to Filebin. This process only submits technical benchmark data, excluding any prompts, model outputs, or raw system files. Users who prefer not to share their data can opt out entirely by using the --no-upload-results flag during execution.

Limitations and Troubleshooting

While the suite is highly versatile, there are documented limitations and common troubleshooting steps. For example, while Ollama can leverage various GPUs, XGBoost may default to CPU-only execution if the CUDA toolkit is not correctly configured or detected. Users experiencing issues with XGBoost on NVIDIA hardware should verify their installation by running nvidia-smi and nvcc -V to confirm the presence of the GPU and the correct CUDA version. Additionally, for older hardware, it is advised to limit the benchmark to smaller models like phi3:3.8b to ensure the tests complete successfully. If you encounter issues with downloading datasets, installing pip-system-certs via uv may resolve certificate-related errors.