AI Already Did It › System & Utilities › Hardware & Sensors › Benchmarking

Ollama Bench

Benchmark your GPUs or CPU with real LLM inference through Ollama: tokens/s, time to first token, VRAM and utilisation, with a CLI and a desktop app.

Screenshots

Description

Ollama Bench measures how fast your hardware actually runs local LLMs. It drives Ollama with reproducible prompts and reports the numbers that matter:

• Generation speed (tokens/s), prefill speed on a ~2k-token prompt, time to first token, and model load time
• Where the model runs: 100% GPU, 100% CPU, or a CPU/GPU offload split
• GPU utilisation, VRAM, power and temperature sampled during every test (NVIDIA and AMD), plus CPU and RAM

It is built to push hardware to its limit, not just to time a single request:

• Stress mode runs several concurrent streams. Ollama only batches requests when the server allows it, so the tool starts a temporary Ollama server with OLLAMA_NUM_PARALLEL set for you and reports the effective parallelism.
• VRAM fill searches for the largest context size that still fits entirely in GPU memory.
• Per-GPU passes: on multi-GPU machines it benchmarks each card on its own (pinned temporary servers, no reconfiguration), then all cards together.
• CPU-only mode for machines without a GPU, or for a direct GPU-vs-CPU comparison.
• Self-repair: a model whose files fail to load is re-pulled automatically and the run continues.

Two front ends share one engine:

• a CLI with a coloured summary table, JSON and Markdown reports, and comparison against earlier runs
• a desktop app (Qt) with live per-GPU gauges, a rolling utilisation history plot, a live log, and a results table with charts

Ships as an AppImage and a .deb for Linux, an MSI for Windows (bundling its own Python and Qt), or runs from source on any OS. Requires a running Ollama with at least one model. MIT licensed.

Example results on an RTX 4090 + RTX 5060 Ti: gpt-oss:20b generates at 199 tok/s on the 4090 and 98 tok/s on the 5060 Ti; four concurrent streams lift aggregate throughput to 328 tok/s at 93% GPU utilisation.

If the site's editor does not like hard line breaks inside paragraphs and list items, join each wrapped line onto one line. The content is otherwise plain CommonMark: bold, one link, inline code and bullet lists

Downloads

• OllamaBench-2.0.0-x64.msi — Windows, 38.2 MB
• OllamaBench-2.0.0-x86_64.AppImage — Linux, 77.5 MB
• ollama-bench_2.0.0_amd64.deb — Linux, 75.1 MB
• llm_benchmark.tar.gz — Source, 306.2 KB

Original prompt(s)

Prompt 1

Create a benchmark software which tests GPUs/CPus (if GPU is not avaliable)  performance by using ollama.
For example my current system has rtx 4090 24GB and rtx 5060 ti 16GB GPUs and ollama. I want te test its performenace, for exmaple token / sec.

The Benchmark software should also use GPU at full power (Volatile GPU-Util %100 and Memory Usage %100 ) if it is possible

if GPU is not enable on the system, behcmark can also test the system by using CPU

Tags: benchmark, cuda, gpu, hardware-testing, inference, llm, ollama, performance, tokens-per-second

Open in AI Already Did It