Run large language models locally without sending data to the cloud with Ollama

The problem Ollama solves

Every call to a hosted model API ships your prompt — often customer data, proprietary code, or unfinished ideas — to someone else's server, metered per token. For prototypes that is fine. For internal tools, regulated data, or offline work, it is a non-starter.

Ollama fixes this by moving inference to localhost. Models run on your CPU or GPU, the API listens on http://localhost:11434, and nothing leaves the machine. The trade-off is explicit: you give up frontier-model quality and infinite scale, and in return you get zero marginal cost, zero network dependency, and complete data privacy.

In this tutorial, you'll run your first local model, call it from curl and Python, and learn the two operational details that trip up every beginner: model loading latency and streaming output.

How to get started with Ollama in 5 minutes

1. Install. On Linux: curl -fsSL https://ollama.com/install.sh | sh. On Windows (10 or later) in PowerShell: irm https://ollama.com/install.ps1 | iex, or grab OllamaSetup.exe from the download page. macOS has a native app. (Source: ollama.com/download.)
2. Pull a model. ollama pull llama3.2 downloads the default 3B model. Model names follow a model:tag format; omitting the tag gives you latest.
3. Chat once to verify. ollama run llama3.2 opens a prompt. Type something, confirm words come out, type /bye to exit.
4. Confirm the API is up. curl http://localhost:11434/api/tags should list your downloaded models as JSON.
5. Pick your next model. Small machine: stay with 3B models like llama3.2. With 16GB+ RAM: ollama pull mistral or a coder model for programming tasks.

How to use Ollama: complete tutorial

Step 1: Call the API with curl

The POST /api/generate endpoint takes a model and a prompt. Set "stream": false to get one JSON object back instead of a token stream — always do this first, while you are learning the shape of the API:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain what a REST API is in one sentence.",
  "stream": false
}'

The response contains a response string plus timing statistics (eval_count / eval_duration are tokens and nanoseconds — divide to get tokens/sec). Endpoint and parameters per the official API reference.

Step 2: Call it from Python with zero dependencies

No SDK to install — the standard library is enough, which also means there is no SDK version to go stale:

import json, urllib.request

payload = json.dumps({
    "model": "llama3.2",
    "prompt": "Explain what a REST API is in one sentence.",
    "stream": False,
}).encode()

req = urllib.request.Request(
    "http://localhost:11434/api/generate",
    data=payload,
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(req, timeout=120) as res:
    print(json.loads(res.read())["response"])

Edge case — the first call is slow: Ollama unloads models after 5 minutes idle (keep_alive defaults to 5m), so the first request pays the full model-load cost and can take a minute on CPU. The generous timeout=120 above is load-bearing, not decorative. Subsequent calls are fast until the model unloads again. If you need deterministic latency, send a warm-up request or raise keep_alive.

Step 3: Hold a conversation with /api/chat

Single prompts are stateless. For chat, POST /api/chat accepts a messages array with user, assistant, and system roles — same shape as the OpenAI chat API, and Ollama additionally serves an OpenAI-compatible endpoint so existing OpenAI client code works by pointing base_url at http://localhost:11434/v1:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    { "role": "system", "content": "Answer in one sentence." },
    { "role": "user", "content": "Why is the sky blue?" }
  ],
  "stream": false
}'

Edge case — streaming output: with streaming on (the default), the endpoint returns newline-delimited JSON objects, one per token, not one JSON document — calling json.loads on the whole body fails. Either keep stream: false for scripts, or parse line-by-line and concatenate each chunk's response (or message.content for chat). If you render chunks into a web page as they arrive, buffer them client-side and flush on a short timer instead of touching the DOM per token, or rapid updates will visibly flicker.

Ollama: pros & cons

Ollama pricing: free vs paid

There is no paid tier. Ollama is free to download and use with no account and no usage billing — the economics are that you pay in hardware instead of tokens. A rough rule of thumb: 3B-parameter models run on almost anything, 7–8B models want around 8GB of free RAM, and anything larger is GPU territory. Electricity and your machine are the entire cost model.

Check the official download page for current platform support.

Who is Ollama best for?

For developers building with LLM features: a local backend makes integration tests deterministic, free, and offline — no test-suite API bills, no leaked fixtures.

For privacy-constrained teams: healthcare, legal, and enterprise code that cannot leave the building finally get a usable LLM story.

For learners: breaking things locally is free. Prompting, embeddings, tool-calling experiments — all repeatable at zero marginal cost.

Who should not use Ollama?

If you need the strongest available reasoning, long-context analysis, or production traffic at scale, use a hosted frontier API instead — a local 3B model will disappoint you and the article would be lying to suggest otherwise. Ollama is also the wrong choice on machines with no headroom: swapping a 7B model on 4GB of RAM produces tokens slower than you read.

Alternatives to Ollama

LM Studio offers a friendlier desktop UI over the same local-model idea. llama.cpp is the lower-level C++ engine underneath much of this ecosystem, for maximum control. GPT4All targets non-technical users with one-click model installs. Ollama remains the best pick when you want scriptability: its CLI-plus-REST-API combination is what the others make you work harder for.

How we evaluated Ollama

This tutorial is based on Ollama's official documentation as fetched on September 7, 2026: the download page for install commands and the project's API reference for every endpoint, parameter, and default cited above (including keep_alive: 5m and the model:tag convention). Commands are quoted from those sources. No hands-on benchmark was run for this draft — timings described are the documented mechanics, not measurements.

Final verdict: is Ollama worth it?

For any developer curious about local LLMs, Ollama is the obvious starting point: ten minutes from zero to a working private API, with an interface clean enough to build on. It will not replace hosted frontier models, and it does not try to.