Series: 3 Parts • Published: August 2026 • Reading Time: 8–10 mins per part
It Started With a Simple Question
Can an ordinary Windows or Ubuntu development machine run a genuinely useful AI coding assistant locally?
Not a demo. Not a chatbot sitting in a browser. A real coding workflow where the model can inspect a repository, suggest changes, edit code, and work inside a developer-controlled environment.
That question turned into a surprisingly interesting engineering journey: choosing the model, downloading
the right GGUF, building llama.cpp, discovering that llama-server was not where we expected it,
connecting Aider correctly, measuring tokens per second, tuning memory and context, and finally deciding
what we actually needed — and what we didn’t.
The Three-Part Journey
- Part 1 — Make It Work: Build the local coding stack.
- Part 2 — Make It Fast: Tune the machine and eliminate wasted work.
- Part 3 — Make It Useful: Turn the local model into a practical coding workflow.
Part 1 — Make It Work
1. The Architecture
The first important discovery was that Aider does not need to load the GGUF model itself.
Developer | v Aider | | OpenAI-compatible HTTP API v llama-server | v GGUF coding model | v Your codebase
💡 Architect’s Insight
Keep the responsibilities separate. Aider is the coding agent. llama.cpp is the inference engine. The GGUF is the model.
2. Choosing the Starting Model
For a 16–32 GB development machine, the practical starting point was a 7B/8B coding model in Q4_K_M GGUF format. The specific journey used Qwen3-8B-Q4_K_M.gguf.
| Host | Starting Point | Why |
|---|---|---|
| Ubuntu 16 GB | 7/8B Q4_K_M | Keep memory pressure under control |
| Windows 32 GB | 7/8B Q4_K_M | Same model, Windows-specific runtime |
| Ubuntu 16 GB | 7/8B Q4_K_M | More room for context |
| Windows 32 GB | 7/8B Q4_K_M | More room for context |
3.1 Ubuntu: Build llama.cpp
sudo apt update sudo apt install -y \ build-essential cmake git curl wget \ python3 python3-pip ripgrep fd-find git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build -DCMAKE_BUILD_TYPE=Release cmake --build build --target llama-server -j$(nproc)
One of the first real lessons came from a build that apparently had server support enabled, while
llama-server was not in the expected executable list.
cmake --build build --target help | grep llama
If llama-server appears there, build that target explicitly.
⚠️ Failure That Taught Us Something
Seeing LLAMA_BUILD_SERVER=ON in CMake configuration is not the same thing as verifying that the
final llama-server executable is available where you expect it.
3.2 Windows: Same Model, Different Build
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build --config Release --target llama-server
The model remains portable. The build and hardware configuration are what change between operating systems.
4. Download the GGUF
The same GGUF model can be used on both Ubuntu and Windows. The only difference is how we download it and where we store it.
Ubuntu
mkdir -p /models
wget -O /models/Qwen3-8B-Q4_K_M.gguf \
"https://huggingface.co/Qwen/Qwen3-8B-GGUF/resolve/main/Qwen3-8B-Q4_K_M.gguf"
Windows PowerShell
New-Item -ItemType Directory -Force C:\models
Invoke-WebRequest `
-Uri "https://huggingface.co/Qwen/Qwen3-8B-GGUF/resolve/main/Qwen3-8B-Q4_K_M.gguf" `
-OutFile "C:\models\Qwen3-8B-Q4_K_M.gguf"
Both systems now have the same model file, stored under /models on Ubuntu and
C:\models on Windows.
5. Start the Model Server
With the GGUF downloaded, start llama-server and point it at the model. The command differs slightly
between Ubuntu and Windows because of the executable location and path syntax.
Ubuntu
./build/bin/llama-server \
-m /models/Qwen3-8B-Q4_K_M.gguf \
-c 4096 \
-t 4 \
-ngl 0 \
--host 127.0.0.1 \
--port 1111
Windows PowerShell
.\build\bin\Release\llama-server.exe `
-m C:\models\Qwen3-8B-Q4_K_M.gguf `
-c 4096 `
-t 4 `
-ngl 0 `
--host 127.0.0.1 `
--port 1111
The exact values are starting points, not universal truths. We will tune them in Part 2.
6. Verify the Server Before Debugging Aider
curl http://127.0.0.1:1111/v1/models
✅ Decision Rule
If the server does not answer here, fix llama.cpp first. Don’t start debugging Aider yet.
7. Connect Aider
The important detail is the provider prefix. Aider should connect to the OpenAI-compatible server; it should not be given the GGUF filesystem path as though Aider itself were the model runtime.
export OPENAI_API_BASE=http://127.0.0.1:1111/v1 export OPENAI_API_KEY=local aider --model openai/Qwen3-8B-Q4_K_M.gguf
On Windows PowerShell, set the equivalent environment variables and launch Aider with the same
openai/... model naming pattern.
🎯 Part 1 Outcome
We have crossed the first boundary: a local GGUF is running behind llama-server, and Aider is talking to that server. The machine is now a local coding workstation.
Coming Soon
Part 2 — Make It Fast
