The technology behind local AI.

Ubuntu Server LTS (22.04 / 24.04)

For AI workloads, stability matters more than the newest feature. Ubuntu LTS has the driver support and long lifecycle that fit infrastructure intended to run for years.

NVIDIA driver + CUDA + cuDNN — matched

The most common cause of issues in local AI stacks is a mismatch between the driver, CUDA version and the version expected by PyTorch, vLLM or llama.cpp. We define one verified combination and pin it.

By default, we enable nvidia-persistenced to avoid driver startup latency during inference calls.

vLLM for throughput, llama.cpp for flexibility

vLLM is suitable for many parallel requests, continuous batching and tensor parallelism across multiple GPUs. llama.cpp is strong for GGUF models, flexible quantisation and smaller servers.

Open models selected per task

For Dutch documents and translation, Qwen, Mistral and Gemma often work well. The choice depends on context length, VRAM, desired latency and output quality.

For speech we use Whisper, where regional Dutch audio can sometimes perform better on a different model or fine-tune than standard benchmarks suggest.

nginx reverse proxy + Python/Node.js services

Inference-servers staan niet direct publiek. nginx regelt TLS, routing, buffering en streaming. Daarachter draaien FastAPI, Flask of Node.js services.

Where public reachability is needed, we prefer tunnels or reverse proxies over open ports.

systemd services with restart policies

Every component gets its own systemd service with clear dependencies, restart conditions and logging. After a reboot, the stack comes back in the right order.

Small details make a big difference

Checking PCIe lanes, stabilising GPU clocks, tuning batch size to real VRAM usage, setting watchdogs and determining hallucination thresholds for transcription. These are project details that make the difference between a demo and production.