Building a RAG Pipeline with Uteke and Local LLMs

Building a RAG Pipeline with Uteke and Local LLMs

Retrieval-Augmented Generation (RAG) is the standard pattern for giving LLMs access to your own data. Most tutorials point you at cloud vector databases, but what if you want everything running locally? This post walks through building a complete RAG pipeline using Uteke as the memory layer and a local LLM for generation.

Why Local RAG

Running RAG locally gives you privacy — your data never leaves your machine. It also eliminates API costs and latency for every retrieval round-trip. For developer tools, documentation assistants, and internal knowledge bases, local RAG is often the pragmatic choice.

Architecture Overview

The pipeline has four stages: ingest, embed, retrieve, and generate. Uteke handles the first three.

• Ingest: chunk your documents into paragraphs or sections
• Embed: Uteke automatically generates embeddings using its built-in model (all-minilm-L6-v2)
• Retrieve: semantically search memories with the recall endpoint
• Generate: pass retrieved context to your local LLM via Ollama or llama.cpp

Step 1: Start Uteke

Run Uteke with its default settings. The embedded model handles 384-dimensional vectors without any external embedding service.

Install via brew: brew install codecoradev/tap/uteke or download the binary from GitHub Releases.

Step 2: Ingest Your Documents

Break your source material into meaningful chunks. For technical docs, paragraph-level chunks work well. For code, function-level chunks with surrounding context are better. Each chunk becomes a memory in Uteke via the remember endpoint.

A typical ingestion script reads your markdown files, splits them by heading boundaries, and sends each section to Uteke with relevant tags for filtering later.

Step 3: Build the Retrieval Layer

When a user asks a question, embed it with the same model and query Uteke for the top-K most relevant memories. Uteke returns results in milliseconds, so retrieval is never the bottleneck in your pipeline.

For better results, use Uteke's hybrid search mode which combines vector similarity with keyword matching via FTS5. This catches exact term matches that pure vector search sometimes misses.

Step 4: Generate with a Local LLM

Pass the retrieved memories as context to your local LLM. With Ollama, this is a simple API call with a system prompt that includes your retrieved context. The key is keeping the context window focused — include only the top 5-10 most relevant chunks, not everything.

Performance Notes

The retrieval step with Uteke adds roughly 5-10ms to your pipeline. For a local LLM generating 200 tokens, expect 2-8 seconds depending on your hardware. The total latency is dominated by generation, not retrieval — which is exactly what you want from a memory layer.

The full source for this pipeline is available in the Uteke examples repository. Contributions and feedback welcome.