Building a RAG Pipeline from Scratch with ChromaDB and LLaMA 3.2

Retrieval-Augmented Generation (RAG) is the difference between a chatbot that sounds confident and one that is actually correct. Instead of hoping a model memorized your documentation, you retrieve the relevant passages at query time and feed them to the model as grounded context. In this guide we walk through the exact local-first pipeline we ship for clients — built on ChromaDB and LLaMA 3.2 running through Ollama — with citations and zero per-token API cost.

Why build RAG yourself?

Hosted RAG services are fast to start but slow to control. When you own the pipeline you control chunking, embedding quality, retrieval ranking, and prompt assembly — the four levers that actually move accuracy. A local stack also keeps sensitive documents on your own infrastructure, which matters for healthcare, legal, and fintech clients. This is the same philosophy behind our Local AI vs. API comparison.

The architecture at a glance

A minimal but production-shaped RAG pipeline has five stages:

• Ingest — load documents (PDF, Markdown, HTML) and normalize to clean text.
• Chunk — split text into overlapping, semantically coherent passages.
• Embed — convert each chunk into a vector with an embedding model.
• Retrieve — embed the user query and pull the nearest chunks from the vector store.
• Generate — assemble a grounded prompt and let the LLM answer with citations.

Step 1: Chunking and embeddings

Chunk size is the single most under-rated knob. Too large and retrieval gets noisy; too small and you lose context. We start at ~800 characters with a 100-character overlap and tune from there. For embeddings we use a sentence-transformers model so nothing leaves the machine.

from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")

def chunk(text: str, size: int = 800, overlap: int = 100):
    step = size - overlap
    return [text[i:i + size] for i in range(0, len(text), step)]

chunks = chunk(document_text)
vectors = embedder.encode(chunks).tolist()

Step 2: Storing vectors in ChromaDB

ChromaDB gives you a persistent, embedded vector store with almost no ceremony. Store the chunk text and a source reference in metadata so you can cite it later.

import chromadb

client = chromadb.PersistentClient(path="./vectordb")
collection = client.get_or_create_collection("docs")

collection.add(
    ids=[f"chunk-{i}" for i in range(len(chunks))],
    documents=chunks,
    embeddings=vectors,
    metadatas=[{"source": "handbook.pdf", "chunk": i} for i in range(len(chunks))],
)

Step 3: Retrieval + generation with LLaMA 3.2

At query time, embed the question with the same model, pull the top matches, and assemble a prompt that explicitly tells the model to answer only from the supplied context.

import ollama

q_vec = embedder.encode([question]).tolist()
hits = collection.query(query_embeddings=q_vec, n_results=4)
context = "\n\n".join(hits["documents"][0])

prompt = f"""Answer using ONLY the context below. Cite sources by number.
Context:
{context}

Question: {question}"""

resp = ollama.chat(model="llama3.2", messages=[{"role": "user", "content": prompt}])
print(resp["message"]["content"])

Getting citations right

Citations are what make RAG trustworthy. Because every chunk carries a source in its metadata, you can render footnotes that link back to the original document and page. If a claim has no supporting chunk, instruct the model to say so rather than guess — hallucinations drop sharply once “I don’t know” is an allowed answer.

Performance and cost notes

• Cache query embeddings; repeated questions should never re-embed.
• Re-rank the top 20 candidates down to 4 with a cross-encoder when precision matters.
• Running LLaMA 3.2 locally means your only cost is electricity — no per-token billing as traffic scales.

Key takeaways

• Own the four levers: chunking, embeddings, retrieval, prompt.
• Store source metadata from day one so citations are free.
• Local models remove API cost and keep data in-house.

Want a grounded, private RAG assistant trained on your own knowledge base? That is core to our AI & Machine Learning practice — talk to our engineers about a pilot.