Ollama RAG Guide

Build RAG systems with Ollama - run locally or use cloud for massive models.

Ollama Cloud Models (Dec 2025)

Access via ollama signin (v0.12+). No local storage needed, privacy preserved.

Model	Params	Context	Best For
`deepseek-v3.2:cloud`	671B	160K	GPT-5 level, reasoning
`deepseek-v3.1:671b-cloud`	671B	160K	Thinking + non-thinking hybrid
`qwen3-coder:480b-cloud`	480B	256K-1M	Agentic coding, repo-scale
`minimax-m2:cloud`	230B (10B active)	128K	#1 open-source, tools
`gpt-oss:120b-cloud`	120B	128K	OpenAI open weights
`glm-4.6:cloud`	-	-	Code generation

# Sign in to access cloud
ollama signin

# Run cloud models
ollama run deepseek-v3.2:cloud
ollama run qwen3-coder:480b-cloud
ollama run minimax-m2:cloud

Local Models (Dec 2025)

Reasoning Models

Model	Params	Context	Best For
`nemotron-3-nano`	30B (3.6B active)	1M tokens	Agents, long docs, code
`deepseek-r1`	7B-671B	128K	Reasoning, math, code
`qwq`	32B	32K	Logic, analysis
`llama4`	109B/400B	128K	General, multimodal

Fast/Efficient Models

Model	Size	RAM	Speed
`llama3.2:3b`	2GB	8GB	Very fast
`mistral-small-3.1`	24B	16GB	Fast
`gemma3`	4B-27B	8-32GB	Balanced

Embedding Models

Model	Dims	Context	MTEB Score
`snowflake-arctic-embed2`	1024	8K	67.5
`mxbai-embed-large`	1024	512	64.68
`nomic-embed-text`	768	8K	53.01

Recommendation: snowflake-arctic-embed2 for accuracy, nomic-embed-text for speed.

Quick Start

Cloud (No Local Resources)

ollama signin
ollama run deepseek-v3.2:cloud  # GPT-5 level
ollama run qwen3-coder:480b-cloud  # 1M context for huge repos

Local

ollama pull nemotron-3-nano  # 1M context, 24GB VRAM
ollama pull snowflake-arctic-embed2

# Or for lower RAM (8GB)
ollama pull llama3.2:3b
ollama pull nomic-embed-text

Stack Options

Option A: LangChain + ChromaDB (Most Common)

from langchain_ollama import OllamaLLM, OllamaEmbeddings
from langchain_chroma import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFLoader

# Load and split
loader = PyPDFLoader("document.pdf")
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
docs = splitter.split_documents(loader.load())

# Embed and store
embeddings = OllamaEmbeddings(model="snowflake-arctic-embed2")
vectorstore = Chroma.from_documents(docs, embeddings, persist_directory="./db")

# Query - LOCAL
llm = OllamaLLM(model="nemotron-3-nano")

# Or CLOUD (GPT-5 level, no local resources)
llm = OllamaLLM(model="deepseek-v3.2:cloud")

retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
answer = qa.invoke("What is the main topic?")

Option B: LlamaIndex (Better Accuracy)

from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings

# Configure
Settings.llm = Ollama(model="nemotron-3-nano", request_timeout=300.0)
Settings.embed_model = OllamaEmbedding(model_name="snowflake-arctic-embed2")

# Load and index
documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)

# Query
query_engine = index.as_query_engine()
response = query_engine.query("Summarize the key findings")

Option C: Direct Ollama API (Minimal Dependencies)

import ollama
import chromadb

# Embed
def embed(text):
    return ollama.embed(model="nomic-embed-text", input=text)["embeddings"][0]

# Store in ChromaDB
client = chromadb.PersistentClient(path="./db")
collection = client.get_or_create_collection("docs")
collection.add(ids=["1"], documents=["text"], embeddings=[embed("text")])

# Retrieve and generate
results = collection.query(query_embeddings=[embed("query")], n_results=3)
context = "\n".join(results["documents"][0])

response = ollama.chat(
    model="nemotron-3-nano",
    messages=[{"role": "user", "content": f"Context:\n{context}\n\nQuestion: ..."}]
)

Vector Database Options

Database	Install	Best For
ChromaDB	`pip install chromadb`	Simple, embedded
FAISS	`pip install faiss-cpu`	Fast similarity
Qdrant	`pip install qdrant-client`	Production scale
Weaviate	Docker	Full-featured

Nemotron 3 Nano Deep Dive

Why Nemotron for RAG:

1M token context = entire codebases, long documents
Hybrid Mamba-Transformer = 4x faster inference
MoE (3.6B active params) = runs on 24GB VRAM
Apache 2.0 license = commercial use OK

# For very long documents
llm = OllamaLLM(
    model="nemotron-3-nano",
    num_ctx=131072,  # 128K context, increase as needed
    temperature=0.1,  # Lower for factual RAG
)

Hardware Requirements

Model	RAM	GPU VRAM
3B models	8GB	4GB
7-8B models	16GB	8GB
30B models	32GB	24GB
70B+ models	64GB+	48GB+

ollama-rag