Support running the RAG service against a local model served by Ollama, so retrieval becomes semantic and POST /answer returns real generated answers instead of the mock echo — without giving up the dependency-free default build.
The EmbeddingProvider and LlmProvider traits (#5) were designed for exactly this: an Ollama backend is a drop-in implementation of each, with no changes to the retriever, generator, or handlers.
Requirements
OllamaLlmProvider implementing LlmProvider against POST /api/generate ("stream": false), returning the response field
OllamaEmbeddingProvider implementing EmbeddingProvider against POST /api/embed, batching inputs in one call ("input": [...] → "embeddings": [[...]])
- Gate both behind an optional
ollama cargo feature that pulls in an HTTP client (e.g. reqwest), keeping the default build dependency-light
- CLI wiring:
serve --llm ollama --llm-model <name> --embed-model <name> [--ollama-url <url>] replacing the hardcoded mocks when the feature is enabled
- Surface backend failures as
LlmError::Provider / EmbeddingError::Provider so the existing HTTP error mapping (502 for backend failures) applies unchanged
- Document setup in the README (
ollama pull, choosing an embedding model, min_score tuning now that scores are real cosine similarities)
Notes
- One embedding model per index: the vector store pins dimensionality on first insert (e.g.
nomic-embed-text is 768-dim), so index and query must embed with the same model.
- Integration tests should run against a live Ollama only when one is reachable and skip otherwise; the mock-based tests remain the CI gate.
Support running the RAG service against a local model served by Ollama, so retrieval becomes semantic and
POST /answerreturns real generated answers instead of the mock echo — without giving up the dependency-free default build.The
EmbeddingProviderandLlmProvidertraits (#5) were designed for exactly this: an Ollama backend is a drop-in implementation of each, with no changes to the retriever, generator, or handlers.Requirements
OllamaLlmProviderimplementingLlmProvideragainstPOST /api/generate("stream": false), returning theresponsefieldOllamaEmbeddingProviderimplementingEmbeddingProvideragainstPOST /api/embed, batching inputs in one call ("input": [...]→"embeddings": [[...]])ollamacargo feature that pulls in an HTTP client (e.g.reqwest), keeping the default build dependency-lightserve --llm ollama --llm-model <name> --embed-model <name> [--ollama-url <url>]replacing the hardcoded mocks when the feature is enabledLlmError::Provider/EmbeddingError::Providerso the existing HTTP error mapping (502for backend failures) applies unchangedollama pull, choosing an embedding model,min_scoretuning now that scores are real cosine similarities)Notes
nomic-embed-textis 768-dim), so index and query must embed with the same model.