Skip to content

Add Ollama-backed LLM and embedding providers #19

Description

@admin-derinworks

Support running the RAG service against a local model served by Ollama, so retrieval becomes semantic and POST /answer returns real generated answers instead of the mock echo — without giving up the dependency-free default build.

The EmbeddingProvider and LlmProvider traits (#5) were designed for exactly this: an Ollama backend is a drop-in implementation of each, with no changes to the retriever, generator, or handlers.

Requirements

  • OllamaLlmProvider implementing LlmProvider against POST /api/generate ("stream": false), returning the response field
  • OllamaEmbeddingProvider implementing EmbeddingProvider against POST /api/embed, batching inputs in one call ("input": [...]"embeddings": [[...]])
  • Gate both behind an optional ollama cargo feature that pulls in an HTTP client (e.g. reqwest), keeping the default build dependency-light
  • CLI wiring: serve --llm ollama --llm-model <name> --embed-model <name> [--ollama-url <url>] replacing the hardcoded mocks when the feature is enabled
  • Surface backend failures as LlmError::Provider / EmbeddingError::Provider so the existing HTTP error mapping (502 for backend failures) applies unchanged
  • Document setup in the README (ollama pull, choosing an embedding model, min_score tuning now that scores are real cosine similarities)

Notes

  • One embedding model per index: the vector store pins dimensionality on first insert (e.g. nomic-embed-text is 768-dim), so index and query must embed with the same model.
  • Integration tests should run against a live Ollama only when one is reachable and skip otherwise; the mock-based tests remain the CI gate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions