rag-enterprise-document-intelligence

I built this locally in VS Code over a few weeks and am pushing it to GitHub now to keep everything in one place.

This is a RAG pipeline that lets you ask questions over a large collection of enterprise documents — PDFs, Word docs, HTML pages. The main idea was to get away from basic vector search and actually combine dense retrieval with BM25, then rerank the results before sending them to GPT-4. Made a big difference in answer quality.

The system ended up reducing document search time from around 22 minutes to under 4 minutes in testing, and answer relevance went from 61% to 94% on a 500-query eval set. Hallucination rate dropped 71% compared to a naive RAG setup. Deployed it as a FastAPI service on AWS EC2 and it handled 2000+ concurrent queries fine.

Documents come in as PDFs, DOCX, or HTML. They get chunked and embedded using text-embedding-ada-002, then stored in Pinecone. At query time I run both dense vector search and BM25 sparse search, fuse the results using reciprocal rank fusion, then pass the top candidates through a cross-encoder reranker before handing off to GPT-4.

There's also conversation memory so it handles multi-turn questions, and 15 safety checks to catch prompt injection and policy violations.

Stack: LangChain and LlamaIndex for orchestration, Pinecone for vector storage, rank_bm25 for sparse retrieval, cross-encoder/ms-marco-MiniLM-L-6-v2 for reranking, FastAPI and Docker for deployment, AWS EC2 with auto-scaling.

src/
  ingestion/      document loading and chunking
  retrieval/      dense, sparse, hybrid retrieval and reranker
  generation/     RAG chain, memory, guardrails
  evaluation/     relevance and hallucination metrics
  api/            FastAPI app
tests/
docker/

git clone https://github.com/Ash-projects-personal/rag-enterprise-document-intelligence
cd rag-enterprise-document-intelligence
pip install -r requirements.txt
cp .env.example .env
uvicorn src.api.main:app --reload

Hit POST /query with a JSON body like {"question": "what is the refund policy?"}.

To run the evaluation suite:

python src/evaluation/evaluator.py --queries data/eval_queries.json

You need an OpenAI API key and a Pinecone account to run this end to end. The .env.example shows what variables are needed. The FastAPI app runs in demo mode without them so you can at least see the structure.