KAMCO Data Collector

An integrated solution for collecting, normalizing, and analyzing KAMCO (Korea Asset Management Corporation) public auction data using RAG (Retrieval-Augmented Generation).

---

📌 프로젝트 개요

캠코(KAMCO) 공매 물건 데이터를 수집·정규화·벡터화하여 RAG(검색 증강 생성) 기반 AI 챗봇으로 자연어 검색과 질의응답을 제공하는 통합 데이터 파이프라인입니다. 공공데이터포털의 캠코 공매 OpenAPI에서 원시 데이터를 수집해 MongoDB에 저장하고, Qdrant 벡터 DB에 임베딩한 뒤 FastAPI + Ollama(로컬 LLM)로 "서울 강남 아파트 5억 이하" 같은 질문에 출처까지 포함한 답변을 생성합니다. 벡터 DB가 없어도 MongoDB 키워드 검색으로 자동 폴백(fallback)되는 하이브리드 검색 구조를 갖습니다.

🧰 기술 스택

분류	기술	용도
언어	Python 3.9+	전체 백엔드
API 서버	FastAPI 0.115+	RAG 질의응답 REST API (/ask, /api/chat)
웹 UI	Flask 3.0+	관리 대시보드 · 데이터 조회 · AI 챗봇 웹 인터페이스
문서 DB	MongoDB 7.0+	원시/정규화/청크 데이터 저장 + 키워드 폴백 검색
벡터 DB	Qdrant 1.9+	임베딩 유사도 검색 (Top-k)
LLM	Ollama (deepseek-r1 생성 · nomic-embed-text 임베딩)	로컬 LLM 추론
에이전트 연동	MCP Server	ChatGPT 데스크톱에서 직접 검색/수집/질의
배포	Docker / Docker Compose	멀티 컨테이너 오케스트레이션
터널 (선택)	Cloudflare Tunnel / Workers	외부 접근 노출

📸 Screenshots

Dashboard

Main dashboard showing data statistics and quick actions

Data Collection Interface

KAMCO API data collection interface with real-time progress

Data List View

Browse and search collected auction data

Detailed View

Detailed property information with all auction details

AI Chatbot Interface

RAG-based AI chatbot for intelligent property search and Q&A

✨ Features

• 🔍 Automated data collection from KAMCO OpenAPI
• 📊 Data normalization and structuring
• 🤖 RAG-based AI Chatbot with intelligent search and Q&A
• 🔄 Hybrid search: Vector (Qdrant) + Keyword (MongoDB) fallback
• 🌐 FastAPI RESTful endpoints
• 💾 MongoDB + Qdrant vector database
• 🧠 Ollama local LLM integration (deepseek-r1, nomic-embed-text)
• 🖥️ Web-based management interface with real-time stats
• 🔌 MCP (Model Context Protocol) server for ChatGPT integration
• 🐳 Docker deployment ready

🏗️ Architecture

[ChatGPT Desktop]
        ↓ (MCP Protocol)
[MCP Server] ←→ [FastAPI Server]
     ↓              ↓
[MongoDB] ←→ [Qdrant] ←→ [Ollama]
     ↓
[KAMCO OpenAPI]

Optional:
[Cloudflare Front]
        ↓
[Cloudflare Workers]
        ↓ (Tunnel)
[Mac mini / Server]

📋 Prerequisites

1. MongoDB 7.0

brew tap mongodb/brew
brew install mongodb-community@7.0
brew services start mongodb-community

2. Ollama

brew install ollama

# Pull required models for RAG chatbot
ollama pull nomic-embed-text:latest     # Embedding model
ollama pull deepseek-r1:latest          # LLM for generation

3. Qdrant

docker run -d \
  --name qdrant \
  -p 6333:6333 \
  -v ~/qdrant:/qdrant/storage \
  qdrant/qdrant

Database Schema

kamco
 ├─ raw_items          # Raw API responses
 ├─ normalized_items   # Normalized text data
 └─ chunks             # Chunked data for RAG (matches Qdrant payload)

🚀 Quick Start

Option 1: Docker (Recommended)

# Clone repository
git clone https://github.com/lhg96/kamco-data-collector.git
cd kamco-data-collector

# Configure environment
cp .env.example .env
nano .env  # Edit with your KAMCO API key

# Start all services
docker-compose up -d

# Pull Ollama model
docker exec -it kamco-ollama ollama pull qwen2.5:latest

# Collect and process data
docker exec -it kamco-mcp-server python -m rag.manager

Option 2: Manual Installation

git clone https://github.com/lhg96/kamco-data-collector.git
cd kamco-data-collector
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -r requirements.txt

⚙️ Configuration

Create a .env file in the project root:

# KAMCO API Key (get from https://www.data.go.kr/)
KAMCO_API_KEY=your_api_key_here

# MongoDB Configuration
MONGO_URI=mongodb://localhost:27017
MONGO_DB_NAME=kamco
MONGO_COLLECTION_NAME=collected_items

# Qdrant Configuration
QDRANT_HOST=localhost
QDRANT_PORT=6333
QDRANT_COLLECTION=kamco

# Ollama Models
EMBED_MODEL=nomic-embed-text:latest     # For embeddings
LLM_MODEL=deepseek-r1:latest           # For generation

# Flask Configuration
FLASK_SECRET_KEY=your_random_secret_key_here
FLASK_PORT=5001                        # Default: 5000 (avoid conflicts with macOS Control Center)

📖 Usage

MCP Server for ChatGPT (Recommended)

The MCP server allows ChatGPT to directly interact with KAMCO data.

1. Start Services

docker-compose up -d

2. Collect and Process Data

# Using RAG Manager (all-in-one)
docker exec -it kamco-mcp-server python -m rag.manager

# Or step by step:
docker exec -it kamco-mcp-server python -m services.kamco_collector_service
docker exec -it kamco-mcp-server python -m normalize.kamco_normalizer
docker exec -it kamco-mcp-server python -m rag.embed

3. Configure ChatGPT

See MCP_SETUP_GUIDE.md for detailed instructions.

Available MCP Tools:

• search_kamco - Search auctions by natural language
• get_kamco_by_id - Get detailed item information
• get_recent_kamco - Get recent listings
• ask_kamco - Ask questions with RAG answers
• collect_kamco_data - Trigger data collection
• embed_kamco_data - Process and embed data

Data Collection Pipeline (Manual)

1. Collect Data

Fetch raw data from KAMCO OpenAPI and store in MongoDB:

python collector/kamco_fetcher.py
# Or use the service:
python -m services.kamco_collector_service

2. Normalize Data

Convert raw data to human-readable text format:

python normalize/kamco_normalizer.py

3. Embed Data

Generate embeddings and store in Qdrant vector database:

python rag/embed.py

⚠️ Note: setup_collection() recreates the collection, deleting existing data.

4. Start RAG API Server

uvicorn api.main:app --host 0.0.0.0 --port 8000
# Or with Docker:
docker-compose up -d api

Available endpoints:

• GET /ask?q=your_question - RAG-based question answering (Top-5 vector search)
• GET /health - Health check

Web Interface

Start the Flask web application:

python web/app.py

Access at http://localhost:5001

Features:

• 📊 Real-time data collection monitoring
• 🔍 Advanced search with filters
• 📝 Detailed property information view
• 🤖 AI Chatbot - RAG-based intelligent Q&A system
• 📈 Dashboard with statistics

AI Chatbot Features

The web interface includes an AI chatbot powered by RAG (Retrieval-Augmented Generation) technology:

Key Features:

• 🧠 Natural Language Search: Ask questions in natural Korean
• 🔍 Intelligent Query Processing: Automatically detects search intent (region, price, property type)
• 📊 Hybrid Search Mode:
  • Primary: Vector-based semantic search with Qdrant + Ollama
  • Fallback: MongoDB keyword/regex search when vector DB is unavailable
• 🎯 Context-Aware Responses: Recent data queries automatically pull latest collections
• 📎 Source Citations: All answers include source documents with relevance scores
• 🔗 Direct Links: Includes original KAMCO auction URLs in responses

Query Examples:

"최근 수집된 데이터 보여줘"
→ Shows 5 most recent items with details

"서울 강남 아파트 5억 이하"
→ Searches for apartments in Gangnam, Seoul under 500M won

"이 물건의 감정가는 얼마인가요?"
→ RAG-based answer with specific property details

Technical Implementation:

1. RAG Mode (When Qdrant + Ollama Available):

• Embeds user question using nomic-embed-text model
• Performs vector similarity search in Qdrant (top-3)
• Generates natural language answer using deepseek-r1 LLM
• Returns answer with source documents and confidence scores

2. Fallback Mode (MongoDB Only):

• Uses Ollama to analyze query intent (JSON extraction)
• Constructs MongoDB query from extracted parameters:
  • Region: basic_info.lctnAddr regex match
  • Property type: basic_info.pblancObjSe regex match
  • Price range: Numerical comparison
  • Keywords: Multi-field OR search
• Returns structured results with formatted descriptions

3. Recent Data Optimization:

• Detects keywords: '최근', '최신', '수집', '목록'
• Bypasses RAG for direct MongoDB lookup
• Sorts by collected_at timestamp (descending)
• Faster response time for common queries

API Endpoint:

POST /api/chat
Content-Type: application/json

{
  "question": "서울 강남구 아파트 매물 보여줘"
}

Response:
{
  "success": true,
  "answer": "서울 강남구 아파트 검색 결과입니다:\n\n1. ...",
  "sources": [
    {
      "text": "공고명: ...\n원문 URL: https://...",
      "score": 0.95
    }
  ],
  "fallback": false  // true if using MongoDB fallback
}

Dependencies:

• Qdrant: Vector database for embeddings (optimal)
• Ollama: Local LLM for embeddings + generation (optimal)
• MongoDB: Document store + fallback search (required)

💡 Tip: For best results, ensure Qdrant and Ollama are running. The system gracefully falls back to MongoDB-only mode if vector services are unavailable.

⚡ 실행 예시

1. RAG 질의응답 API (FastAPI)

# 서버 기동
uvicorn api.main:app --host 0.0.0.0 --port 8000

# 자연어 질문 → Top-5 벡터 검색 기반 답변
curl "http://localhost:8000/ask?q=서울%20강남구%20아파트%20매물"

# 헬스 체크
curl "http://localhost:8000/health"
# → {"status": "ok"}

2. 웹 챗봇 API (Flask)

# Flask 웹 서버 기동 (http://localhost:5001)
python web/app.py

# 챗봇 질문
curl -X POST http://localhost:5001/api/chat \
  -H "Content-Type: application/json" \
  -d '{"question": "최근 수집된 공매 물건 보여줘"}'

{
  "success": true,
  "answer": "최근 수집된 공매 물건 5건입니다:\n1. ...",
  "sources": [
    {
      "text": "공고명: ...\n원문 URL: https://...",
      "score": 0.95
    }
  ],
  "fallback": false
}

3. 수집 → 정규화 → 임베딩 파이프라인

# 1) KAMCO OpenAPI에서 원시 데이터 수집 (MongoDB 저장)
python -m services.kamco_collector_service

# 2) 사람이 읽을 수 있는 텍스트로 정규화
python -m normalize.kamco_normalizer

# 3) 청크 임베딩 생성 → Qdrant 벡터 DB 저장
python -m rag.embed

4. MCP를 통한 ChatGPT 연동 (권장)

docker-compose up -d
docker exec -it kamco-mcp-server python -m rag.manager

ChatGPT 데스크톱에서 search_kamco, ask_kamco, collect_kamco_data 등 MCP 도구를 직접 호출할 수 있습니다. 자세한 설정은 MCP_SETUP_GUIDE.md를 참고하세요.

🔧 Cloudflare Tunnel Setup (Optional)

For external access via Cloudflare Workers:

cloudflared tunnel create kamco-rag
cloudflared tunnel run kamco-rag

Configure Workers to proxy /ask endpoint to your local FastAPI server.

🧪 Testing

API Integration Test

python test_api.py

Unit Tests

pytest

Test Coverage

pytest --cov=. --cov-report=html

📁 Project Structure

kamco-data-collector/
├── api/
│   └── main.py                # FastAPI RAG endpoints
├── collector/
│   └── kamco_fetcher.py       # KAMCO OpenAPI data collector
├── mcp_server/
│   ├── __init__.py
│   └── server.py              # MCP server for ChatGPT integration
├── normalize/
│   └── kamco_normalizer.py    # Data normalization
├── rag/
│   ├── embed.py               # Embedding & Qdrant operations
│   └── manager.py             # RAG pipeline orchestration
├── services/
│   └── kamco_collector_service.py  # Service layer
├── web/
│   ├── app.py                 # Flask web interface
│   ├── templates/             # HTML templates
│   └── static/                # Static assets
├── tests/                     # Test files
├── screenshots/               # Application screenshots
├── Dockerfile                 # Docker image definition
├── docker-compose.yml         # Multi-container orchestration
├── .dockerignore              # Docker build exclusions
├── requirements.txt           # Python dependencies
├── .env.example               # Example environment variables
├── mcp_config.json            # MCP server configuration
├── MCP_SETUP_GUIDE.md         # MCP setup instructions
├── .gitignore
└── README.md

📚 API Reference

KAMCO OpenAPI

Get your API key from 공공데이터포털 (data.go.kr)

• Search for "한국자산관리공사_공매"
• Register and obtain your service key
• Use the URL-encoded version of the key

📝 PublicDataReader Library - History and Future Plans

Background

During development, we evaluated the PublicDataReader library, which provides a convenient wrapper around various Korean public data APIs, including KAMCO.

Why We Didn't Use PublicDataReader

Advantages of PublicDataReader:

✅ Returns data as pandas DataFrame for easy analysis
✅ Automatic Korean column name conversion
✅ Simple installation via pip
✅ Unified interface for multiple public APIs

Limitations for Our Use Case:

❌ Difficult to access raw data structure
❌ Additional external dependencies
❌ Less control over data processing pipeline
❌ Not suitable for MongoDB storage workflow
❌ Limited customization for RAG pipeline

Our Custom Implementation Benefits:

✅ Direct XML/Dict data processing
✅ Built-in MongoDB storage
✅ Integrated Flask web UI
✅ Duplicate removal logic
✅ Minimal external dependencies
✅ Optimized for RAG pipeline

Future Considerations

We maintain PublicDataReader compatibility for users who prefer DataFrame-based analysis:

# Optional: Use PublicDataReader for analysis
import PublicDataReader as pdr
api = pdr.Kamco(service_key)
df = api.get_data(service="캠코공매물건", function="물건목록")

Recommended Approach:

• Data Collection & Storage: Use our KamcoCollectorService
• Data Analysis: Consider PublicDataReader for exploratory analysis
• Web Interface: Use our Flask application
• RAG Pipeline: Use our custom implementation

For detailed PublicDataReader test results and comparisons, see tests/README_PUBLICDATAREADER.md.

🔮 Future Development Plans

Phase 1: Enhanced Data Collection (Q1 2026)

• Implement incremental updates (delta sync)
• Add support for additional KAMCO API endpoints
• Implement retry logic with exponential backoff
• Add data validation and quality checks

Phase 2: Advanced RAG Features (Q2 2026)

• AI Chatbot with natural language search ✅
• Hybrid search (vector + keyword fallback) ✅
• Recent data query optimization ✅
• Multi-language support (English/Korean)
• Enhanced vector search with re-ranking
• Implement query expansion techniques
• Conversation history and context memory

Phase 3: Production Readiness (Q3 2026)

• Containerization with Docker Compose ✅
• MCP server for ChatGPT integration ✅
• Add monitoring and logging (Prometheus/Grafana)
• Implement API rate limiting
• Add authentication and authorization
• Performance optimization and caching

Phase 4: Advanced Features (Q4 2026)

• Real-time data streaming
• Machine learning-based recommendations
• Advanced analytics dashboard
• Mobile application support
• Integration with external platforms

Development Setup

# Clone the repository
git clone https://github.com/lhg96/kamco-data-collector.git
cd kamco-data-collector

# Create virtual environment
python -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Run tests
pytest

📄 License

Custom License - Free for Personal Use, Commercial License Required

This software is free to use for personal, educational, and non-commercial purposes. Commercial use requires a separate license agreement.

• ✅ Free: Personal use, education, research
• ❌ Requires License: Commercial use, production deployment, integration into commercial products

For commercial licensing inquiries, please contact: hyun.lim@okkorea.net

Data Attribution

Exchange rate data collected from Investing.com and other public financial data sources. KAMCO auction data provided by Korea Asset Management Corporation through public data portal. Accurate as of December 2024.

📞 Contact & Services

💼 Development Consulting & Outsourcing Available

We provide professional consulting and development services for IoT, AI, and embedded systems projects.

👨‍💼 Project Manager Contact

• Email: hyun.lim@okkorea.net
• Homepage: https://www.okkorea.net
• LinkedIn: https://www.linkedin.com/in/aionlabs/

🛠️ Technical Expertise / 기술 전문 분야

• IoT System Design and Development / IoT 시스템 설계 및 개발
• Embedded Software Development / 임베디드 소프트웨어 개발 (Arduino, ESP32)
• AI Service Development / AI 서비스 개발 (LLM, MCP Agent)
• Cloud Service Architecture / 클라우드 서비스 구축 (Google Cloud Platform)
• Hardware Prototyping / 하드웨어 프로토타이핑

💼 Services / 서비스

Technical Consulting / 기술 컨설팅

• IoT project planning and design consultation / IoT 프로젝트 기획 및 설계 자문
• System architecture design / 시스템 아키텍처 설계
• Technology stack selection / 기술 스택 선정

Development Outsourcing / 개발 외주

• Full-stack development from firmware to cloud / 펌웨어부터 클라우드까지 Full-stack 개발
• Proof of Concept (PoC) development / 개념 검증 개발
• Production-ready system development / 상용 시스템 개발

🙏 Acknowledgments

• KAMCO (Korea Asset Management Corporation) for providing public auction data
• 공공데이터포털 for API access
• PublicDataReader by WooilJeong for Python library inspiration
• All contributors and users of this project

---

Made with ❤️ by OK Korea