VisionSeek AI

Semantic Image Retrieval for Custom Datasets

VisionSeek AI is a Content-Based Image Retrieval (CBIR) system that retrieves visually relevant images using semantic class keywords, reference images, or a combination of both. The backend leverages SigLIP, FAISS, and FastAPI to generate embeddings and perform efficient vector search, while the React frontend provides a modern, responsive user interface.

---

Features

Search

• Text Search — retrieve images using a semantic class keyword (for example: cat, dog, or car).
• Image Search — find visually similar images using a reference image.
• Hybrid Search — combine text and image queries with an adjustable image weight.

Retrieval Engine

• SigLIP Embeddings — generate unified image and text embeddings using Google's SigLIP vision-language model.
• FAISS Vector Search — perform fast cosine-similarity search over large embedding collections.
• Threshold Filtering — filter low-confidence results with configurable similarity thresholds for each search mode.

Dataset Management

• Folder Upload — import an entire image dataset directly from the browser.
• Append / Replace Mode — add new images to the existing dataset or rebuild it from scratch.
• Automatic Processing — compress, upload, extract, validate images, generate embeddings, rebuild the FAISS index, and reload the search engine automatically after each upload.

Developer Experience

• REST API — clean FastAPI endpoints with interactive Swagger (/docs) and ReDoc (/redoc) documentation.
• Responsive Interface — modern dark-themed dashboard optimized for desktop and mobile devices.

---

Demo Video

Watch the full demonstration on YouTube:

Tech Stack

Backend

Technology	Purpose
Python 3.12	Runtime
FastAPI	REST framework
Uvicorn	ASGI server
SigLIP google/siglip-so400m-patch14-384	Image & text embeddings
FAISS IndexFlatIP	Vector similarity search
Pillow	Image loading & validation
NumPy	Array operations
Pydantic	Request/response validation
pydantic-settings	Environment-based configuration
transformers	HuggingFace model loading
torch	PyTorch backend for SigLIP

Frontend

Technology	Purpose
React 19	UI library
Vite 6	Build tool & dev server
JSZip	Browser-side ZIP creation
Fetch API	HTTP client (no external library)
CSS (vanilla)	Custom dark theme, no framework

---

Project Structure

visionseek-ai/
├── backend/
│   ├── app/
│   │   ├── api/
│   │   │   └── routes/        # health, search, index, dataset
│   │   ├── config/            # pydantic-settings
│   │   ├── indexes/           # FAISS wrapper
│   │   ├── models/            # SigLIP encoder
│   │   ├── schemas/           # Pydantic models
│   │   ├── search/            # SearchEngine orchestrator
│   │   ├── services/          # Business logic layer
│   │   ├── utils/             # image loading, logging, viz
│   │   └── main.py            # FastAPI entry point
│   ├── scripts/               # CLI tools (index, search)
│   ├── tests/                 # test stubs
│   ├── data/
│   │   ├── images/            # dataset (gitignored)
│   │   └── faiss/             # index + metadata (gitignored)
│   ├── Dockerfile
│   └── requirements.txt
│
├── frontend/
│   ├── src/
│   │   ├── api/               # API client
│   │   ├── components/        # Reusable UI
│   │   ├── hooks/             # Custom hooks
│   │   ├── pages/             # Home, Dataset, About
│   │   └── styles/            # global.css
│   ├── index.html
│   ├── vite.config.js
│   └── package.json
│
├── docker-compose.yml
├── Makefile
└── README.md

---

Requirements

• Python 3.12+
• Node.js 20+ and npm
• uv (recommended) or pip

---

Installation

git clone <repo-url>
cd visionseek-ai

Backend

cd backend

# Using uv (recommended)
uv sync

# Alternatively, using pip
python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Note: uv sync reads requirements.txt and creates a virtual environment automatically. If you use pip, activate the virtual environment before running any commands.

Frontend

cd frontend
npm install

---

Running the Project

Option 1 — Makefile

Command	Description
make install	Install backend Python dependencies
make api	Start the FastAPI server on port 8000
make frontend	Start the React dev server on port 5173
make index	Build the FAISS index from data/images/ via CLI
make search QUERY="dog" TOP_K=5	Search by text from the command line
make clean	Remove generated index files and __pycache__ directories

# Terminal 1
make api

# Terminal 2
make frontend

Open http://localhost:5173 in your browser.

Option 2 — Direct commands

# Backend (from backend/)
uv run uvicorn app.main:app --reload
# or with pip: uvicorn app.main:app --reload

# Frontend (from frontend/)
npm run dev

Option 3 — Docker

docker compose up --build

Backend on localhost:8000, frontend on localhost:5173.

Option 4 — CLI scripts

cd backend

# Index images
uv run python -m scripts.cli index

# Search by text
uv run python -m scripts.cli search --text "dog" --top_k 5

# Search by image
uv run python -m scripts.cli search --image path/to/photo.jpg --top_k 10

---

Fast Test Dataset

For a quick end-to-end test, download the curated sample images and index them:

make download-dataset
make index

The sample contains these class labels: dog, cat, pizza, car, and flower. Use one of those class names as the text-search query. For example:

make search QUERY="dog" TOP_K=5

Do not use a long image description for this test dataset; use the category keyword that represents the object class.

---

Dataset Workflow

The Dataset Management page guides you through importing images in three steps.

1. Select Dataset

Click Select Dataset Folder and choose any folder from your computer. The browser scans the folder and displays the number of supported images (.jpg, .jpeg, .png, .bmp, .webp), unsupported files, and total size.

2. Choose Mode

• Append — add the new images to the existing dataset (existing images are kept).
• Replace — delete the current dataset on the server before importing. Only the new images remain.

3. Upload

Click Upload & Process. A single click triggers the full pipeline automatically:

1. Compress — the browser creates a ZIP archive preserving the folder structure (JSZip).
2. Upload — the ZIP is sent to POST /dataset/upload-and-index with the chosen mode.
3. Extract — the backend extracts the archive into data/images/.
4. Validate — every file is checked against the supported extensions and verified as a valid image with Pillow. Unsupported or corrupt files are reported individually without failing the batch.
5. Build embeddings — SigLIP generates embeddings for every valid image.
6. Build index — a new FAISS index is created and saved to disk.
7. Reload — the search engine loads the updated index into memory.

After completion, a success card shows upload statistics (uploaded, skipped, invalid, errors). Click Done to import another dataset.

---

Search Workflow

The Home page offers three search modes. All modes return results sorted by descending cosine similarity, filtered by a configurable threshold.

Text Search

Use the image class name as the query, not a long description of the image. For the included test dataset, enter the exact semantic category such as cat, dog, car, pizza, or flower. These class keywords are the intended way to validate text retrieval.

Example: use dog instead of a happy brown dog running in a sunny park. The latter is an image caption and is not the expected test query for this class-based dataset.

SigLIP encodes the class keyword into a 768-dimensional L2-normalized embedding. FAISS inner-product search finds the nearest neighbours. Results below the text threshold (default 0.08) are discarded.

Image Search

Upload a query image. SigLIP encodes it into a normalized embedding. FAISS retrieves nearest neighbours. Results below the image threshold (default 0.50) are discarded.

Hybrid Search

Provide both text and an image. The two embeddings are combined as a weighted average controlled by image_weight (default 0.70, weighted toward the image). The combined vector is normalized and searched. Results below the hybrid threshold (default 0.30) are discarded.

Click any result card to view the image in a full-screen overlay.

---

API Overview

Interactive API documentation is available through Swagger (/docs) and ReDoc (/redoc) once the backend is running.

System

Method	Endpoint	Description
GET	/	API status
GET	/health	Health check
GET	/stats	Engine statistics

Search

Method	Endpoint	Description
POST	/search/text	Search by text query
POST	/search/image	Search by reference image
POST	/search/hybrid	Combined text and image search

Index Management

Method	Endpoint	Description
POST	/index/build	Build the FAISS index
POST	/index/reload	Reload the index into memory

Dataset

Method	Endpoint	Description
POST	/dataset/upload-and-index	Upload, validate, index, and reload a dataset

Note: Complete request/response schemas, parameters, and example payloads are available in the interactive API documentation at /docs.

---

Configuration

Application settings are loaded from backend/.env using pydantic-settings.

All configurable options, default values, and descriptions are documented directly in the .env file. Simply copy .env.example (or edit .env) to customize the application.

---

License

This project is licensed under the MIT License.

---

Operational safety

Dataset updates are processed as a single POST /dataset/upload-and-index operation. The backend validates ZIP archives, applies configurable upload limits, builds the FAISS index, and reloads it while holding an in-process update lock. In replace mode, the prior dataset is retained and restored if indexing fails.

Configure these values in backend/.env (copy from .env.example):

CORS_ORIGINS=http://localhost:5173,http://127.0.0.1:5173
MAX_UPLOAD_BYTES=52428800
MAX_ZIP_FILES=5000
MAX_ZIP_UNCOMPRESSED_BYTES=524288000
MAX_IMAGE_PIXELS=40000000

CORS_ORIGINS must list the browser origins that may call the API. Set limits according to the capacity of the deployment; public deployments should additionally add authentication and reverse-proxy request limits.

Tests

cd backend
uv sync --group dev
uv run pytest

The tests use lightweight fixtures and do not download the SigLIP model.

GitHub Codespaces

When the frontend runs on a GitHub Codespaces forwarded port, its browser cannot use http://localhost:8000 as the API URL. Configure the two forwarded HTTPS URLs and restart both development servers:

# frontend/.env
VITE_API_BASE_URL=https://YOUR-CODESPACE-8000.app.github.dev

# backend/.env
CORS_ORIGINS=https://YOUR-CODESPACE-5173.app.github.dev,http://localhost:5173,http://127.0.0.1:5173

Then run the applications in separate terminals:

make backend
make frontend

Verify the API at https://YOUR-CODESPACE-8000.app.github.dev/health. Public forwarded ports are convenient for testing, but the backend exposes dataset and index management endpoints without authentication; switch port 8000 back to Private after testing or add authentication before deployment.