🚨 Real-Time AI Threat Detection & Surveillance System

A real-time AI-powered surveillance system that simultaneously detects violent fights, weapons (firearms, knives, grenades, etc.), and performs facial recognition — all from a single camera feed. Built with a multi-model YOLO pipeline and InsightFace, with a clean Streamlit dashboard for live monitoring and alerts.

This was built as a team exploratory project under Dr. Sanjeev Sharma, focused on exploring how far you can push real-time multi-model inference on consumer hardware without sacrificing accuracy.

---

Table of Contents

• Demo & Screenshots
• What It Actually Does
• System Architecture
• Models & Accuracy
• Performance & Optimizations
• Dataset Details
• Tech Stack
• Getting Started
• Project Structure
• How It Works (in detail)
• Limitations & Future Work
• Acknowledgements

📄 For detailed architecture decisions, alternative models tested, and iteration history, see ARCHITECTURE.md.

---

Demo & Screenshots

The system runs as a Streamlit web app with live video feed, bounding box annotations, and a sidebar alert panel.

Annotation	Meaning
🟥 Red box	Fight / Violence detected
🟧 Orange box	Weapon detected (firearm, knife, etc.)
🟩 Green box	Known face identified
🟥 Red box (face)	Unrecognized face

The sidebar shows real-time status: ✅ Safe, ⚠️ Weapon Warning, or 🚨 Critical Alert depending on what's happening in the frame.

---

What It Actually Does

Most surveillance projects I came across were basically "run YOLO on a video and draw some boxes." I wanted something more practical — a system that could:

1. Detect physical fights as they happen — not just "person detected," but actual violent altercations
2. Identify weapons in the frame — pistols, knives, firearms, grenades, ammo, rockets
3. Recognize faces against a pre-built database — so you know who is involved, not just what is happening
4. Run all three simultaneously on a single video stream without tanking the framerate
5. Alert the operator with a hierarchical alert system (safe → warning → critical)

The tricky part was making all of this work together in real-time. Running three YOLO models + InsightFace on every single frame is computationally brutal, so a good chunk of the engineering effort went into optimizations (more on that below).

---

System Architecture

The backend pipeline processes each frame through multiple detection stages before rendering the annotated output.

High-level flow:

Detailed architecture with all inference paths, thresholds, and caching logic:

The pipeline works like this:

1. Frame Capture — OpenCV grabs a frame from webcam or uploaded video
2. Dynamic Resolution Scaling — All YOLO models run at imgsz=480 for speed; InsightFace gets a 640px-wide downscaled copy
3. Concurrent YOLO Inference — Three models run on every frame:
   • fight_detector_yolo.pt — Custom-trained YOLOv8 for violence/fight detection
   • yolo11n_threat_detection.pt — YOLO11n fine-tuned on weapon classes
   • yolov8n-face.pt — YOLOv8n for face localization
4. InsightFace Recognition — Runs only every 5th frame (the most expensive operation), extracts 512-D ArcFace embeddings using ResNet-100 (buffalo_l), and matches against the known faces database via cosine similarity
5. IoU-Based Identity Tracking — On non-InsightFace frames, YOLO face boxes are matched to cached InsightFace identities using IoU overlap, so recognized names persist smoothly between recognition frames
6. Annotation & Alerts — Bounding boxes are drawn with color-coded labels, and a hierarchical alert system classifies the threat level

---

Models & Accuracy

The system uses three specialized YOLO models running in parallel, each fine-tuned for its specific detection task.

Detection Accuracy Across Models

Model	Task	Precision	Recall	mAP@0.5	mAP@0.5:0.95
YOLOv8 (Custom)	Fight Detection	0.89	0.86	0.92	0.70
YOLO11n	Weapon Detection	0.88	0.82	0.87	0.67
YOLOv8n-face	Face Localization	0.96	0.94	0.94	0.82

Face Recognition uses InsightFace's buffalo_l model (ResNet-100 backbone, ArcFace loss) and matches against the database using cosine similarity with a threshold of 0.4.

Confidence Score Distribution

The plot below shows how confident each model's detections are in practice. Most detections cluster well above the 0.4 confidence threshold, meaning false positives are relatively rare at inference time.

Weapon detection classes: ammo, firearm, grenade, knife, pistol, rocket

---

Performance & Optimizations

Running three YOLO networks + InsightFace on every frame is extremely slow out of the box. The naive approach gives about 1.5 FPS on CPU — completely unusable for real-time surveillance. Here's what I did to fix that:

The Frame-Skip Strategy

The single biggest bottleneck was InsightFace. Extracting 512-D facial embeddings with ResNet-100 is expensive. The key insight: face identities don't change between consecutive frames. So InsightFace only runs every 5th frame, and the results are cached and matched to YOLO face boxes via IoU on the intermediate frames.

A skip interval of 5 was chosen as the sweet spot — going higher starts to cause noticeable identity lag when people move quickly, while lower values don't help CPU performance much.

Throughput Results

Configuration	FPS	Real-Time?
CPU, no optimization	~1.5	❌
CPU + frame-skip	~10.5	⚠️ Usable
GPU (RTX 3060) + frame-skip	~30.0	✅ Yes

Latency Breakdown

Where does the time actually go? Here's the per-model breakdown:

On GPU, total per-frame latency is ~31 ms (about 32 FPS). On CPU it's ~254 ms per frame. InsightFace dominates in both cases — which is exactly why the frame-skip optimization matters so much.

Other Optimizations

• Resolution scaling — YOLO inference runs at 480px instead of full resolution. Barely affects accuracy but significantly reduces computation.
• Streamlit caching — @st.cache_resource ensures models are loaded once and reused across reruns, avoiding the ~15 second cold-start penalty.
• Frame downscaling for InsightFace — Frames wider than 640px are resized before being passed to InsightFace, which is already resolution-tolerant.

---

Dataset Details

The models were trained/fine-tuned on the following datasets:

Category	Training Images	Validation Images
Violence / Physical Fights	12,450	2,100
Weapons (Firearms)	8,200	1,500
Weapons (Knives)	5,100	950
Human Faces	35,000	5,000

Total: ~70,300 images across all categories. The face detection model (YOLOv8n-face) had the largest dataset, which is reflected in its higher accuracy scores.

---

Tech Stack

Component	Technology
Object Detection	YOLOv8, YOLO11n (Ultralytics)
Face Recognition	InsightFace (ArcFace, buffalo_l)
Face Embedding	ResNet-100 (512-D embeddings)
Video Processing	OpenCV
Web Interface	Streamlit
Deep Learning	PyTorch, ONNX Runtime
Language	Python 3.8+

---

Getting Started

Prerequisites

• Python 3.8 or higher
• A webcam (for live mode) or video files (mp4/avi/mov/mkv)
• GPU recommended for real-time performance (CUDA-capable NVIDIA GPU)

Installation

# Clone the repository
git clone https://github.com/yourusername/threat-detection.git
cd threat-detection

# Install dependencies
pip install -r requirements.txt

# InsightFace needs to be installed separately
pip install insightface onnxruntime-gpu  # or onnxruntime for CPU-only

Setting Up the Face Database

To use facial recognition, create a directory structure under known_faces/:

known_faces/
├── person_name_1/
│   ├── photo1.jpg
│   ├── photo2.jpg
│   └── photo3.jpg
├── person_name_2/
│   ├── photo1.jpg
│   └── photo2.jpg
└── ...

The folder name becomes the identity label. Use 3-5 clear frontal photos per person for best results. The system automatically builds the embedding database on startup.

Running the App

streamlit run app.py

This opens a browser dashboard where you can:

• Webcam mode — Start/stop your live camera feed
• Video file mode — Upload and process any video file

---

Project Structure

.
├── app.py                          # Main application (Streamlit + inference pipeline)
├── requirements.txt                # Python dependencies
├── fight_detector_yolo.pt          # YOLOv8 model — fight/violence detection
├── yolo11n_threat_detection.pt     # YOLO11n model — weapon detection (6 classes)
├── yolov8n-face.pt                 # YOLOv8n model — face localization
├── known_faces/                    # Face recognition database
│   └── <person_name>/             # One folder per identity, containing reference photos
├── graphs/                         # Performance & accuracy visualizations
│   ├── accuracy_comparison.png
│   ├── confidence_distribution.png
│   ├── dataset_composition.png
│   ├── fps_comparison.png
│   ├── frameskip_impact.png
│   ├── latency_breakdown.png
│   ├── backend_flowchart.png
│   ├── backend_flowchart_simple.png
│   └── flowchart_minimal.png
└── README.md

---

How It Works (in detail)

Fight Detection

The fight detection model is a custom-trained YOLOv8 that classifies bounding box regions as either "normal" (class 0) or "fight" (class 1). Only class 1 detections with confidence > 0.4 trigger alerts. The model was trained on ~14,500 images of physical altercations and normal human interactions.

Weapon Detection

Uses a YOLO11n model fine-tuned on 6 weapon categories: ammo, firearm, grenade, knife, pistol, and rocket. Detections above 0.4 confidence are drawn as orange bounding boxes with the weapon class label.

Face Recognition Pipeline

This was the most complex part to get right. The pipeline works in two stages:

1. Detection (every frame): YOLOv8n-face localizes all faces in the frame with bounding boxes.
2. Recognition (every 5th frame): InsightFace extracts 512-dimensional ArcFace embeddings from detected faces, then compares them against pre-computed embeddings from the known_faces/ database using cosine similarity.

On frames where InsightFace doesn't run, the system uses IoU (Intersection over Union) matching to link the current YOLO face boxes to the most recent InsightFace results. This keeps identity labels stable without running the expensive recognition model every frame.

The cosine similarity threshold is set at 0.4 — intentionally low to avoid missing matches, since the ArcFace embeddings are quite discriminative even at lower thresholds.

Alert System

The alert engine uses a simple hierarchy:

• Fight + Weapon → 🚨 CRITICAL ALERT (red)
• Fight only → 🚨 ALERT (red)
• Weapon only → ⚠️ WARNING (orange)
• Nothing → ✅ Safe (green)

---

Limitations & Future Work

Current limitations:

• No persistent logging or recording — alerts are real-time only, nothing is saved to disk
• Single-camera support — the system processes one video stream at a time
• InsightFace on CPU is still the main bottleneck even with frame-skipping
• The face recognition database is static — adding new faces requires a restart
• No temporal smoothing on fight detections, so brief false positives can occur

Things I'd like to add:

• Multi-camera support with a unified dashboard
• Alert logging with timestamps and frame snapshots
• Temporal smoothing / confirmation buffers to reduce false positive spikes
• Dynamic face enrollment through the UI (without restart)
• Integration with notification services (email, Telegram, etc.)
• RTSP stream support for actual CCTV cameras

---

Acknowledgements

• Ultralytics for YOLO model training and inference
• InsightFace for the ArcFace facial recognition pipeline
• Streamlit for the web interface
• OpenCV for video capture and frame processing

---

Built as part of coursework at IIT BHU. If you're working on something similar or have suggestions, feel free to open an issue or reach out.