🛡️ Threat Detector CLI API

A lightweight, CLI-first threat intelligence tool that wraps the ThreatFox API, letting analysts query and submit Indicators of Compromise (IOCs) from the terminal, a browser, or curl — with clean, human-readable output instead of raw JSON.

Deployed across multiple HTTP servers behind an HAProxy load balancer for high availability.

---

Table of Contents

• Overview
• Features
• Architecture
• Endpoints
• Submitting IOCs
• Example Usage
• Setup & Deployment
• Validation & Security
• Known Limitations
• Future Improvements
• License
• Credits

---

Overview

Threat Detector is implemented as a Python CLI-style server using http.server, designed to preserve the feel of a command-line tool while being reachable over standard HTTP. It accepts GET requests and returns formatted plain text, making it easy to pipe into other tools or read directly in a browser.

Deployment topology:

                ┌───────────┐
   curl/browser │  Lb01     │  HAProxy round-robin
   ────────────▶│ (port 80) │
                └─────┬─────┘
                      │
          ┌───────────┴───────────┐
          ▼                       ▼
     ┌─────────┐             ┌─────────┐
     │  Web01  │             │  Web02  │
     │ :8080   │             │ :8080   │
     └─────────┘             └─────────┘

---

Features

• 🔍 Query IOCs by ID, keyword (IP/domain), or file hash (MD5, SHA1, SHA256)
• 🧫 Fetch recent IOCs from the past 1–7 days
• 🧠 List known malware families
• 📤 Submit new IOCs directly to ThreatFox
• 📃 Human-readable output — formatted plain text, not raw JSON
• ⚖️ Load-balanced across multiple backend servers via HAProxy

---

Architecture

Component	Role
main.py	HTTP server handling routing and ThreatFox API calls
Web01 / Web02	Identical backend instances of the app
Lb01 (HAProxy)	Round-robin load balancer in front of the backends
.env	Stores the ThreatFox API key (excluded via .gitignore)

---

Endpoints

All endpoints are GET requests and return plain text.

Endpoint	Description
/	Health check
/recent_iocs?days=3	Recent IOCs from the past N days (1–7)
/ioc_by_id?id=123456	IOC details by ID
/ioc_by_keyword?keyword=badsite.com&exact_match=true	Search IOCs by IP/domain
/ioc_by_hash?hash=<your_hash>	Search IOC by file hash
/malware_list	List of known malware families
/ioc_by_malware?family=ZLoader&limit=20	IOCs related to a malware family
/submit_ioc?...	Submit a new IOC (see below)

---

Submitting IOCs

Example submission:

/submit_ioc?ioc_type=domain&iocs=malicious.site&threat_type=phishing&malware=ZLoader&confidence=85&comment=Seen+in+open+phishing+campaign

Required fields:

Field	Description
ioc_type	domain, ip, url, md5, sha1, or sha256
iocs	The actual indicator
threat_type	e.g. phishing, malware
malware	Name of the malware family
confidence	Integer, 0–100
comment	(optional) free-text context

---

Example Usage

Query an IOC by ID:

curl http://localhost:8080/ioc_by_id?id=123456

Output:

🎯 IOC found by id entered:
IOC: badsite.com | Type: domain | Tags: phishing,malware
Threat type of IOC: phishing - Credential harvesting
Malware name in IOC: ZLoader
Level of malice carried: 95
...

---

Setup & Deployment

1. Install dependencies

pip install -r requirements.txt

2. Run locally

python3 main.py

The server runs on port 8080 by default.

3. Run with Docker

docker build -t threat_detector .
docker run -p 8080:8080 threat_detector

4. HAProxy load balancer (sample config)

/etc/haproxy/haproxy.cfg:

frontend http_front
   bind *:80
   default_backend webapps

backend webapps
   balance roundrobin
   server web01 172.20.0.11:8080 check
   server web02 172.20.0.12:8080 check

Reload HAProxy after editing the config:

docker exec -it lb-01 sh -c 'haproxy -sf $(pidof haproxy) -f /etc/haproxy/haproxy.cfg'

Test from the host machine:

curl http://localhost

(using whichever port is exposed on lb-01)

---

Validation & Security

• All user input is validated: IPs, domains, and hashes are checked with regex before use
• Malformed or missing parameters return clear, descriptive error messages
• Confidence values are constrained to the 0–100 range
• No credentials are stored in source control — the ThreatFox API key is loaded from a .env file excluded via .gitignore

---

Known Limitations

During development, two issues stood out and are worth flagging honestly rather than glossing over:

1. CLI vs. HTTP mismatch. Getting a CLI tool to behave consistently over HTTP (via http.server) took real effort — the browser and terminal responses initially diverged.
2. Inconsistent output post-deployment. After deployment, the browser began returning JSON instead of the intended plain-text format, while the CLI stayed correct. Rather than continuing to patch around it and risking a more tangled codebase, this was left as a known issue for a future rewrite with a proper front end.

---

Lessons Learnt

• A CLI tool and an HTTP server are not the same UX problem. Reusing http.server to preserve a "CLI feel" saved time early on, but it also imported CLI assumptions (plain text, single format) into an environment — the browser — that has its own defaults. The mismatch between terminal and browser output was a symptom of not deciding early which client the tool was really for.
• Silent divergence is worse than an early crash. The plain-text/JSON inconsistency only showed up after deployment, not locally. Testing against the actual deployed environment (not just localhost) earlier would have caught this before it became confusing to unwind.
• Don't keep patching around an unclear design decision. Every attempt to fix the output mismatch in place made the code harder to reason about. Recognizing that and stopping — rather than layering on more conditionals — was the right call; the correct fix is a proper front end, not another patch.
• Validate input at the boundary, not deep in the logic. Doing regex validation for IPs/domains/hashes right at the request-handling layer, before touching the ThreatFox API, kept error handling predictable and made bad requests fail fast with a clear message.
• Load balancing surfaces state bugs you don't see with one server. Running Web01/Web02 behind HAProxy is a good habit to build early — any place the app accidentally depended on local state (instead of being stateless per request) would show up as inconsistent behavior across requests, even though this project stayed simple enough to avoid it.
• Keep secrets out of git from day one. Using .env + .gitignore from the start avoided the much more painful problem of scrubbing an API key from git history later.

---

Future Improvements

Roughly ordered from quick wins to larger undertakings:

Near-term

• Build a simple web front end (HTML/CSS/JS) so the tool has a real UI instead of raw-text browser output — directly addresses the CLI/browser output mismatch above
• Add a ?format=json query parameter so the API can serve both plain text (CLI feel) and structured JSON (for scripting/integration) from the same endpoints
• Centralize input validation and error handling into a shared module/middleware
• Add a requirements.txt version pin and a .env.example for easier onboarding
• Write unit tests for each endpoint (mock the ThreatFox API calls)

Mid-term

• Add response caching (e.g. Redis or in-memory TTL cache) to reduce redundant ThreatFox API calls and improve latency under load
• Add rate limiting per client IP to prevent abuse of /submit_ioc
• Add structured logging (JSON logs) and basic request metrics, exportable to a log aggregator
• Add a docker-compose.yml that spins up Web01, Web02, and Lb01 together for one-command local deployment
• Add authentication (API key or token) for the /submit_ioc endpoint specifically, since it's a write operation
• Add Shodan enrichment for IP-based IOCs: alongside the ThreatFox lookup, query Shodan's /host/{ip} endpoint to surface open ports, banners, exposed services, and known CVEs tied to that host. This turns a bare "this IP is malicious" result into "this IP is malicious and is running an exposed RDP service on port 3389 with an outdated banner" — much more actionable for triage
• Extend the same enrichment idea to Censys as a second vantage point, since Shodan and Censys don't always index the same hosts

Longer-term

• Export query results in STIX 2.1 or CSV format for direct ingestion into a SIEM or a MISP instance
• Add a lightweight local IOC cache/database (SQLite) to support offline queries and historical trend lookups
• Add a webhook/alerting mode: notify a Slack/Discord/email endpoint when a submitted or queried IOC matches a watchlist
• Add HTTPS termination at the load balancer (HAProxy + Let's Encrypt) instead of plain HTTP
• Add CI/CD (GitHub Actions) to run tests and build/push the Docker image on every merge to main

---

Credits

• Abuse.ch — ThreatFox threat intelligence feed
• Spamhaus