Kalman Tracker (Color-Based Object Tracking)

Author: Alvaro Balbin University of Bath

This project is a real-time Kalman filter tracker for a colored object.
It uses HSV color segmentation to detect the object in each frame, then uses a Kalman filter to predict and smooth its motion.

The code runs live from a webcam and visualizes:

• the raw detection (red),
• the bounding box (green),
• and the Kalman estimate (blue).

iPhone Tracking

Tracking is smooth because the iPhone has a consistent color, so the HSV mask isolates it cleanly.

Detection Result:

---

Leaf Tracking

Tracking is more difficult because the leaf contains multiple shades of green and varying lighting.
The HSV mask flickers and sometimes includes background areas, which makes the Kalman filter work harder to stay stable.

Detection Result:

How It Works

1. Capture a frame from the webcam.
2. Convert RGB → HSV and threshold (hue, saturation, value) to build a binary mask for the target color.
3. Clean the mask using bwareaopen and imfill, then pick the largest blob.
4. Extract the blob centroid [x, y] and bounding box.
5. Initialize a constant-velocity Kalman filter with state [x, y, vx, vy].
6. Every frame:
   • Predict next position using the motion model.
   • If a detection exists, correct with that measurement.
   • Draw overlays and log both the measured position and the Kalman-estimated position.

After the run, the script computes pixel RMSE between the raw detections and the Kalman estimates and plots both trajectories for comparison.

All main logic is in kalman_tracker.m.

Observations

• With the iPhone:

  • The color is consistent.
  • The HSV thresholds isolate it as one solid blob.
  • The Kalman filter just has to smooth minor jitter.

• With the leaf:

  • The “green” is actually a bunch of different hues and brightness levels.
  • Shadows, highlights, and background foliage also fall into the green range.
  • The mask breaks into multiple blobs or jumps to background.
  • The Kalman filter can smooth noise, but if the detection itself jumps, the track will jump.

In other words: the bottleneck wasn’t the Kalman math, it was reliable color segmentation on a messy object.

Repository Structure

Path	Description
kalman_tracker.m	Main MATLAB script. Includes webcam capture, HSV-based detection, Kalman filter (init / predict / correct), visualization, and RMSE analysis.
images/	Screenshots of tracking results (iPhone vs leaf).
videos/	Short demo clips of both tests.
.gitkeep	Keeps the folders in Git even if empty.

Possible Improvements

These are the next steps I would try:

• Automate HSV calibration

  • The script already includes an HSV check section that lets the user click on the target object to get its color values.
  • A possible next step would be to make this automatic: use the sampled HSV range directly to set thresholds and start tracking without editing the code.

• Rejection logic / data association

  • If multiple green blobs show up, choose the one closest to the last Kalman prediction instead of just “largest area.” That prevents jumping to background.

• Occlusion handling

  • If no detection this frame, keep predicting with the Kalman filter instead of snapping to nothing.

References

Key MATLAB documentation and resources used during development:

• MATLAB: Object Detection (Computer Vision Toolbox)
• MATLAB: Understanding Color Spaces
• MATLAB: insertShape
• Kalman Filter (Wikipedia)
• Kalman Gain Explanation
• Root Mean Square Error (C3.ai)
• Shades of Green (Wikipedia)