3D Skeleton & Landmark Extraction Pipeline for Video Datasets (MediaPipe)

A robust, dataset-agnostic pipeline designed to extract, geometrically normalize, and batch-process 3D human skeletons (body, hands) from any video dataset using Google's MediaPipe Holistic model.

Whether you are working on Sign Language Recognition (SLR), Action Recognition, or Human-Computer Interaction (HCI), this tool converts raw video pixels into clean, normalized PyTorch tensors (.pt) or NumPy arrays (.npy), ready to be fed directly into deep learning models (like LSTMs, Transformers, or GCNs).

💡 Featuring seamless integration with the AUTSL Sign Language Dataset: While generalized for any dataset, this repository provides built-in support for processing the large-scale AUTSL dataset right out of the box.

Features

• Full Skeleton Tracking: Extracts 75 3D landmarks per frame (33 body, 21 left hand, 21 right hand).
• Geometric Normalization: Automatically centers the skeleton to the shoulders, scales it by shoulder width, and rotates it to face the camera. This drastically improves model training performance.
• Memory Leak Protection: The batch processing pipeline utilizes subprocesses to prevent MediaPipe's notorious memory and graph leaks during large-scale dataset processing.
• Versatile: Can process entire datasets in batch, extract from a single video, or create skeleton animation videos for debugging.

Installation

1. Clone this repository or download the source code.
2. It is highly recommended to use a virtual environment (like venv or conda).
3. Install the required dependencies:

pip install -r requirements.txt

Usage

This project is divided into three main tasks:

1. Batch Dataset Processing

Use this to convert an entire directory of videos into .pt files containing normalized tensors.

python tools/process_dataset.py --source_dir /path/to/videos --target_dir /path/to/save/landmarks --pattern "*.mp4"

Note for AUTSL Dataset: To specifically target AUTSL videos, simply change the pattern argument to --pattern "*_color.mp4".

2. Single Video Processing

If you just want to extract landmarks from a single video, use core/pose_estimator.py. You can save the output as a PyTorch Tensor (.pt), a NumPy Array (.npy), or an annotated video (.mp4).

# Save as PyTorch Tensor
python core/pose_estimator.py --input_video sample.mp4 --output_file output.pt

# Save as NumPy Array
python core/pose_estimator.py --input_video sample.mp4 --output_file output.npy

# Save as Annotated Video
python core/pose_estimator.py --input_video sample.mp4 --output_file annotated_video.mp4

3. Landmark Visualization

If you have generated a .pt file and want to see what the normalized skeleton looks like, use the tools/visualizer.py script. It will generate an animation of the skeleton on a black background.

python tools/visualizer.py --input_pt output.pt --output_video skeleton_animation.mp4 --width 1280 --height 720 --fps 30

Output Data Format

For .pt and .npy outputs, the resulting tensor/array has the shape: (number_of_frames, 75, 4)

• 75 Landmarks: Indices 0-32 are Body, 33-53 are Left Hand, 54-74 are Right Hand.
• 4 Coordinates: (x, y, z, visibility). Note that hand landmarks do not have a confidence/visibility score in MediaPipe, so their visibility is hardcoded to 1.0. All coordinates are geometrically normalized.

Troubleshooting

MediaPipe RegEx / Graph Error on Linux (Turkish Locale Bug): If your system language is Turkish (or another language with different uppercase/lowercase rules for 'i') and you encounter an error like: ValueError: while processing the input streams of subgraph node InferenceCalculator: TAG:index:name is invalid... does not match... This is a known bug in MediaPipe's string parser. To fix this, simply prefix your command with LC_ALL=C to enforce standard English locale temporarily. For example:

LC_ALL=C python core/pose_estimator.py --input_video sample.mp4 --output_file output.pt