CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgery

Overview

CAT-SG is the first large-scale dynamic scene graph dataset specifically designed for fine-grained understanding of cataract surgery workflows. Built upon the CATARACTS dataset, CAT-SG provides structured annotations of tool-tissue interactions, procedural variations, and temporal dependencies with over 1.8 million annotated relations.

Key Features

• 🔍 Fine-grained Annotations: Detailed semantic relationships between surgical tools and anatomical structures
• 📊 Large Scale: 50 high-resolution videos with 164,162 annotated frames
• ⏱️ Temporal Resolution: 5 fps sampling for capturing precise surgical dynamics
• 🔧 Multi-task Support: Enables scene graph generation, surgical workflow recognition, and technique recognition
• 🎯 Novel Tasks: Introduces surgical technique recognition (Stop and Chop vs. Divide and Conquer)

Dataset Statistics

Metric	Value
Videos	50
Average Duration	11 minutes
Annotated Frames	164,162
Unique Objects	29
Relation Types	9
Total Relations	1,811,252
Annotation Hours	1,200+

Annotation Categories

Objects (29 classes)

• Anatomical Structures: Pupil, Iris, Cornea, Skin
• Surgical Tools: Hand, Eye Retractors, Hydrodissection Cannula, Viscoelastic Cannula, Capsulorhexis Cystotome, Rycroft Cannula, Bonn Forceps, Primary Knife, Phacoemulsification Handpiece, Lens Injector, Irrigation/Aspi- ration Handpiece, Secondary Knife, Micromanipulator, Capsulorhexis Forceps, Suture Needle, Needle Holder, Charleux Cannula, Vitrectomy Handpiece, Mendez Ring, Marker, Troutman Forceps, Cotton, Iris Hooks, Vannas Scissors

Relations (9 types)

Relation	Description	Count
Close to	Spatial proximity	1,677,724
Activation	Tool activation/usage	44,552
Inserting	Tool insertion through incision	34,016
Retracting	Tool retraction	23,886
Holding	Tool fixation	13,380
Pulling	Pulling actions	11,895
Pushing	Pushing/rotating actions	3,874
Cutting	Cutting operations	1,925

Tasks Supported

1. Surgical Scene Graph Generation

Automatically construct structured scene graphs from surgical videos, detecting instruments, anatomical structures, and their interactions over time.

2. Surgical Workflow Recognition

Recognize surgical phases using the 19 predefined surgical steps from the CATARACTS dataset, enhanced with scene graph structural information.

3. Surgical Technique Recognition

Distinguish between two nucleus-breaking techniques:

• Stop and Chop
• Divide and Conquer

Dataset Structure

The naming of the annotation files corresponds to the names of the videos of the CATARACTS datset. Training videos are named "train01" to "train25", and val and test videos are named "test01" to "test25".

The subdirectory "splits" contains a copy of the dataset which is set up for being efficiently processed into an InMemoryDataset from pytorch geometric. The "raw" subdirectories contain the json annotation files, loading the data will create "processed" folders. The test and val datasets are further separated into subdirectories (i.e. test/test01/, test/test02 etc.) so that they can be easily loaded as individual dataloaders and evaluated separately for per-video metrics.

CAT-SG/annotations/
├── all/
│   ├── test01.json             # All 50 annotation files
│   ...
└── splits/                     
    ├── test/                   # File selections reflect CATARACTS train/test/val split
    │   ├── test01/raw/
    │   │   └── test02.json
    │   ├── test02/raw/
    │   │   └── test03.json
    │   ...    
    ├── train/raw/
    │   └── train01.json
    │   ...
    └── val/
        ├── val01/raw/
        │   └── test01.json
        ...

Annotation File Structure

The scene graph annotations are stored as JSON files with frame-level annotations. Each annotation file corresponds to one video of CATARATCS of the same name. Each frame is identified by its frame number and contains the following structure:

Field	Type	Description
entities	List[str]	List of all objects present in the frame (anatomical structures and surgical tools)
geometric_relations	List[List[str]]	Pairs of entities that have spatial "close to" relationships
semantic_relations	List[List[str]]	Pairs of entities with semantic relationships (holding, activation, cutting, etc.)
step	str	Current surgical phase/step (e.g., "Idle", "Incision", "Nucleus Breaking")
technique	str	Surgical technique used ("Stop and Chop", "Divide and Conquer", or empty string)
bounding_box	List[List[float]]	Normalized bounding boxes [x, y, width, height] for each entity (0-1 range)
pos	List[List[float]]	Normalized center positions [x, y] for each entity (0-1 range)
size	List[float]	Normalized area/size of each entity's segmentation (0-1 range)

Example Annotation Structure

{
  "1": {
    "entities": ["Pupil", "Surgical Tape", "Eye Retractors", "Iris", "Skin", "Cornea"],
    "geometric_relations": [["Iris", "Pupil"], ["Eye Retractors", "Surgical Tape"]],
    "semantic_relations": [["Bonn Forceps", "Cornea", "holding"]],
    "step": "Idle",
    "technique": "",
    "bounding_box": [[0.421875, 0.160185, 0.225, 0.376852], ...],
    "pos": [[0.534335, 0.351811], ...],
    "size": [0.066701, 0.414560, ...]
  }
}

Bouding Box, Position and Size labels are derived from a segmentation model trained on CaDIS. They might contain errors and should be considered pseudolabels.

Getting Started

Download CATARACTS

To access the 50 underlying videos of this dataset, please refer to the CATARACTS dataset page on IEEE DataPort. Please consider their license and conditions as well.

Download CAT-SG

# Clone the repository
git clone https://github.com/felixholm/CAT-SG.git
cd CAT-SG

Loading Data

If you want to use the dataset as dynamic scene graphs like we did in the paper, feel free to use the attached 'dataloader.py' for reference on how to load the data using the efficient InMemoryDataset class of pytorch_geometric.

Benchmarks

Scene Graph Generation Results

Method	Micro F1	Macro F1
ORacleSV	72.45	34.65
ORacleSVT	66.19	34.49
CatSGG	89.78	42.08
CatSGG+	89.78	43.11

Surgical Workflow Recognition

Method	Accuracy	F1 Score
Baseline (Holm et al.)	73.77	64.93
GATv2 + CAT-SG	78.63	70.15

Surgical Technique Recognition

Method	Accuracy	F1 Score
GATv2 (10s, 5fps)	68.75 ± 4.11	48.40 ± 1.72

Citation

If you use CAT-SG in your research, please cite our paper:

@inproceedings{10.1007/978-3-032-05114-1_10,
    author = {Holm, Felix and \"{U}nver, G\"{o}zde and Ghazaei, Ghazal and Navab, Nassir},
    title = {CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgery},
    year = {2025},
    isbn = {978-3-032-05113-4},
    publisher = {Springer-Verlag},
    address = {Berlin, Heidelberg},
    url = {https://doi.org/10.1007/978-3-032-05114-1_10},
    doi = {10.1007/978-3-032-05114-1_10},
    booktitle = {Medical Image Computing and Computer Assisted Intervention – MICCAI 2025: 28th International Conference, Daejeon, South Korea, September 23–27, 2025, Proceedings, Part IX},
    pages = {96–106},
    numpages = {11},
    keywords = {Scene Graphs, Surgical Data Science, Dataset, Cataract Surgery},
    location = {Daejeon, Korea (Republic of)}
}

License

This dataset is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.

Key Terms:

• ✅ Attribution: You must give appropriate credit and indicate if changes were made
• ✅ Research Use: Free to use for academic research and educational purposes
• ❌ No Commercial Use: Commercial use is not permitted without explicit permission
• ✅ Share Alike: You may distribute, remix, and build upon the material

See LICENSE for full details.

Acknowledgments

• Supported by Carl Zeiss AG
• Built upon the CATARACTS dataset
• Segmentation masks based on the CaDIS dataset

Contact

For questions or issues, please contact:

• Felix Holm: felix.holm@tum.de
• Open an issue on this repository