A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.
-
Updated
Feb 9, 2026 - Python
#
webdataset
Here are 15 public repositories matching this topic...
Minimal sharded dataset loaders, decoders, and utils for multi-modal document, image, and text datasets.
computer-vision pdf-document datasets distributed-training dataloading document-understanding multi-modal-learning webdataset
-
Updated
Apr 3, 2024 - Python
Unofficial DynaDUSt3R reimplementation trained on Stereo4D (research only).
-
Updated
Oct 18, 2025 - Python
[CVPR 2021: Oral] In this work, we show that high frequency Fourier spectrum decay discrepancies are not inherent characteristics for existing CNN-based generative models.
pytorch dcgan gans fourier-transform lsgan wgan-gp stargan pytorch-lightning cvpr2021 fourier-spectrum-discrepancies webdataset
-
Updated
Jun 12, 2021 - Python
DALLE-tools provided useful dataset utilities to improve you workflow with WebDatasets.
-
Updated
Mar 9, 2022 - Python
High-level API for tar-based dataset
-
Updated
Feb 3, 2024 - Python
This repo is the official released code of FoPro (AAAI-2023)
-
Updated
Dec 2, 2022 - Python
A sample subset of the NIH Chest X-ray Dataset. At only 2.4% of the size of the original dataset, it allows creating an accurate classifier using the Augmented Chest X-Ray repository.
-
Updated
Mar 29, 2022
Scripts to collect data from CARLA and save them as Webdataset
-
Updated
Sep 21, 2024 - Python
High-Throughput PyTorch Sequential Data Loaders for GPU Starvation Reduction
streaming numpy pandas python3 pytorch parquet dataloader abtesting sequential-data etl-pipeline adlsgen2 webdataset modeltraining highthrough-put gpu-starvation-reduction 4d-tensors 5d-tensors
-
Updated
May 6, 2026 - Python
Stream WebDataset shards straight from Backblaze B2 object storage into PyTorch training — no local staging disk. A FastAPI + Next.js sample app that packs media into .tar shards, writes them to B2, and streams them back as an IterableDataset with live throughput and a distributed worker/node shard split.
python machine-learning deep-learning nextjs s3 pytorch sample-app object-storage backblaze-b2 data-streaming cloudstorage model-training distributed-training fastapi webdataset b2-labs
-
Updated
Aug 10, 2026 - TypeScript
Bulk-download image-text datasets into Backblaze B2 as reproducible WebDataset tar shards with img2dataset — no local staging disk. Sample app (FastAPI + Next.js) that streams shards to S3-compatible object storage, validates download yield, and streams them back for PyTorch/JAX training.
machine-learning nextjs s3 pytorch sample-app dataset object-storage backblaze-b2 cloudstorage training-data fastapi webdataset img2dataset b2-labs
-
Updated
Aug 7, 2026 - TypeScript
Web project, using SparQL on dbpedia and wikidata for mental disorder search feature
-
Updated
Jan 11, 2024 - PHP
Systematic VLA training optimization on 2× RTX 3090. WebDataset + FlashAttention-2 + FSDP → 3.3× throughput, 26% VRAM reduction. Profiler traces and W&B report linked. Reproducible in one command.
robotics pytorch tensorboard profiling imitation-learning multi-gpu vla gpu-optimization huggingface weights-and-biases webdataset fsdp flash-attention vision-language-action training-efficiency ml-infra
-
Updated
Jul 1, 2026 - Python
Improve this page
Add a description, image, and links to the webdataset topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the webdataset topic, visit your repo's landing page and select "manage topics."