Skip to content

README.md

Latest commit

 

History

History
161 lines (115 loc) · 4.67 KB

README.md

File metadata and controls

161 lines (115 loc) · 4.67 KB
Raw

Filecoin Storage Benchmark

Python Filecoin License CI

Benchmarking and stress testing tools for the Filecoin decentralized storage network via the web3.storage API.

Overview

This project provides experimental benchmarking tools to evaluate Filecoin storage performance:

Feature Description
Upload throughput Measures speed & latency to store data on Filecoin via web3.storage
Retrieval latency Tests data retrieval speed through the w3s.link gateway
Concurrent stress Simulates parallel workers to measure performance under load
Result visualization Auto-generates Matplotlib charts from JSON result files

The goal is to explore how Filecoin performs for real-world workloads such as AI dataset hosting, media distribution, and decentralized application storage.

Repository Structure

filecoin-storage-benchmark/
│
├── benchmark/
│   ├── benchmark.py        # Upload / retrieval / latency benchmark
│   └── stress_test.py      # Concurrent stress test
│
├── dashboard/
│   └── plot_results.py     # Matplotlib chart generator
│
├── results/                # JSON output files (auto-created)
│   └── README.md
│
├── .github/
│   └── workflows/
│       └── ci.yml          # Lint & import checks
│
├── requirements.txt
└── README.md

Quick Start

1. Prerequisites

2. Clone this repository

git clone https://github.com/hosammahdy91/filecoin-storage-benchmark.git
cd filecoin-storage-benchmark

3. Install dependencies

pip install -r requirements.txt

4. Set your API token

export W3S_TOKEN="your_web3_storage_token_here"

5. Run the benchmark

# Default: tests 0.5 MB, 1 MB, 5 MB files
python benchmark/benchmark.py

# Custom file sizes
python benchmark/benchmark.py --sizes 1 10 50

# More latency repetitions
python benchmark/benchmark.py --sizes 1 5 --latency-repeat 10

6. Run the stress test

# Default: 8 workers, 256 KB per file
python benchmark/stress_test.py

# Custom
python benchmark/stress_test.py --workers 16 --file-size-kb 512

7. Visualize results

# Benchmark chart
python dashboard/plot_results.py --type benchmark

# Stress test chart
python dashboard/plot_results.py --type stress

Charts are saved to dashboard/benchmark_chart.png and dashboard/stress_chart.png.

Example Output

══════════════════════════════════════════════════════════════
  Filecoin / web3.storage Benchmark
══════════════════════════════════════════════════════════════

── 1.0 MB ───────────────────────────────────────────────────
  ↑  Uploading 1.00 MB  [1.0MB]
     ✓  CID: bafybeig...  |  1.243s  |  824.36 KB/s
  ↓  Retrieving bafybeig...  [1.0MB]
     ✓  1.00 MB  |  0.412s  |  2.43 MB/s

── Latency ──────────────────────────────────────────────────
  ⏱   Latency test  (×5)
     ✓  avg=398.2ms  min=341.1ms  max=471.6ms

✅  Results saved → results/benchmark_20250314_153201.json
══════════════════════════════════════════════════════════════

Configuration

Parameter Default Description
--sizes 0.5 1 5 File sizes in MB
--latency-repeat 5 Latency test repetitions
--workers 8 Concurrent workers
--file-size-kb 256 KB per worker file

Example Use Cases

  • AI dataset storage benchmarking
  • Large media file distribution
  • NFT asset and metadata storage
  • Decentralized app (dApp) data hosting
  • Comparing Filecoin vs centralized cloud storage

License

MIT © Hosam