Concurrent s3 parts upload

[barnabehvrd]

Following PR pelican/panel#2510 on the Panel, this PR makes Wings upload S3 backup parts concurrently instead of one after another.
The number of parts in flight is bounded by the max_concurrent_uploads value returned by the Panel alongside the part URLs, and the upload stops as soon as any part fails permanently.

Uploading parts in parallel was not possible before: every part read from the same shared io.Reader, so concurrent reads would have interleaved and produced a corrupted archive.

Each part now gets its own io.SectionReader over the backup file, which is safe for concurrent use and lets a failed part be retried from its correct offset. Parts are uploaded through an errgroup whose limit comes from the max_concurrent_uploads value returned by the Panel alongside the part URLs. Results are written to a pre-allocated slice so the part list stays ordered regardless of completion order.

If the Panel does not include max_concurrent_uploads in its response (older Panel version) or sends an invalid value, Wings falls back to 1, preserving the previous sequential behavior. Upgrading Wings without upgrading the Panel is therefore safe.

This also fixes the retry path, which could never work: the request was built once outside the backoff loop, so a retry replayed a body that had already been consumed and sent an empty payload with a mismatched Content-Length.

The request is now rebuilt on every attempt and the section is rewound first.

The backoff's elapsed-time bound has been removed as well, since a single part can take longer than the previous one-minute budget, which meant a second attempt never happened. The two-hour request timeout and the retry count are unchanged.

Idle connections are now kept in the pool between parts instead of being renegotiated, and response bodies are drained so connections can be reused.

The results are as follows:

Concurrency	Duration	Throughput	Speedup
1	45.2 s	23.2 MB/s	baseline
5	5.37 s	195 MB/s	8.4x
10	4.31 s	244 MB/s	10.5x
20	3.46 s	303 MB/s	13.1x

Note :
These results were obtained on a fresh GCP E2 instance (2vCPU / 4GB of RAM / roughly 3-4 Gbps bandwith) for a bakcup of ~ 1000 MiB saved at BackBlaze B2. All parts were 50MiB except the last one (just a few KiB)

The benchmark backup only contains ~20 parts, which caps the benefit of higher concurrency levels: at 20 concurrent uploads, half the backup is in flight at once, so the marginal gain flattens. On real-world backups (10-50+ GiB, hundreds of parts), a default of 10 keeps the upload pipeline full throughout the transfer without hammering the S3 endpoint.

Most of the gain is already there at the lower concurrency levels, and the highest setting only buys a little more while getting close to the link's capacity. The sequential baseline is also the median of several runs that varied widely: with a single connection the total time depends entirely on how that one connection happens to behave, and one stalled part goes straight into the total. Concurrent uploads absorb that, so the win is in predictability as much as in raw throughput.

Summary by CodeRabbit

• New Features

  • Added support for reporting the maximum number of concurrent backup uploads.
  • Backup uploads now run concurrently with a controlled limit, improving throughput.
  • Added automatic retries for temporary upload failures, including rate limiting responses.

• Reliability

  • Uploads now respond more effectively to cancellation and retry individual parts when needed.
  • Improved connection handling helps maintain reliable multipart uploads.

[coderabbitai]

📝 Walkthrough

Walkthrough

The upload response now reports the maximum concurrent upload count. S3 multipart uploads now run with bounded concurrency, independent file sections, context cancellation, dedicated HTTP transport settings, and expanded retry handling.

Changes

Backup upload flow

Layer / File(s)	Summary
Upload concurrency contract remote/types.go	BackupRemoteUploadResponse now includes MaxConcurrentUploads serialized as max_concurrent_uploads.
Concurrent S3 multipart execution server/backup/backup_s3.go	Multipart uploads use bounded errgroup workers and independent io.SectionReader instances. Requests use the provided context, rebuild bodies on retries, drain response bodies, retry HTTP 429 and 5xx responses, and detect wrapped backoff errors with errors.As.

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested reviewers: quintenqvd0

Sequence Diagram(s)

sequenceDiagram
  participant BackupUploader
  participant osFile as *os.File
  participant errgroup
  participant HTTPClient
  participant S3Endpoint
  BackupUploader->>osFile: Read file size
  BackupUploader->>HTTPClient: Retrieve upload URLs with context
  BackupUploader->>errgroup: Start bounded upload workers
  errgroup->>osFile: Read independent io.SectionReader parts
  errgroup->>HTTPClient: Send multipart part request
  HTTPClient->>S3Endpoint: Upload part
  S3Endpoint-->>HTTPClient: Return response
  HTTPClient->>S3Endpoint: Retry HTTP 429 or 5xx response

Loading

Poem

I’m a rabbit with uploads to make,
Many small parts hop in the wake.
Workers bound, retries grow wise,
Drained responses keep connections alive.
One field now tells how many may fly.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: concurrent S3 part uploads.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}