The new AWS SDK version allows automatic multi-upload for large files with multipartEnabled(true), so that client doesn't have to do multipart upload manually, and instead can just use PutUploadRequest as usual.
Based on our investigation, uploading a dummy 10G file with zio-s3 multipartUpload took 72 seconds in our r8a.4xlarge (EBS, gp2 volume, 250MB/s max burst), while using automatic multipart upload took only 32 seconds.
I think we can have the multipartEnabled by default, and allow customizing the thresholdInBytes.
P/S: There seems to be another overhead due to the conversion between ZStream and AsyncRequestBody. We found out that just using AsyncRequestBody.fromFile() with S3AsyncClient took less than 10 seconds to upload the same file. Will try to trace down more.
The new AWS SDK version allows automatic multi-upload for large files with
multipartEnabled(true), so that client doesn't have to do multipart upload manually, and instead can just usePutUploadRequestas usual.Based on our investigation, uploading a dummy 10G file with zio-s3
multipartUploadtook 72 seconds in our r8a.4xlarge (EBS, gp2 volume, 250MB/s max burst), while using automatic multipart upload took only 32 seconds.I think we can have the multipartEnabled by default, and allow customizing the
thresholdInBytes.P/S: There seems to be another overhead due to the conversion between
ZStreamandAsyncRequestBody. We found out that just usingAsyncRequestBody.fromFile()withS3AsyncClienttook less than 10 seconds to upload the same file. Will try to trace down more.