I am trying to use .NET 8 to zip a large amount of files stored in S3 and save the archive in another S3 location, while consuming a limited amount of memory.

Basically my needs are:

• I have a bunch of files in a S3 bucket.
• I want to zip them all and store the archive in another S3 bucket.
• I don’t want my script to store more than 100 MB of archive data in memory.

So the script should be able to start downloading & zipping the files into a memory buffer, stream the archive back to S3 in parallel, “forget” about the buffered data it has already uploaded back to S3, and pause the download if the 100 MB in-memory buffer is full (upload speed might be limited).

I tried to fiddle with memory streams but could not come up with a working solution. Here are some working code samples though:

Download S3 files to zip stream:

private async Task DownloadFromS3AndZip(string bucketName, Stream zipStream)
{
    var request = new ListObjectsV2Request() { BucketName = bucketName };
    ListObjectsV2Response listObjects = await this._s3Client.ListObjectsV2Async(request);

    using var zipArchive = new ZipArchive(zipStream, ZipArchiveMode.Create, leaveOpen: true);
    foreach (S3Object s3Object in listObjects.S3Objects)
    {
        GetObjectResponse file = await s3Client.GetObjectAsync(s3Object.BucketName, s3Object.Key);
        ZipArchiveEntry entry = zipArchive.CreateEntry(file.Key);
        await using Stream entryStream = entry.Open();
        await file.ResponseStream.CopyToAsync(entryStream);
    }
}

Upload zip stream to S3 :

private async Task UploadToS3(string bucketName, string objectKey, Stream stream)
{
    var request = new TransferUtilityUploadRequest()
    {
        BucketName = bucketName,
        Key = objectKey,
        InputStream = stream,
        ContentType = "application/zip",
    };
    await this._s3Transfer.UploadAsync(request);
}