I have a (possibly) quite large
ResultSet when querying a (Vertica) SQL Database, and because it won’t fit in memory I’m trying to find a way to stream the results (in CSV form) directly to an S3 bucket.
Most S3-related documentation and questions revolve around uploading Files or directories, where you can just pass in an InputStream to the base S3 SDK, or the TransferManager, but those aren’t applicable.
It also seems like in all cases the S3 SDK requires to know the content length beforehand, but that’s obviously not possible with a query result.
There is https://github.com/alexmojaki/s3-stream-upload, but that uses the soon (2025) to be deprecated v1 of the sdk, so that doesn’t seem like a viable long-term option either.
Will my only option be to manually create batches of 5MB and do a multi-part upload using a more low-level SDK? Or are there other options?