laserdisc-io/fs2-aws

Split up S3 into its own module and update to v2

gvolpe opened this issue · 7 comments

gvolpe commented

Hi,

Any interest in having a single module with only the awssdk:s3 version 2 dependency? I adapted all the functions in https://github.com/laserdisc-io/fs2-aws/blob/master/fs2-aws/src/main/scala/fs2/aws/s3/package.scala and I'd be happy to contribute that but it now lives in a private repo (we are using it in production at the moment).

Let me know what you think.

semenodm commented

Hi, thank you for using this library. Indeed it is in our roadmap to migrate to aws sdk v2 where possible and separate the kinesis, sqs and s3 into separate modules.
I would really appreciate your PR on this matter, we can definitely do this.

semenodm commented

this issue related to #275 S3 might be next candidate to be decoupled from fs2-aws module

gvolpe commented

Cool! I'll put up a PR with the new implementation very soon :)

gvolpe commented

These are the type signatures I have at the moment:

/* A purely functional abstraction over the S3 API based on fs2.Stream */
trait S3[F[_]] {
  def uploadFile(bucket: S3.BucketName, key: S3.FileKey): Pipe[F, Byte, S3.ETag]
  def uploadFileMultipart(bucket: S3.BucketName, key: S3.FileKey, partSize: S3.PartSizeMB): Pipe[F, Byte, S3.ETag]
  def readFile(bucket: S3.BucketName, key: S3.FileKey): Stream[F, Byte]
  def readFileMultipart(bucket: S3.BucketName, key: S3.FileKey, partSize: S3.PartSizeMB): Stream[F, Byte]
}

S3.PartSizeMB is a refinement type but we need to consider whether we want a dependency on refined or not. I am personally okay with it since I use it everywhere but other users might prefer to get an Exception thrown if the input data is invalid.

I have this because each part must be at least 5 MB in size. More about it here.

BucketName and FileKey are newtypes but we can make them simple values classes. The important thing is to avoid having two arguments of type String.

Thoughts on this?

gvolpe commented

I realized that refined is already a dependency of fs2-aws so I guess we are fine with having it?

semenodm commented

Yes refined is fine

gvolpe commented

Great. I'll put up a PR this weekend. Got the implementation but need to adapt the existing tests and I don't have much time now.