4.0.0-RC1 and Amazon S3

[mn98]

Apologies if I am behind the times here. I recently upgraded a project of mine to CE3 which uses the excellent fs2-aws-s3 module to stream data from Amazon S3 buckets. In 4.0.0 I only see kinesis-style streaming. Is there a plan to interface with S3 directly or is that now gone in favour of kinesis and should I be looking to move my data from S3 to a kinesis stream? I may well have missed something obvious, if so I would grateful if someone could demonstrate how to pull from S3 using fs2-aws-kinesis.

[semenodm]

Hello.
Help me understand better your use case, because the link you provided describes how to stream S3 data into kinesis using AWS Database Migration service, and you probably don't need to write your own code and hence use this library.
However if you want to stream data yourself you could use fs2-aws-s3 and fs2-aws kinesis, both are published to maven central under 4.0.0-RC1 version

[mn98]

Ok, entirely my misunderstanding, thank you! I didn't realise fs2-aws-s3 had been published under 4.0.0-RC1.
I am using this similarly to how I'd used it previously (minus the now defunct Blocker, which btw still appears in some of the docs, which I'd be happy to rectify if you'd like as penance for raising this issue). This works fine but suggestions on 'better' ways to write this are always welcome.

  private val s3ClientResource: Resource[IO, S3AsyncClient] =
    Resource.fromAutoCloseable(
      IO(S3AsyncClient.builder().build())
    )

  def dataFromS3[A](
                    bucketName: String,
                    fileNames: Seq[String]
                  ): Stream[IO, A] =
    for {
      client <- Stream.resource(s3ClientResource)
      s3 <- Stream.eval(S3.create[IO](Interpreter[IO].create(client)))
      data <-
        fileNames.foldLeft(Stream[IO, Byte]()) { case (s, fileName) =>
          s ++ s3.readFileMultipart( ...

[semenodm]

All contributions are welcome.
so, there is separate module in this project with different examples. As far as i understand, you want to read from S3 and put data into Kinesis.
For streaming data from S3, take a look here. It has simple read file, if you are dealing with big files, you should use readFileMultipart API
Writing into kinesis should relatively easy, since all you need to produce messages into the Stream and you can look here
Both theses examples will give you idea how to prepare the SDK clients. Indeed, you don't need the Blocker anymore in 4.X.X series as well as we removed blocker dependency from pure-aws modules in 3.X.X series, thanks to @fernandomora

[mn98]

All is well, thanks for taking the time to educate me!