Kinesis validates the AWS credentials and fail with fake ones (LocalStack)

[DavidCorral94]

We use LocalStack to mock Kinesis, among other AWS services, and fs2-aws as library to communicate with AWS services. Recently, I've noticed an exception while executing some functional tests, and, after debugging it for a while I came to the conclusion that the credentials that are passed when building the KinesisProducerConfiguration, are validated somehow against AWS and, if they are not valid set of credentials, it will fail.

I've tested this not only with our code but with the available examples in this very same repo.

The exception, when an invalid AWS pair of credentials are provided, is the following:

[kpl-daemon-0003] 2023-05-22 11:18:47 INFO  c.a.s.k.p.LogInputStreamReader - [2023-05-22 11:18:47.814345] [0x000007fc][0x00007fd60a201700] [info] [kinesis_producer.cc:226] Created pipeline for stream "example"
[kpl-daemon-0003] 2023-05-22 11:18:48 INFO  c.a.s.k.p.LogInputStreamReader - [2023-05-22 11:18:48.499900] [0x000007fc][0x00007fd60a201700] [info] [AWS Log: WARN](AWSErrorMarshaller)Encountered AWSError 'InvalidClientTokenId': The security token included in the request is invalid.
[kpl-daemon-0003] 2023-05-22 11:18:48 WARN  c.a.s.k.p.LogInputStreamReader - [2023-05-22 11:18:48.500009] [0x000007fc][0x00007fd60a201700] [warning] [AWS Log: ERROR](AWSXmlClient)HTTP response code: 403
Resolved remote host IP address: 52.119.198.128
Request ID: 863b452e-35d0-451b-96d3-53fc8dbb9777
Exception name: InvalidClientTokenId
Error message: The security token included in the request is invalid.
4 response headers:
content-length : 306
content-type : text/xml
date : Mon, 22 May 2023 09:18:49 GMT
x-amzn-requestid : 863b452e-35d0-451b-96d3-53fc8dbb9777
[kpl-daemon-0003] 2023-05-22 11:18:48 INFO  c.a.s.k.p.LogInputStreamReader - [2023-05-22 11:18:48.500083] [0x000007fc][0x00007fd60a201700] [info] [AWS Log: WARN](AWSClient)If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[kpl-daemon-0003] 2023-05-22 11:18:48 ERROR c.a.s.k.p.LogInputStreamReader - [2023-05-22 11:18:48.500124] [0x000007fc][0x00007fd60a201700] [error] [pipeline.h:228] Failed to get StreamARN using STS GetCallerIdentity | Code: InvalidClientTokenId | Message: The security token included in the request is invalid. | Request was: Action=GetCallerIdentity&Version=2011-06-15
[kpl-daemon-0005] 2023-05-22 11:18:49 ERROR c.a.s.k.p.KinesisProducer - Error in child process
java.lang.RuntimeException: EOF reached during read
        at com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:532)
        at com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:508)
        at com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:504)
        at com.amazonaws.services.kinesis.producer.Daemon.readSome(Daemon.java:553)
        at com.amazonaws.services.kinesis.producer.Daemon.receiveMessage(Daemon.java:243)
        at com.amazonaws.services.kinesis.producer.Daemon.access$500(Daemon.java:61)
        at com.amazonaws.services.kinesis.producer.Daemon$3.run(Daemon.java:298)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

If you put any other credentials, that are currently active in AWS (no matter to which project or user the credentials belong), it will work.

I believe this didn't happen a while ago and has started recently, but not sure when.

If it's of any importance, with S3 and LocalStack, it works no matter which credentials are used.

[semenodm]

when you create KPL cleint try to provide the STS client as well

def producerConfig: KinesisProducerConfiguration = {
      val credentials = new BasicAWSCredentials(kinesisAppConfig.awsKeyId, kinesisAppConfig.awsKey)

      new KinesisProducerConfiguration()
        .setCredentialsProvider(new AWSStaticCredentialsProvider(credentials))
        .setStsPort(kinesisAppConfig.awsPort)
        .setStsEndpoint(kinesisAppConfig.awsHost)
        .setKinesisEndpoint(kinesisAppConfig.awsHost)
        .setKinesisPort(kinesisAppConfig.awsPort)
        .setCloudwatchEndpoint(kinesisAppConfig.awsHost)
        .setCloudwatchPort(kinesisAppConfig.awsPort)
        .setVerifyCertificate(false)
        .setRegion(kinesisAppConfig.awsRegion.id())
    }

[DavidCorral94]

@semenodm thanks, it fixed the problem!