NerdWalletOSS/kinesis-python

Pass-in and override endpoint_url in boto3 Kinesis client and DynamoDB resource

nicholasball opened this issue · 1 comments

nicholasball commented

Hi,

We are looking to use this library in our project and would be happy to pick up some of the project maintenance if needed.

However we do need a submit a PR, which adds the possibility to pass in an endpoint_url for both the Kinesis (consumer.KinesisConsumer) and DynamoDB (state.DynamoDB) setup, along with a session. We need this so we can work with Localstack.

I'll put together the PR but just wanted to check that you're open to this beforehand!

Thanks

cjgordon commented

Hi @nicholasball

I had this same issue trying to test this project with LocalStack.

I have come up with the following solution.

  1. Install this python lib: https://github.com/localstack/localstack-python-client
  2. Pass custom boto3 session into KinesisConsumer constructor - example consumer below:
import pprint
import localstack_client.session
from kinesis.consumer import KinesisConsumer

session = localstack_client.session.Session()

consumer = KinesisConsumer(stream_name='Logs-Dev', boto3_session=session)
for message in consumer:
    pprint.pprint(message)

If you need to customize the endpoint URL further (use url other than LocalStack default) set environment variable LOCALSTACK_HOST to the hostname require, USE_SSL to set http or https.

Other customization may require editing:
https://github.com/localstack/localstack-python-client/blob/master/localstack_client/config.py

Cheers
Chris

  • 👍1