kedro-org/kedro

`credentials` param in `kedrio.io.csv_s3.CSVS3DataSet` breaks since s3fs 0.4.1.

Closed this issue · 1 comments

michaelchia commented

Description

Using dict keys 'aws_access_key_id' and 'aws_secret_access_key' in the credentials param in kedrio.io.csv_s3.CSVS3DataSet causes errors since s3fs 0.4.1.
I have made a pull request to s3fs to fix it on their side. But this report is just to let you know in case my PR is rejected or ignored. And even if the PR is accepted, since s3fs 0.4.1 & 0.4.2 are already out perhaps you'll need to be aware of this if your users report such issues and to fix your requirements.txt to avoid it. Either s3fs<=0.4.0 or hopefully >=0.4.3 if my PR gets accepted by next release.

Context

I am unable to pass credentials to CSVS3DataSet as per its docstring. And would have to downgrade s3fs to get it to work.

Steps to Reproduce

  1. Upgrade s3fs to 0.4.1 or 0.4.2 (current latest)
  2. CSVS3DataSet(filepath="bucket/test.csv", credentials={"aws_access_key_id": "foo", "aws_secret_access_key": "bar"})

Expected Result

Instantiate dataset obj.

Actual Result

Raised TypeError

TypeError: create_client() got multiple values for keyword argument 'aws_access_key_id'
# from s3fs.core

Your Environment

s3fs==0.4.1
kedro==0.15.8

michaelchia commented

They accepted my PR. Bug should be fixed in next version. You may close this issue.
fsspec/s3fs#308

  • 👍2