paws-r/paws

Do you have suggestions for reading many csv files

Closed this issue · 2 comments

I would like to do something like a glob so I can read many csv files from a directory. I was curious was kind of solutions paws has for this. Currently, when I run get_object I can't pass something like dir/*.csv as a key. I have to pass dir/file1.csv.

There is no multiple download as far as I know. I think the most straightforward way would be to call list_objects_v2, filter the result however you want, and download each one at a time. It might looks something like this (not tested). Hopefully this helps.

objects <- s3$list_objects_v2(Bucket = "my-bucket")
to_download <- Filter(function(x) grepl(".*\\.csv", x), a$Contents)
for (file in to_download) {
   ...
}

Hopefully that helps

Closing. Let us know if this is still an issue!