noctua: data not shown in s3 if overwrite=TRUE, but shown if overwrite=FALSE and append=TRUE
DyfanJones opened this issue · 1 comments
DyfanJones commented
Hi Dyfan,
I am uploading data into s3 buckets and using for that purpose your package “noctua” -- which is great. But, there seems to be one thing I have come across which I have not yet managed after endless testing using this code snippet:
### Connect to database
con <- dbConnect(noctua::athena()
, aws_access_key_id = aws_access_key_id
, aws_secret_access_key = aws_secret_access_key
, aws_session_token = aws_session_token
, s3_staging_dir = 's3://bucket/path/to/folder/'
, region = 'us-east-1'
, schema = 'schema'
, work_group = 'abc_ef_gh_jklmnoqr'
)
### Upload to Athena
dbWriteTable(conn = con
, name = schema.aaa_bbb_cccccc'
, value = AbcDt
, overwrite = ifelse(r==1, TRUE, FALSE)
, append = ifelse(r==1, FALSE, TRUE)
, file.type = 'json'
, partition=c("ZEITSTEMPEL" = format(Sys.Date(), "%Y%m%d"))
, s3.location = 's3://bucket/path/to/folder/'
)
I am using this code in a loop. By doing so, I have noticed that for the first loop (r==1), the data can be queried in Athena but does not seem to be available or visible in s3. For the following loops (r>1), i.e. if the data is appended, the data is shown in s3, too.
Is this anything you can reproduce?
Many thanks in advance (and for the great work you have put into the package)
Christian
DyfanJones commented
This issue has been fixed through emails