paws-r/paws

Support for configuring endpoint URLs in config file

Closed this issue · 20 comments

dpprdan commented

AWS SDKs recently gained support for configuring endpoint URLs in the config file (docs, blog post)
It would be great if paws could support this as well.

With this ~/aws/config

# https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html
# https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html#ss-endpoints-config

[default]
# "Explicit is better than implicit."

[profile play]
region = us-east-1
services = play-s3

[services play-s3]
S3 =
  endpoint_url = https://play.min.io:9000
  signature_version = s3v4

s3api =
  endpoint_url = https://play.min.io:9000

and this ~/aws/credentials

# https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html

[default]

[play]
aws_access_key_id = Q3AM3UQ867SPQQA43P2F
aws_secret_access_key = zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG

(Config and credentials taken from MinIO Play, as this is a public S3 test server)

I currently see

library(paws.storage)

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    )
  )
)

play$list_buckets()
#> Error: InvalidAccessKeyId (HTTP 403). The AWS Access Key Id you provided does not exist in our records.
# It tries to match a key at AWS not MinIO, IIUC

versus explictly specifing the endpoint with s3(endpoint)

playa <- s3(
  config = list(
    credentials = list(
      profile = "play"
    ),
    endpoint = "https://play.min.io:9000"
  )
)

playa$list_buckets()[["Buckets"]][[1]] # subsetting a long list
#> $Name
#> [1] "0520mdn-test"
#> 
#> $CreationDate
#> [1] "2023-09-07 03:39:25 GMT"
Session info 
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.1 (2023-06-16 ucrt)
#>  os       Windows 10 x64 (build 19044)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language en
#>  collate  German_Germany.utf8
#>  ctype    German_Germany.utf8
#>  tz       Europe/Berlin
#>  date     2023-09-07
#>  pandoc   3.1.7 @ C:/Users/DANIEL/AppData/Local/Pandoc/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version date (UTC) lib source
#>  cli            3.6.1   2023-03-23 [1] CRAN (R 4.3.0)
#>  crayon         1.5.2   2022-09-29 [1] CRAN (R 4.3.0)
#>  curl           5.0.2   2023-08-14 [1] CRAN (R 4.3.1)
#>  digest         0.6.33  2023-07-07 [1] CRAN (R 4.3.1)
#>  evaluate       0.21    2023-05-05 [1] CRAN (R 4.3.0)
#>  fastmap        1.1.1   2023-02-24 [1] CRAN (R 4.3.0)
#>  fs             1.6.3   2023-07-20 [1] CRAN (R 4.3.1)
#>  glue           1.6.2   2022-02-24 [1] CRAN (R 4.3.0)
#>  htmltools      0.5.6   2023-08-10 [1] CRAN (R 4.3.1)
#>  httr           1.4.7   2023-08-15 [1] CRAN (R 4.3.1)
#>  knitr          1.43    2023-05-25 [1] CRAN (R 4.3.0)
#>  lifecycle      1.0.3   2022-10-07 [1] CRAN (R 4.3.0)
#>  magrittr       2.0.3   2022-03-30 [1] CRAN (R 4.3.0)
#>  paws.common    0.6.0   2023-09-06 [1] CRAN (R 4.3.1)
#>  paws.storage * 0.3.0   2023-06-22 [1] CRAN (R 4.3.1)
#>  purrr          1.0.2   2023-08-10 [1] CRAN (R 4.3.1)
#>  R.cache        0.16.0  2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3    1.8.2   2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo           1.25.0  2022-06-12 [1] CRAN (R 4.3.0)
#>  R.utils        2.12.2  2022-11-11 [1] CRAN (R 4.3.0)
#>  R6             2.5.1   2021-08-19 [1] CRAN (R 4.3.0)
#>  Rcpp           1.0.11  2023-07-06 [1] CRAN (R 4.3.1)
#>  reprex         2.0.2   2022-08-17 [1] CRAN (R 4.3.0)
#>  rlang          1.1.1   2023-04-28 [1] CRAN (R 4.3.0)
#>  rmarkdown      2.24    2023-08-14 [1] CRAN (R 4.3.1)
#>  rstudioapi     0.15.0  2023-07-07 [1] CRAN (R 4.3.1)
#>  sessioninfo    1.2.2   2021-12-06 [1] CRAN (R 4.3.0)
#>  styler         1.10.2  2023-08-29 [1] CRAN (R 4.3.1)
#>  vctrs          0.6.3   2023-06-14 [1] CRAN (R 4.3.1)
#>  withr          2.5.0   2022-03-03 [1] CRAN (R 4.3.0)
#>  xfun           0.40    2023-08-09 [1] CRAN (R 4.3.1)
#>  xml2           1.3.5   2023-07-06 [1] CRAN (R 4.3.1)
#>  yaml           2.3.7   2023-01-23 [1] CRAN (R 4.3.0)
#> 
#>  [1] C:/Users/Daniel/AppData/Local/R/win-library/4.3
#>  [2] C:/Program Files/R/R-4.3.1/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Compare AWS CLI:

PS C:\Users\Daniel> aws --profile play s3 ls
2023-09-07 05:39:25 0520mdn-test
[...]

I get the same results with just this simple config

[profile play]
region = us-east-1
endpoint_url = https://play.min.io:9000
dpprdan commented

This probably ought to include support for the corresponding endpoint environment variables, including environment variables for service-specific endpoints.

Or should that be a separate issue?

DyfanJones commented

Hi @dpprdan it looks like not all AWS SDKs support this feature. However that doesn't mean we shouldn't :). I am currently in the process of preparing to release paws 0.4.0. Once that has been released I can add this to the next release of paws.common. In the meantime we welcome all PR's, so if you want to tackle this please raise a PR :) I am more than happy to review and work on it together 😄

dpprdan commented

@DyfanJones Thanks, sounds good! As for a PR: I am still trying to navigate the codebase for (what feels like) basic stuff, so I feel like I am still far away from submitting an (what I'm expecting to be) involved PR like that.

Another question:

According to the docs, the region can be specified in ~/.aws/config.

paws/docs/credentials.md

Lines 372 to 375 in 61e0fb8

You can set the region using the AWS config file in `~/.aws/config`.
[profile my-profile]
region=us-east-2

However, with the config in the original post, I don’t see the region passed through to svc$.internal$config$region

library(paws.storage)

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    )
  )
)

play$.internal$config$region
#> [1] ""
#> attr(,"tags")
#> attr(,"tags")$type
#> [1] "scalar"

Which maybe isn’t so surprising, because the profile is only passed to credentials, not config in the s3() call.
Passing it to config errors.

play <- s3(
  config = list(
    profile = "play"
  )
)
#> Error: invalid name: profile

Is the region stored somewhere else? (Probably not, since both my ~/aws/credentials and e.g. an explicitly, i.e. in the function call, configured endpoint show up in svc$.internal$config?).

Session info 
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.1 (2023-06-16 ucrt)
#>  os       Windows 10 x64 (build 19044)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language en
#>  collate  German_Germany.utf8
#>  ctype    German_Germany.utf8
#>  tz       Europe/Berlin
#>  date     2023-09-08
#>  pandoc   3.1.7 @ C:/Users/DANIEL/AppData/Local/Pandoc/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version date (UTC) lib source
#>  cli            3.6.1   2023-03-23 [1] CRAN (R 4.3.0)
#>  crayon         1.5.2   2022-09-29 [1] CRAN (R 4.3.0)
#>  curl           5.0.2   2023-08-14 [1] CRAN (R 4.3.1)
#>  digest         0.6.33  2023-07-07 [1] CRAN (R 4.3.1)
#>  evaluate       0.21    2023-05-05 [1] CRAN (R 4.3.0)
#>  fastmap        1.1.1   2023-02-24 [1] CRAN (R 4.3.0)
#>  fs             1.6.3   2023-07-20 [1] CRAN (R 4.3.1)
#>  glue           1.6.2   2022-02-24 [1] CRAN (R 4.3.0)
#>  htmltools      0.5.6   2023-08-10 [1] CRAN (R 4.3.1)
#>  httr           1.4.7   2023-08-15 [1] CRAN (R 4.3.1)
#>  knitr          1.43    2023-05-25 [1] CRAN (R 4.3.0)
#>  lifecycle      1.0.3   2022-10-07 [1] CRAN (R 4.3.0)
#>  magrittr       2.0.3   2022-03-30 [1] CRAN (R 4.3.0)
#>  paws.common    0.6.0   2023-09-06 [1] CRAN (R 4.3.1)
#>  paws.storage * 0.4.0   2023-09-06 [1] https://paws-r.r-universe.dev (R 4.3.1)
#>  purrr          1.0.2   2023-08-10 [1] CRAN (R 4.3.1)
#>  R.cache        0.16.0  2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3    1.8.2   2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo           1.25.0  2022-06-12 [1] CRAN (R 4.3.0)
#>  R.utils        2.12.2  2022-11-11 [1] CRAN (R 4.3.0)
#>  R6             2.5.1   2021-08-19 [1] CRAN (R 4.3.0)
#>  Rcpp           1.0.11  2023-07-06 [1] CRAN (R 4.3.1)
#>  reprex         2.0.2   2022-08-17 [1] CRAN (R 4.3.0)
#>  rlang          1.1.1   2023-04-28 [1] CRAN (R 4.3.0)
#>  rmarkdown      2.24    2023-08-14 [1] CRAN (R 4.3.1)
#>  rstudioapi     0.15.0  2023-07-07 [1] CRAN (R 4.3.1)
#>  sessioninfo    1.2.2   2021-12-06 [1] CRAN (R 4.3.0)
#>  styler         1.10.2  2023-08-29 [1] CRAN (R 4.3.1)
#>  vctrs          0.6.3   2023-06-14 [1] CRAN (R 4.3.1)
#>  withr          2.5.0   2022-03-03 [1] CRAN (R 4.3.0)
#>  xfun           0.40    2023-08-09 [1] CRAN (R 4.3.1)
#>  yaml           2.3.7   2023-01-23 [1] CRAN (R 4.3.0)
#> 
#>  [1] C:/Users/Daniel/AppData/Local/R/win-library/4.3
#>  [2] C:/Program Files/R/R-4.3.1/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────
DyfanJones commented

Region is handled by the function paws.common:::get_region. This is called from new_service which is handled by a private method within each category package i.e. paws.storage.

If the values aren't explicitly added to the config i.e.

library(paws.common)
config(credentials(profile = "play"), region = "made-up")

Then the svc$.internal$config will be set to the default values i.e.

library(paws.common)
config()

However you can get the region per profile by calling get_region explicitly. For example:

paws.common:::get_region("play")

paws/paws.common/R/config.R

Lines 291 to 308 in 61e0fb8

get_region <- function(profile = "") {
region <- get_env("AWS_REGION")
if (region != "") {
return(region)
}
# Check if default region is specified
region <- get_env("AWS_DEFAULT_REGION")
if (region != "") {
return(region)
}
region <- check_config_file_region(profile)
if (is.null(region)) stop("No region provided")
return(region)
}

Note: helper config functions config(), credentials() and creds() are all re-exported from paws.common in paws v0.4.0. So they can be called without explicitly importing paws.common.

  • 👀1
dpprdan commented

Region is handled by the function paws.common:::get_region. This is called from new_service which is handled by a private method within each category package i.e. paws.storage.

paws/paws/R/s3_service.R

Lines 233 to 236 in 34b6d72

.s3$service <- function(config = list()) {
handlers <- new_handlers("restxml", "s3")
new_service(.s3$metadata, handlers, config)
}

But shouldn't region show up in play$.internal$config$region then?

In the end, I'd like paws to tell me which config it is actually using. This would be useful for debugging, e.g. when you're getting an InvalidAccessKeyId error, which is really caused by a faulty endpoint config (see my original post).

DyfanJones commented

Sorry for the confusion. No the svc$.internal$config$region wouldn't get populated as new_service only gets called for each operation call for example:

svc$list_buckets <- function () {
    op <- new_operation(name = "ListBuckets", http_method = "GET", 
        http_path = "/", paginator = list(result_key = "Buckets"))
    input <- .s3$list_buckets_input()
    output <- .s3$list_buckets_output()
    config <- get_config()
    svc <- .s3$service(config) # This part calls new_service
    request <- new_request(svc, op, input, output)
    response <- send_request(request)
    return(response)
}

Paws does support some level of debugging. To do this you can set the log_level to 3+ and this will output the http logs when calling aws services.

library(paws)
options(paws.log_level = 3L)

svc_euwest1 <- s3()
svc_useast1 <- s3(config(credentials(profile = "paws")))

svc_euwest1$.internal$config$region
#> [1] ""
#> attr(,"tags")
#> attr(,"tags")$type
#> [1] "scalar"
svc_useast1$.internal$config$region
#> [1] ""
#> attr(,"tags")
#> attr(,"tags")$type
#> [1] "scalar"

resp <- svc_euwest1$list_buckets()
#> INFO [2023-09-11 17:26:43.458]: -> GET / HTTP/1.1
#> -> Host: s3.eu-west-1.amazonaws.com
#> -> Accept-Encoding: deflate, gzip
#> -> Accept: application/json, text/xml, application/xml, */*
#> -> User-Agent: paws/0.6.0 (R4.3.1; darwin20; aarch64)
#> -> Content-Length: 0
#> -> X-Amz-Date: 20230911T162643Z
#> -> X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
#> -> Authorization: AWS4-HMAC-SHA256 Credential=AKIA37AZSIJW6NWKW3HH/20230911/eu-west-1/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date, Signature=bd3a56b7c1813bb7e32ad352898e4bbffa03f9b5db0b7f59f34afd1ed93915d9
#> -> 
#> INFO [2023-09-11 17:26:43.596]: <- HTTP/1.1 200 OK
#> INFO [2023-09-11 17:26:43.596]: <- x-amz-id-2: HchQiR76QDcl3Ywz8Z2FA0lZmIYNVeXCHSnWcobRIm4vRzlmZc4RdUcMVVifhMZWsx0tLt16OCQ=
#> INFO [2023-09-11 17:26:43.596]: <- x-amz-request-id: 8H2FA4BHRM71CW70
#> INFO [2023-09-11 17:26:43.596]: <- Date: Mon, 11 Sep 2023 16:26:44 GMT
#> INFO [2023-09-11 17:26:43.596]: <- Content-Type: application/xml
#> INFO [2023-09-11 17:26:43.596]: <- Transfer-Encoding: chunked
#> INFO [2023-09-11 17:26:43.597]: <- Server: AmazonS3
#> INFO [2023-09-11 17:26:43.597]: <- 
resp <- svc_useast1$list_buckets()
#> INFO [2023-09-11 17:26:44.100]: -> GET / HTTP/1.1
#> -> Host: s3.amazonaws.com
#> -> Accept-Encoding: deflate, gzip
#> -> Accept: application/json, text/xml, application/xml, */*
#> -> User-Agent: paws/0.6.0 (R4.3.1; darwin20; aarch64)
#> -> Content-Length: 0
#> -> X-Amz-Date: 20230911T162643Z
#> -> X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
#> -> Authorization: AWS4-HMAC-SHA256 Credential=AKIAZ4ZAA5DV64XRDOJ4/20230911/us-east-1/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date, Signature=deb90731fad79656079f9a1d07c0d2b53780079b62765147dfb6145bd04addcc
#> -> 
#> INFO [2023-09-11 17:26:44.248]: <- HTTP/1.1 200 OK
#> INFO [2023-09-11 17:26:44.248]: <- x-amz-id-2: neUY2ldRmCbgSt7if25CBt1AzlketS1hziefWKWrN2SBsDRvCp2hd5e+c83ctIUnCmomvUgyaoo=
#> INFO [2023-09-11 17:26:44.248]: <- x-amz-request-id: HQAH6S3J6JDN149F
#> INFO [2023-09-11 17:26:44.248]: <- Date: Mon, 11 Sep 2023 16:26:45 GMT
#> INFO [2023-09-11 17:26:44.249]: <- Content-Type: application/xml
#> INFO [2023-09-11 17:26:44.249]: <- Transfer-Encoding: chunked
#> INFO [2023-09-11 17:26:44.249]: <- Server: AmazonS3
#> INFO [2023-09-11 17:26:44.249]: <-

Created on 2023-09-11 with reprex v2.0.2

  • 👍1
DyfanJones commented

I guess in future iterations we can move the config build to the operation initialisation but that would require a big piece of redesign in the current architecture. We will have to weigh up the pros and cons of making that change :)

  • 👍1
dpprdan commented

@DyfanJones with paws.common 0.6.1 and the config/credentials in the OP, I see

library(paws.storage)

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    )
  )
)

play$list_buckets()
#> Error in get_region(cfgs$credentials$profile): No region provided

Also with explicitly specified endpoint.

playa <- s3(
  config = list(
    credentials = list(
      profile = "play"
    ),
    endpoint = "https://play.min.io:9000"
  )
)

playa$list_buckets()[["Buckets"]][[1]]
#> Error in get_region(cfgs$credentials$profile): No region provided
Session info 
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.1 (2023-06-16 ucrt)
#>  os       Windows 11 x64 (build 22621)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language en
#>  collate  German_Germany.utf8
#>  ctype    German_Germany.utf8
#>  tz       Europe/Berlin
#>  date     2023-09-29
#>  pandoc   3.1.8 @ C:/PROGRA~1/Pandoc/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version date (UTC) lib source
#>  cli            3.6.1   2023-03-23 [1] CRAN (R 4.3.0)
#>  crayon         1.5.2   2022-09-29 [1] CRAN (R 4.3.0)
#>  curl           5.0.2   2023-08-14 [1] CRAN (R 4.3.1)
#>  digest         0.6.33  2023-07-07 [1] CRAN (R 4.3.1)
#>  evaluate       0.22    2023-09-29 [1] CRAN (R 4.3.1)
#>  fastmap        1.1.1   2023-02-24 [1] CRAN (R 4.3.0)
#>  fs             1.6.3   2023-07-20 [1] CRAN (R 4.3.1)
#>  glue           1.6.2   2022-02-24 [1] CRAN (R 4.3.0)
#>  htmltools      0.5.6   2023-08-10 [1] CRAN (R 4.3.1)
#>  httr           1.4.7   2023-08-15 [1] CRAN (R 4.3.1)
#>  knitr          1.44    2023-09-11 [1] CRAN (R 4.3.1)
#>  lifecycle      1.0.3   2022-10-07 [1] CRAN (R 4.3.0)
#>  magrittr       2.0.3   2022-03-30 [1] CRAN (R 4.3.0)
#>  paws.common    0.6.1   2023-09-21 [1] CRAN (R 4.3.1)
#>  paws.storage * 0.4.0   2023-09-06 [1] https://paws-r.r-universe.dev (R 4.3.1)
#>  purrr          1.0.2   2023-08-10 [1] CRAN (R 4.3.1)
#>  R.cache        0.16.0  2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3    1.8.2   2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo           1.25.0  2022-06-12 [1] CRAN (R 4.3.0)
#>  R.utils        2.12.2  2022-11-11 [1] CRAN (R 4.3.0)
#>  R6             2.5.1   2021-08-19 [1] CRAN (R 4.3.0)
#>  Rcpp           1.0.11  2023-07-06 [1] CRAN (R 4.3.1)
#>  reprex         2.0.2   2022-08-17 [1] CRAN (R 4.3.0)
#>  rlang          1.1.1   2023-04-28 [1] CRAN (R 4.3.0)
#>  rmarkdown      2.25    2023-09-18 [1] CRAN (R 4.3.1)
#>  rstudioapi     0.15.0  2023-07-07 [1] CRAN (R 4.3.1)
#>  sessioninfo    1.2.2   2021-12-06 [1] CRAN (R 4.3.0)
#>  styler         1.10.2  2023-08-29 [1] CRAN (R 4.3.1)
#>  vctrs          0.6.3   2023-06-14 [1] CRAN (R 4.3.1)
#>  withr          2.5.1   2023-09-26 [1] CRAN (R 4.3.1)
#>  xfun           0.40    2023-08-09 [1] CRAN (R 4.3.1)
#>  yaml           2.3.7   2023-01-23 [1] CRAN (R 4.3.0)
#> 
#>  [1] C:/Users/Daniel.AK-HAMBURG/AppData/Local/R/win-library/4.3
#>  [2] C:/Program Files/R/R-4.3.1/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

BTW paws.common 0.6.1 hasn't been tagged in https://github.com/paws-r/paws/releases, yet.

DyfanJones commented

I believe if you set us-east-1 region it should work fine :) Ah yes I forgot to add the release, thanks for reminding me. I will add it when I come back from my short holiday :D

  • 👍1
DyfanJones commented

I believe I have a working example for this:

pak::pkg_install("dyfanjones/paws/paws.common@service_endpoint")

# OR

remotes::install_github("dyfanjones/paws/paws.common", ref = "service_endpoint")

@dpprdan please feel free to test this out :)

dpprdan commented

@DyfanJones Unfortunately it is not yet working as (I think) it ought to be

library(paws.storage)

# specifying the profile ought to suffice, IMHO

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    )
  )
)

play$list_buckets()[["Buckets"]][[1]]
#> Error in get_region(cfgs[["credentials"]][["profile"]]): No region provided

# try again with region

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    ),
  region = "us-east-1"
  )
)

play$list_buckets()[["Buckets"]][[1]]
#> Error: InvalidAccessKeyId (HTTP 403). The AWS Access Key Id you provided does not exist in our records.

# I think it's using the AWS endpoint again
# try again with endpoint and region

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    ),
  endpoint = "https://play.min.io:9000",
  region = "us-east-1"
  )
)

play$list_buckets()[["Buckets"]][[1]]
#> Error: InvalidAccessKeyId (HTTP 403). The Access Key Id you provided does not exist in our records.
Session info 
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.1 (2023-06-16 ucrt)
#>  os       Windows 11 x64 (build 22621)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language en
#>  collate  German_Germany.utf8
#>  ctype    German_Germany.utf8
#>  tz       Europe/Berlin
#>  date     2023-10-05
#>  pandoc   3.1.8 @ C:/PROGRA~1/Pandoc/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package      * version    date (UTC) lib source
#>  cli            3.6.1      2023-03-23 [1] CRAN (R 4.3.0)
#>  crayon         1.5.2      2022-09-29 [1] CRAN (R 4.3.0)
#>  curl           5.0.2      2023-08-14 [1] CRAN (R 4.3.1)
#>  digest         0.6.33     2023-07-07 [1] CRAN (R 4.3.1)
#>  evaluate       0.22       2023-09-29 [1] CRAN (R 4.3.1)
#>  fastmap        1.1.1      2023-02-24 [1] CRAN (R 4.3.0)
#>  fs             1.6.3      2023-07-20 [1] CRAN (R 4.3.1)
#>  glue           1.6.2      2022-02-24 [1] CRAN (R 4.3.0)
#>  htmltools      0.5.6      2023-08-10 [1] CRAN (R 4.3.1)
#>  httr           1.4.7      2023-08-15 [1] CRAN (R 4.3.1)
#>  knitr          1.44       2023-09-11 [1] CRAN (R 4.3.1)
#>  lifecycle      1.0.3      2022-10-07 [1] CRAN (R 4.3.0)
#>  magrittr       2.0.3      2022-03-30 [1] CRAN (R 4.3.0)
#>  paws.common    0.6.1.9000 2023-10-05 [1] Github (dyfanjones/paws@854c895)
#>  paws.storage * 0.4.0      2023-09-06 [1] https://paws-r.r-universe.dev (R 4.3.1)
#>  purrr          1.0.2      2023-08-10 [1] CRAN (R 4.3.1)
#>  R.cache        0.16.0     2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3    1.8.2      2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo           1.25.0     2022-06-12 [1] CRAN (R 4.3.0)
#>  R.utils        2.12.2     2022-11-11 [1] CRAN (R 4.3.0)
#>  R6             2.5.1      2021-08-19 [1] CRAN (R 4.3.0)
#>  Rcpp           1.0.11     2023-07-06 [1] CRAN (R 4.3.1)
#>  reprex         2.0.2      2022-08-17 [1] CRAN (R 4.3.0)
#>  rlang          1.1.1      2023-04-28 [1] CRAN (R 4.3.0)
#>  rmarkdown      2.25       2023-09-18 [1] CRAN (R 4.3.1)
#>  rstudioapi     0.15.0     2023-07-07 [1] CRAN (R 4.3.1)
#>  sessioninfo    1.2.2      2021-12-06 [1] CRAN (R 4.3.0)
#>  styler         1.10.2     2023-08-29 [1] CRAN (R 4.3.1)
#>  vctrs          0.6.3      2023-06-14 [1] CRAN (R 4.3.1)
#>  withr          2.5.1      2023-09-26 [1] CRAN (R 4.3.1)
#>  xfun           0.40       2023-08-09 [1] CRAN (R 4.3.1)
#>  xml2           1.3.5      2023-07-06 [1] CRAN (R 4.3.1)
#>  yaml           2.3.7      2023-01-23 [1] CRAN (R 4.3.0)
#> 
#>  [1] C:/Users/Daniel.AK-HAMBURG/AppData/Local/R/win-library/4.3
#>  [2] C:/Program Files/R/R-4.3.1/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

The last example works with paws.common v0.6.0.

DyfanJones commented

Interesting it seems to work for me.

library(paws)
client <- s3(config(credentials(profile = "minio")))
client$list_buckets()
#> $Buckets
#> list()
#> 
#> $Owner
#> $Owner$DisplayName
#> [1] "minio"
#> 
#> $Owner$ID
#> [1] "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4"

Created on 2023-10-05 with reprex v2.0.2

Config SetUp:

[profile minio]
region=eu-west-1
output=json
services = profileminio

[services profileminio]
s3 = 
 endpoint_url = http://localhost:9000

Credential Setup:

[minio]
aws_access_key_id=minioadmin
aws_secret_access_key=minioadmin

How is your config set up?

DyfanJones commented 
library(paws.storage)

# specifying the profile ought to suffice, IMHO

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    )
  )
)

play$list_buckets()[["Buckets"]][[1]]
#> Error in get_region(cfgs[["credentials"]][["profile"]]): No region provided

A) You will have to provide the region either hardcoded, environmental variables or in the config file.

# try again with region

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    ),
  region = "us-east-1"
  )
)

play$list_buckets()[["Buckets"]][[1]]
#> Error: InvalidAccessKeyId (HTTP 403). The AWS Access Key Id you provided does not exist in our records.

# I think it's using the AWS endpoint again

A) You should be able to see what endpoint you are using log level 3:

options(paws.log_level = 3L)
library(paws)
client <- s3(config(credentials(profile = "minio")))
resp <- client$list_buckets()
#> INFO [2023-10-05 15:30:05.858]: -> GET / HTTP/1.1
#> -> Host: localhost:9000
#> -> Accept-Encoding: deflate, gzip
#> -> Accept: application/json, text/xml, application/xml, */*
#> -> User-Agent: paws/0.6.1.9000 (R4.3.1; darwin20; aarch64)
#> -> Content-Length: 0
#> -> X-Amz-Date: 20231005T143005Z
#> -> X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
#> -> Authorization: AWS4-HMAC-SHA256 Credential=minioadmin/20231005/us-east-1/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date, Signature=5e5de9e4633b932fcb28cb197cfd80efe1025bca11f548db9dc579c83eaa44a9
#> -> 
#> INFO [2023-10-05 15:30:05.880]: <- HTTP/1.1 200 OK
#> INFO [2023-10-05 15:30:05.881]: <- Accept-Ranges: bytes
#> INFO [2023-10-05 15:30:05.881]: <- Content-Length: 275
#> INFO [2023-10-05 15:30:05.881]: <- Content-Type: application/xml
#> INFO [2023-10-05 15:30:05.881]: <- Server: MinIO
#> INFO [2023-10-05 15:30:05.881]: <- Strict-Transport-Security: max-age=31536000; includeSubDomains
#> INFO [2023-10-05 15:30:05.881]: <- Vary: Origin
#> INFO [2023-10-05 15:30:05.881]: <- Vary: Accept-Encoding
#> INFO [2023-10-05 15:30:05.881]: <- X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
#> INFO [2023-10-05 15:30:05.881]: <- X-Amz-Request-Id: 178B3C802E363E58
#> INFO [2023-10-05 15:30:05.881]: <- X-Content-Type-Options: nosniff
#> INFO [2023-10-05 15:30:05.881]: <- X-Xss-Protection: 1; mode=block
#> INFO [2023-10-05 15:30:05.881]: <- Date: Thu, 05 Oct 2023 14:30:05 GMT
#> INFO [2023-10-05 15:30:05.881]: <-

Created on 2023-10-05 with reprex v2.0.2

# try again with endpoint and region

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    ),
  endpoint = "https://play.min.io:9000",
  region = "us-east-1"
  )
)

play$list_buckets()[["Buckets"]][[1]]
#> Error: InvalidAccessKeyId (HTTP 403). The Access Key Id you provided does not exist in our records.

A) This code won't hit the new get_service_endpoint function.

https://github.com/paws-r/paws/blob/main/paws.common/R/client.R#L140-L153

I am guessing there might be something wrong with your credentials.

dpprdan commented

Interesting indeed. The config is like in the original post. Well, there is a second profile in the config, but that shouldn't interfere.
🤔 Could it be the - in [services play-s3] that's the stumbling block? Or the spaces in e.g. region = us-east-1? Or the capital S in S3 =?
I'll investigate.

DyfanJones commented

The capital S would do it. It has to be lower case :) https://docs.aws.amazon.com/sdkref/latest/guide/ss-endpoints-table.html

serviceId Service identifier key for shared AWS config file AWS_ENDPOINT_URL_{SERVICE} environment variable
S3 s3 AWS_ENDPOINT_URL_S3

I have used this documentation for the logic in how to find the services in config file and environmental variables :)

So in short config file I expect the service to be lower snake case i.e. Sagemaker Edge -> sagemaker_edge

For environmental variables I expect upper snake case i.e. Sagemaker Edge -> AWS_ENDPOINT_URL_SAGEMAKER_EDGE

dpprdan commented

The capital S would do it. It has to be lower case

FWIW It works with the AWS CLI, see original post (and I just checked again), but yeah, it's not according to spec.

I am now using this exact ~/.aws/config

# https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html
# https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html#ss-endpoints-config

[default]
# "Explicit is better than implicit."

[profile play]
region = us-east-1
services = play-s3

[services play-s3]
s3 =
  endpoint_url = https://play.min.io:9000
  signature_version = s3v4

s3api =
  endpoint_url = https://play.min.io:9000

and these ~/.aws/credentials

# https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html

[default]
# "Explicit is better than implicit."

[play]
aws_access_key_id=Q3AM3UQ867SPQQA43P2F
aws_secret_access_key=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG

and see

library(paws.storage)

options("paws.log_level" = 3L)

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    ),
  region = "us-east-1"
  )
)

play$list_buckets()[["Buckets"]][[1]]
#> INFO [2023-10-05 17:09:51.986]: Profile 'play' not found in 'C:\Users\Daniel/.aws/credentials'
#> INFO [2023-10-05 17:09:51.990]: Profile 'profile play' not found in 'C:\Users\Daniel/.aws/config'
#> INFO [2023-10-05 17:09:51.990]: Unable to obtain access_key_id, secret_access_key or session_token
#> INFO [2023-10-05 17:09:52.014]: Unable to obtain iam role
#> Error: No compatible credentials provided.

when I change ~/.aws/credentials to

[play]
aws_access_key_id=Q3AM3UQ867SPQQA43P2F
aws_secret_access_key=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG

I get

library(paws.storage)

options("paws.log_level" = 3L)

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    ),
  region = "us-east-1"
  )
)

play$list_buckets()[["Buckets"]][[1]]
#> INFO [2023-10-05 17:13:43.518]: -> GET / HTTP/1.1
#> -> Host: s3.amazonaws.com
#> -> Accept-Encoding: deflate, gzip
#> -> Accept: application/json, text/xml, application/xml, */*
#> -> User-Agent: paws/0.6.1.9000 (R4.3.1; mingw32; x86_64)
#> -> Content-Length: 0
#> -> X-Amz-Date: 20231005T151343Z
#> -> X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
#> -> Authorization: AWS4-HMAC-SHA256 Credential=Q3AM3UQ867SPQQA43P2F/20231005/us-east-1/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date, Signature=4e8c5e615cec5c105e70b537a4f9dc91bd8293d44ba6c055a22af1dc27369a1d
#> -> 
#> INFO [2023-10-05 17:13:43.628]: <- HTTP/1.1 403 Forbidden
#> INFO [2023-10-05 17:13:43.629]: <- x-amz-request-id: 9G2G0NV3TETN3DN5
#> INFO [2023-10-05 17:13:43.629]: <- x-amz-id-2: dpkQIKowh+PeXfzZ/Cm+dvQZD9cr1TDhOj3dxfMkDn39B7mXHibY71XfdAr0cRt8LjN29vCK8lk=
#> INFO [2023-10-05 17:13:43.629]: <- Content-Type: application/xml
#> INFO [2023-10-05 17:13:43.629]: <- Transfer-Encoding: chunked
#> INFO [2023-10-05 17:13:43.629]: <- Date: Thu, 05 Oct 2023 15:13:43 GMT
#> INFO [2023-10-05 17:13:43.629]: <- Server: AmazonS3
#> INFO [2023-10-05 17:13:43.630]: <- 
#> Error: InvalidAccessKeyId (HTTP 403). The AWS Access Key Id you provided does not exist in our records.

Comments are allowed AFAICT.

(region isn't read from the profile).

DyfanJones commented

@dpprdan Thanks for sharing your config and credentials file. I have updated the read_ini function to better handle comments. Please install latest version and let me know how you get on.

remotes::install_github("dyfanjones/paws/paws.common", ref = "service_endpoint")
  • 🚀1
dpprdan commented

Success!

library(paws.storage)

options("paws.log_level" = 3L)

play <- s3(
  config = list(
    credentials = list(
      profile = "play"
    )
  )
)

play$list_buckets()[["Buckets"]][[1]]
#> INFO [2023-10-05 18:17:37.574]: -> GET / HTTP/1.1
#> -> Host: play.min.io:9000
#> -> Accept-Encoding: deflate, gzip
#> -> Accept: application/json, text/xml, application/xml, */*
#> -> User-Agent: paws/0.6.1.9000 (R4.3.1; mingw32; x86_64)
#> -> Content-Length: 0
#> -> X-Amz-Date: 20231005T161737Z
#> -> X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
#> -> Authorization: AWS4-HMAC-SHA256 Credential=Q3AM3UQ867SPQQA43P2F/20231005/us-east-1/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date, Signature=333cb8b3bc2025c6ac1d98af8e69f411a2359d7deff07a515707b3121b1b54a0
#> -> 
#> INFO [2023-10-05 18:17:37.726]: <- HTTP/1.1 200 OK
#> INFO [2023-10-05 18:17:37.726]: <- Content-Encoding: gzip
#> INFO [2023-10-05 18:17:37.726]: <- Content-Type: application/xml
#> INFO [2023-10-05 18:17:37.727]: <- Server: MinIO
#> INFO [2023-10-05 18:17:37.727]: <- Strict-Transport-Security: max-age=31536000; includeSubDomains
#> INFO [2023-10-05 18:17:37.727]: <- Vary: Origin
#> INFO [2023-10-05 18:17:37.727]: <- Vary: Accept-Encoding
#> INFO [2023-10-05 18:17:37.727]: <- X-Amz-Bucket-Region: us-east-1
#> INFO [2023-10-05 18:17:37.727]: <- X-Amz-Id-2: 3e996b2f640d7e065d3a5c4e39a5538cefb82e3e77771990265e4698d8681eac
#> INFO [2023-10-05 18:17:37.727]: <- X-Amz-Request-Id: 178B425E563F014B
#> INFO [2023-10-05 18:17:37.728]: <- X-Content-Type-Options: nosniff
#> INFO [2023-10-05 18:17:37.728]: <- X-Xss-Protection: 1; mode=block
#> INFO [2023-10-05 18:17:37.728]: <- Date: Thu, 05 Oct 2023 16:17:37 GMT
#> INFO [2023-10-05 18:17:37.728]: <- Content-Length: 1963
#> INFO [2023-10-05 18:17:37.728]: <- 
#> $Name
#> [1] "00test"
#> 
#> $CreationDate
#> [1] "2023-10-05 14:15:45 GMT"
  • 👍1
  • 🎉1
DyfanJones commented

WOOHOO!!!!! Finally!!!

  • ❤️1
dpprdan commented

With #684 & #685 merged, I think this can be closed.
Thanks @DyfanJones for implementing this!

  • 👍1