Reading QS file from website inside R session

[thomasjwood]

Brilliant package, use it all the time. Thanks for all your hard work.

I like to store .rds files I'm recurrently using on some remote website. in those situations I would do something like (this is not a real file):

library(tidyverse)
library(qs)

"https://github.com/user_name/some_repo/raw/master/not_a_real_file.rds" %>% 
  url %>% 
  gzcon %>% 
  readRDS

Is it possible to do the same with a *.qs file? I tried this (a real file this time)

"https://github.com/thomasjwood/ps7160/raw/master/five_hundred_k.qs" %>% 
  url %>% 
  gzcon %>% 
  qread

and get the error

Error in qs::qread(., strict = F) : 
  Expecting a single string value: [type=integer; extent=1].

is this functionality possible with qs files?

Thanks again!

[traversc]

At once point qs had generic R connections, but due to CRAN policy, you're not allowed to use them within C++ code.

You could create a wrapper, e.g.:

library(qs)

qread_http <- function(address, buffer_size = 1048576, ...) {
  con <- file(address, raw=TRUE, open = "rb")
  results <- list()
  i <- 1
  while(length(res <- readBin(con, what = "raw", n = buffer_size)) != 0) {
    results[[i]] <- res
    i <- i + 1
  }
  close(con)
  results <- do.call(c, results)
  qdeserialize(results, ...)
}

x <- qread_http("https://github.com/thomasjwood/ps7160/raw/master/five_hundred_k.qs")

Would that work? I can add that into the package itself if you'd like.

[thomasjwood]

Brilliant! Works perfectly! Thanks!

I'd love for this function to be included (although I don't know how typical this use case is...)

Thanks again!