geneorama/RSocrata

Provides easier interaction with Socrata open data portals http://dev.socrata.com. Users can provide a 'Socrata' data set resource URL, or a 'Socrata' Open Data API (SoDA) web query, or a 'Socrata' "human-friendly" URL, returns an R data frame. Converts dates to 'POSIX' format. Manages throttling by 'Socrata'.

RSocrata

Master


Dev


A tool for downloading and uploading Socrata datasets

Provided with a URL to a dataset resource published on a Socrata webserver, or a Socrata SoDA (Socrata Open Data Application Program Interface) web API query, or a Socrata "human-friendly" URL, read.socrata() returns an R data frame. Converts dates to POSIX format. Supports CSV and JSON download file formats from Socrata. Manages the throttling of data returned from Socrata and allows users to provide an application token. Supports SoDA query parameters in the URL string for further filtering, sorting, and queries. Upload data to Socrata data portals using "upsert" and "replace" methods.

Use ls.socrata() to list all datasets available on a Socrata webserver.

testthat test coverage.

Example: Reading SoDA valid URLs

earthquakesDataFrame <- read.socrata("http://soda.demo.socrata.com/resource/4334-bgaj.csv")
nrow(earthquakesDataFrame) # 1007 (two "pages")
class(earthquakesDataFrame$Datetime[1]) # POSIXlt

Example: Reading "human-readable" URLs

earthquakesDataFrame <- read.socrata("https://soda.demo.socrata.com/dataset/USGS-Earthquakes-for-2012-11-01-API-School-Demo/4334-bgaj")
nrow(earthquakesDataFrame) # 1007 (two "pages")
class(earthquakesDataFrame$Datetime[1]) # POSIXlt

Example: Using API key to read datasets

token <- "ew2rEMuESuzWPqMkyPfOSGJgE"
earthquakesDataFrame <- read.socrata("http://soda.demo.socrata.com/resource/4334-bgaj.csv", app_token = token)
nrow(earthquakesDataFrame)

Example: Download private datasets from portal

# Store user email and password
socrataEmail <- Sys.getenv("SOCRATA_EMAIL", "mark.silverberg+soda.demo@socrata.com")
socrataPassword <- Sys.getenv("SOCRATA_PASSWORD", "7vFDsGFDUG")

privateResourceToReadCsvUrl <- "https://soda.demo.socrata.com/resource/a9g2-feh2.csv" # dataset

read.socrata(url = privateResourceToReadCsvUrl, email = socrataEmail, password = socrataPassword)

Example: List all datasets on portal

allSitesDataFrame <- ls.socrata("https://soda.demo.socrata.com")
nrow(allSitesDataFrame) # Number of datasets
allSitesDataFrame$title # Names of each dataset

Example: Upload data to portal

# Store user email and password
socrataEmail <- Sys.getenv("SOCRATA_EMAIL", "mark.silverberg+soda.demo@socrata.com")
socrataPassword <- Sys.getenv("SOCRATA_PASSWORD", "7vFDsGFDUG")

datasetToAddToUrl <- "https://soda.demo.socrata.com/resource/xh6g-yugi.json" # dataset
 
# Generate some data
x <- sample(-1000:1000, 1)
y <- sample(-1000:1000, 1)
df_in <- data.frame(x,y)
 
# Upload to Socrata
write.socrata(df_in,datasetToAddToUrl,"UPSERT",socrataEmail,socrataPassword)

Issues

Please report issues, request enhancements or fork us at the City of Chicago github.

Contributing

If you would like to contribute to this project, please see the contributing documentation and the product roadmap.