homr() gives Error: Bad gateway (HTTP 502)
windnick opened this issue · 6 comments
Recent changes to NOAA server cause HTTP 502 error. Blog posts suggest this can be avoided by forcing HTTP version 1.0
Session Info
R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19041)
Matrix products: default
locale:
[1] LC_COLLATE=English_United Kingdom.1252
[2] LC_CTYPE=English_United Kingdom.1252
[3] LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United Kingdom.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] rnoaa_1.1.0 RevoUtils_11.0.2 RevoUtilsMath_11.0.0
loaded via a namespace (and not attached):
[1] Rcpp_1.0.5 xml2_1.3.2 magrittr_1.5 rappdirs_0.3.1
[5] tidyselect_1.1.0 munsell_0.5.0 colorspace_1.4-1 R6_2.3.0
[9] rlang_0.4.6 hoardr_0.5.2 dplyr_1.0.0 grid_4.0.2
[13] gtable_0.3.0 ellipsis_0.3.1 digest_0.6.25 httpcode_0.3.0
[17] tibble_3.0.2 lifecycle_0.2.0 crayon_1.3.4 gridExtra_2.3
[21] tidyr_1.1.0 purrr_0.3.4 ggplot2_3.3.2 vctrs_0.3.1
[25] triebeard_0.3.0 curl_3.3 crul_0.9.0 glue_1.4.1
[29] compiler_4.0.2 pillar_1.4.6 urltools_1.7.3 generics_0.0.2
[33] scales_1.1.1 XML_3.99-0.4 lubridate_1.7.9 jsonlite_1.5
[37] pkgconfig_2.0.3 </strong></summary>I see you're using an older version of rnoaa, try upating to latest rnoaa on cran and try again - if that still doens't work share a reproducible example - see https://reprex.tidyverse.org/ if you aren't sure how to do it
Argh! Default ROpen repos is 7 months out of date! Installed rnoaa_1.3.0 from GitHub, but same HTTP 502 error. Presume one or more dependencies are also out of date. Which?
library(rnoaa)
sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19041)
Matrix products: default
locale:
[1] LC_COLLATE=English_United Kingdom.1252
[2] LC_CTYPE=English_United Kingdom.1252
[3] LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United Kingdom.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] rnoaa_1.3.0 RevoUtils_11.0.2 RevoUtilsMath_11.0.0
loaded via a namespace (and not attached):
[1] Rcpp_1.0.5 xml2_1.3.2 magrittr_1.5 rappdirs_0.3.1
[5] tidyselect_1.1.0 munsell_0.5.0 colorspace_1.4-1 R6_2.3.0
[9] rlang_0.4.6 hoardr_0.5.2 dplyr_1.0.0 tools_4.0.2
[13] grid_4.0.2 gtable_0.3.0 ellipsis_0.3.1 digest_0.6.25
[17] httpcode_0.3.0 tibble_3.0.2 lifecycle_0.2.0 crayon_1.3.4
[21] gridExtra_2.3 tidyr_1.1.0 purrr_0.3.4 ggplot2_3.3.2
[25] vctrs_0.3.1 curl_3.3 crul_0.9.0 glue_1.4.1
[29] compiler_4.0.2 pillar_1.4.6 generics_0.0.2 scales_1.1.1
[33] XML_3.99-0.4 lubridate_1.7.9 jsonlite_1.5 pkgconfig_2.0.3
x<-homr(quid="ICAO:KIAD")
Error: Bad Gateway (HTTP 502)
Lets see if it's a general problem with your setup. do any of these work?
library(rnoaa)
ghcnd(stationid = "AGE00147704")
lcd(station = "01338099999", year = 2015)
bsw(date = "2017-10-01")Thanks for your help. Response to your code is:
library(rnoaa)
ghcnd(stationid = "AGE00147704")
using cached file: C:\Users\swift\AppData\Local\Cache/R/noaa_ghcnd/AGE00147704.dly
date created (size, mb): 2021-02-05 10:33:37 (0.487)
A tibble: 963 x 128
id year month element VALUE1 MFLAG1 QFLAG1 SFLAG1 VALUE2 MFLAG2 QFLAG2
1 AGE0~ 1909 11 TMAX NA " " " " " " NA " " " "
2 AGE0~ 1909 11 TMIN NA " " " " " " NA " " " "
3 AGE0~ 1909 11 PRCP NA " " " " " " NA " " " "
4 AGE0~ 1909 12 TMAX 170 " " " " "E" 160 " " " "
5 AGE0~ 1909 12 TMIN 90 " " " " "E" 100 " " " "
6 AGE0~ 1909 12 PRCP 24 " " " " "E" 0 " " " "
7 AGE0~ 1910 1 TMAX 140 " " " " "E" 120 " " " "
8 AGE0~ 1910 1 TMIN 100 " " " " "E" 80 " " " "
9 AGE0~ 1910 1 PRCP 82 " " " " "E" 176 " " " "
10 AGE0~ 1910 2 TMAX 110 " " " " "E" 140 " " " "
... with 953 more rows, and 117 more variables: SFLAG2 , VALUE3 ,
MFLAG3 , QFLAG3 , SFLAG3 , VALUE4 , MFLAG4 ,
QFLAG4 , SFLAG4 , VALUE5 , MFLAG5 , QFLAG5 ,
SFLAG5 , VALUE6 , MFLAG6 , QFLAG6 , SFLAG6 ,
VALUE7 , MFLAG7 , QFLAG7 , SFLAG7 , VALUE8 ,
MFLAG8 , QFLAG8 , SFLAG8 , VALUE9 , MFLAG9 ,
QFLAG9 , SFLAG9 , VALUE10 , MFLAG10 , QFLAG10 ,
SFLAG10 , VALUE11 , MFLAG11 , QFLAG11 , SFLAG11 ,
VALUE12 , MFLAG12 , QFLAG12 , SFLAG12 , VALUE13 ,
MFLAG13 , QFLAG13 , SFLAG13 , VALUE14 , MFLAG14 ,
QFLAG14 , SFLAG14 , VALUE15 , MFLAG15 , QFLAG15 ,
SFLAG15 , VALUE16 , MFLAG16 , QFLAG16 , SFLAG16 ,
VALUE17 , MFLAG17 , QFLAG17 , SFLAG17 , VALUE18 ,
MFLAG18 , QFLAG18 , SFLAG18 , VALUE19 , MFLAG19 ,
QFLAG19 , SFLAG19 , VALUE20 , MFLAG20 , QFLAG20 ,
SFLAG20 , VALUE21 , MFLAG21 , QFLAG21 , SFLAG21 ,
VALUE22 , MFLAG22 , QFLAG22 , SFLAG22 , VALUE23 ,
MFLAG23 , QFLAG23 , SFLAG23 , VALUE24 , MFLAG24 ,
QFLAG24 , SFLAG24 , VALUE25 , MFLAG25 , QFLAG25 ,
SFLAG25 , VALUE26 , MFLAG26 , QFLAG26 , SFLAG26 ,
VALUE27 , MFLAG27 , QFLAG27 , ...
Warning message:
...is not empty.
We detected these problematic arguments:
needs_dots
These dots only exist to allow future extensions and should be empty.
Did you misspecify an argument?
lcd(station = "01338099999", year = 2015)
using cached file: C:\Users\swift\AppData\Local\Cache/R/noaa_lcd/2015_01338099999.csv
date created (size, mb): 2021-02-05 10:34:08 (4.177)
A tibble: 8,096 x 121
station date latitude longitude elevation name report_type source
1 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4
2 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4
3 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4
4 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4
5 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4
6 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4
7 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4
8 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4
9 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4
10 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4
... with 8,086 more rows, and 113 more variables:
hourlyaltimetersetting , hourlydewpointtemperature ,
hourlydrybulbtemperature , hourlyprecipitation ,
hourlypresentweathertype , hourlypressurechange ,
hourlypressuretendency , hourlyrelativehumidity ,
hourlyskyconditions , hourlysealevelpressure ,
hourlystationpressure , hourlyvisibility ,
hourlywetbulbtemperature , hourlywinddirection ,
hourlywindgustspeed , hourlywindspeed , sunrise ,
sunset , dailyaveragedewpointtemperature ,
dailyaveragedrybulbtemperature , dailyaveragerelativehumidity ,
dailyaveragesealevelpressure , dailyaveragestationpressure ,
dailyaveragewetbulbtemperature , dailyaveragewindspeed ,
dailycoolingdegreedays ,
dailydeparturefromnormalaveragetemperature ,
dailyheatingdegreedays , dailymaximumdrybulbtemperature ,
dailyminimumdrybulbtemperature , dailypeakwinddirection ,
dailypeakwindspeed , dailyprecipitation , dailysnowdepth ,
dailysnowfall , dailysustainedwinddirection ,
dailysustainedwindspeed , dailyweather , monthlyaveragerh ,
monthlydayswithgt001precip , monthlydayswithgt010precip ,
monthlydayswithgt32temp , monthlydayswithgt90temp ,
monthlydayswithlt0temp , monthlydayswithlt32temp ,
monthlydeparturefromnormalaveragetemperature ,
monthlydeparturefromnormalcoolingdegreedays ,
monthlydeparturefromnormalminimumtemperature ,
monthlydeparturefromnormalprecipitation ,
monthlydewpointtemperature , monthlygreatestprecip ,
monthlygreatestprecipdate , monthlygreatestsnowdepth ,
monthlygreatestsnowdepthdate , monthlygreatestsnowfall ,
monthlygreatestsnowfalldate , monthlymaxsealevelpressurevalue ,
monthlymaxsealevelpressurevaluedate ,
monthlymaxsealevelpressurevaluetime , monthlymaximumtemperature ,
monthlymeantemperature , monthlyminsealevelpressurevalue ,
monthlyminsealevelpressurevaluedate ,
monthlyminsealevelpressurevaluetime , monthlyminimumtemperature ,
monthlysealevelpressure , monthlystationpressure ,
monthlytotalliquidprecipitation , monthlytotalsnowfall ,
monthlywetbulb , awnd , cdsd , cldd , dsnw ,
hdsd , htdd , normalscoolingdegreeday ,
normalsheatingdegreeday , shortdurationenddate005 ,
shortdurationenddate010 , shortdurationenddate015 ,
shortdurationenddate020 , shortdurationenddate030 ,
shortdurationenddate045 , shortdurationenddate060 ,
shortdurationenddate080 , shortdurationenddate100 ,
shortdurationenddate120 , shortdurationenddate150 ,
shortdurationenddate180 , shortdurationprecipitationvalue005 ,
shortdurationprecipitationvalue010 ,
shortdurationprecipitationvalue015 ,
shortdurationprecipitationvalue020 ,
shortdurationprecipitationvalue030 ,
shortdurationprecipitationvalue045 ,
shortdurationprecipitationvalue060 ,
shortdurationprecipitationvalue080 ,
shortdurationprecipitationvalue100 ,
shortdurationprecipitationvalue120 , ...
Warning message:
...is not empty.
We detected these problematic arguments:
needs_dots
These dots only exist to allow future extensions and should be empty.
Did you misspecify an argument?
bsw(date = "2017-10-01")
Error: Could not resolve host: eclipse.ncdc.noaa.govhomr(quid="ICAO:KIAD")
Error: Bad Gateway (HTTP 502)
homr(quid="COOP:448903")
Error: Bad Gateway (HTTP 502)
homr(quid="WBAN:93738")
Error: Bad Gateway (HTTP 502)sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19041)
Matrix products: default
locale:
[1] LC_COLLATE=English_United Kingdom.1252
[2] LC_CTYPE=English_United Kingdom.1252
[3] LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United Kingdom.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] rnoaa_1.3.0 RevoUtils_11.0.2 RevoUtilsMath_11.0.0
loaded via a namespace (and not attached):
[1] Rcpp_1.0.5 pillar_1.4.6 compiler_4.0.2 tools_4.0.2
[5] digest_0.6.25 jsonlite_1.5 lubridate_1.7.9 lifecycle_0.2.0
[9] tibble_3.0.2 gtable_0.3.0 pkgconfig_2.0.3 rlang_0.4.6
[13] cli_2.0.2 crul_0.9.0 curl_3.3 gridExtra_2.3
[17] dplyr_1.0.0 xml2_1.3.2 generics_0.0.2 vctrs_0.3.1
[21] rappdirs_0.3.1 triebeard_0.3.0 grid_4.0.2 tidyselect_1.1.0
[25] glue_1.4.1 httpcode_0.3.0 data.table_1.12.8 R6_2.3.0
[29] fansi_0.4.1 XML_3.99-0.4 ggplot2_3.3.2 purrr_0.3.4
[33] tidyr_1.1.0 hoardr_0.5.2 magrittr_1.5 scales_1.1.1
[37] urltools_1.7.3 ellipsis_0.3.1 assertthat_0.2.1 colorspace_1.4-1
[41] utf8_1.1.4 munsell_0.5.0 crayon_1.3.4
Okay, i think it's just that some NOAA services are very unreliable; sometimes they work and sometimes they do not work. If homr never works for you that suggests a different problem, but if it sometimes works then we can't do much about that since that's a NOAA problem that we can't solve
Well, I guess the only solution when it's Someone Else's Problem is the write one's own code.
The following will retrieve most useful ASOS station metadata from NOAA HOMR server.
library(stringr)
library(lubridate)
metaASOS=function(qid)
{#----------------------------------- local functions -------------------------------------
numextract <- function(string){str_extract(string, "\-\d+\.\d*")}
# return n characters after expr from str
Seg=function(expr,n,str)
{if (length(str)>1) str<-paste(str,collapse="")
y<-gregexpr(expr,str)
i<-unlist(y)+nchar(expr)
ni<-length(i)
s<-character()
for (j in c(1:ni)) {s<-append(s,substr(str,i[j],i[j]+n-1))}
return(s)
}
# return integer within n characters of expr in str
Id<-function(expr,n,str)
{s<-Seg(expr,n,str)[1]
s<-gsub("[^0-9]", "", s)
return(as.numeric(s))
}
# return first character string within n characters of expr in str
Str<-function(expr,n,str)
{s<-Seg(expr,n,str)[1]
i<-gregexpr('[A-Z]',s,ignore.case=TRUE)[[1]][1]
s<-substr(s,i,n)
i<-gregexpr('"',s)[[1]][1]
return(substr(s,1,i-1))
}
#------------------------------------ get homr page for qid ----------------------------------
url<-paste('http://ncdc.noaa.gov/homr/services/station/search?qid=',qid,'&date=all',sep="")
options(warn=-1)
homr<-readLines(url)
options(warn=0)
#------------------------------------ get current station name, etc ---------------------------
expr<-'","names":\[\{"name"'
StnName<-Str(expr,100,homr)
county<-Str('county',100,homr)
s<-Seg('stateProvinces',100,homr); state<-Str('stateProvince',100,s)
#---------------------------------------- UTC offset --------------------------------------
s<-Seg('utcOffsets',25,homr); Hoffset<-as.numeric(numextract(s))[1]
------------------------------------ get station Id codes ------------------------------
NCDC<-Id('ncdcStnId',20,homr)
COOP<-Id('"COOP","id":"',10,homr)
WBAN<-Id('"WBAN","id":"',10,homr)
WMO<-Id('"WMO","id":"',10,homr)
ICAO<-Seg('"ICAO","id":"',4,homr)[1]
FAA<-substr(ICAO,2,4)
#---------------------------------------- get latitudes and longitudes ------------------
expr<-'latitude_dec":"'
s<-Seg(expr,100,homr)
s<-s[grepl("beginDate",s)] # latitude strings with begin date
latitudes<-as.numeric(numextract(substr(s,1,15)))
dates<-Seg('beginDate":"',10,s)
latDates<-make_datetime(as.numeric(substr(dates,1,4)),
as.numeric(substr(dates,6,7)),as.numeric(substr(dates,9,10)))
expr<-'longitude_dec":"'
s<-Seg(expr,100,homr)
s<-s[grepl("beginDate",s)] # longitude strings with begin date
longitudes<-as.numeric(numextract(substr(s,1,15)))
dates<-Seg('beginDate":"',10,s)
longDates<-make_datetime(as.numeric(substr(dates,1,4)),
as.numeric(substr(dates,6,7)),as.numeric(substr(dates,9,10)))
--------------------------------------- get Sonic install date -----------------------
expr<-'ANEMSONIC'; s<-Seg(expr,50,homr)
expr<-'beginDate'; s<-Seg(expr,15,s)
s<-substr(tail(s,n=1),4,13) # last one found is earliest
sonicDate<-make_datetime(as.numeric(substr(s,1,4)),
as.numeric(substr(s,6,7)),as.numeric(substr(s,9,10)))
--------------------------------------- get elevations --------------------------------
expr<-'elevationType'; s<-Seg(expr,120,homr)
expr<-'GROUND'; s<-Seg(expr,120,s)
expr<-'elevationMeters'; s<-Seg(expr,80,s)
elevations<-as.numeric(numextract(substr(s,1,15)))
dates<-Seg('beginDate":"',10,s)
elevDates<-make_datetime(as.numeric(substr(dates,1,4)),
as.numeric(substr(dates,6,7)),as.numeric(substr(dates,9,10)))
-------------------------------------- compile data list -------------------------------
return(list(StnName=StnName,county=county,state=state,utcOffset=Hoffset,
NCDC=NCDC,COOP=COOP,WBAN=WBAN,WMO=WMO,ICAO=ICAO,FAA=FAA,latitudes=latitudes,
latDates=latDates,longitudes=longitudes,longDates=longDates,elevations=elevations,elevDates=elevDates,
sonicDate=sonicDate))
}