AustralianAntarcticDivision/blueant

Trouble downloading sea ice .nc files from "CERSAT SSM/I sea ice concentration"

jasmin4689 opened this issue · 5 comments

jasmin4689 commented

I am trying to download data from "CERSAT SSM/I sea ice concentration".

I have downloaded multiple data sets using the "blueant" package containing .nc files to extract metadata from these files for a specific Antarctic region. It has worked with no problem until I started downloading this dataset: "CERSAT SSM/I sea ice concentration".

I am running this code in R Studio version 1.1.453 on a MacBook Pro from 2014 with the "High Sierra" operating system. I have successfully downloaded the data from "CERSAT SSM/I sea ice concentration" for the years "2007" and "2008" but now it is not working for "2009" or any other years if I use the "accept_follow" command. It also won't download the entire data set if I run the code below without the "accept_follow" command.

The code I have used is the following:

#### Setup db Sea Ice ####

mysrc2 <- sources("CERSAT SSM/I sea ice concentration") 

## First make sure that the data source doesn't already have an 
accept_follow parameter defined
"accept_follow" %in% names(mysrc2$method[[1]])

## nope, so we can safely go ahead and impose our own
mysrc2$method[[1]]$accept_follow <- "2009"

# Download the data to local folder
cf <- bb_config(local_file_root = "./Data") %>% bb_add(mysrc2)
bb_sync(cf, verbose = TRUE)

The error message that comes up when running the code is the following:

Fri Mar 29 09:56:32 2019
Synchronizing dataset: CERSAT SSM/I sea ice concentration
Source URL ftp://ftp.ifremer.fr/ifremer/cersat/products/gridded/psi-
concentration/data/antarctic/daily/netcdf/

this dataset path is:
/Users/jasmin4689/Documents/Uni/Data/JG_PhD_Data/JG_R/Temporal_Comparison/Data/ftp .ifremer.fr/ifremer/cersat/products/gridded/psi- concentration/data/antarctic/daily/netcdf building file list ... done. visiting ftp://ftp.ifremer.fr/ifremer/cersat/products/gridded/psi- concentration/data/antarctic/daily/netcdf/ ... bb_rget exited with an error (RETR response: 425) download failed or was interrupted: not running post-processing step

Fri Mar 29 09:57:39 2019 dataset synchronization complete: CERSAT SSM/I sea ice
concentration

A tibble: 1 x 5

name id source_url
status files


1 CERSAT SSM/I sea ice … CERSAT_S…
ftp://ftp.ifremer.fr/ifremer/cersat/products/gridded/psi-con… FALSE <tibble
[0…`

raymondben commented

Hi, hmm, that's very odd! I tried just now using your code (on a Linux machine) and it worked fine. Maybe it was a temporary issue at the ifremer server - perhaps try again? Otherwise I'll have a look later when I have some time.

jasmin4689 commented
raymondben commented

@jasmin4689 - do you have (or can you install) wget on your machine? (If you are using homebrew, I think it's just brew install wget --with-libressl - but I don't have a Mac so am not totally sure about that. You will need admin privileges to do it, too).
If you have this, we can try a modified version of mysrc2 that uses wget under the hood instead of the current R code. Sometimes that gets around problems like this one.

jasmin4689 commented
  • 🎉2
raymondben commented

Ah, excellent! Thanks for letting me know. If it's any consolation, we also have problems at work that don't happen elsewhere, due to firewalls/proxies/corporate access rules. And they are often difficult to track down.
I'm going to assume that this particular issue was one of those, and we can't really do much about it in the code here, so I'll close the issue.