USGS-CMG/usgs-cmg-portal

2 new datasets to harvest to portal

Opened this issue · 30 comments

rsignell-usgs commented

@kwilcox , this needs your attention.

emontgomery-usgs commented

and if you want to try to ingest a sample dataset from the Santa Cruz office, I was able to convert this to a nominally CF output: https://gamone.whoi.edu/thredds/silt/usgs/Projects/stellwagen/Data_tests/tst_other_centers/catalog.html
Cheers!

emontgomery-usgs commented

@kwilcox , I see that the data from CHANDELEUR_13 and DAUPHIN got ingested, thanks! I'm still not seeing the original 2 I asked about though (W_GMAINE and GRANDBAY)- is there something wrong with theses files? Thanks!

kwilcox commented

Looking now!

  • 👍2
kwilcox commented

Made some good progress, having it looked over internally and I'll post a link for you to review once that is done.

I switched our import to directly hit your THREDDS files. We were previously downloading the Raw and converting to CF and then to our internal Device files. We are no longer producing the Device files and are harvesting directly from your the CF files - reducing a bunch of complexity and opening up the ability to harvest from other THREDDS endpoints of CF files (i.e. Santa Cruz files).

  • 👍1
emontgomery-usgs commented

FANTASTIC! thanks!

emontgomery-usgs commented

@kwilcox - Any guesses on when these 2 might be on the portal? Doesn't have to be auto-harvested... thanks!

kwilcox commented

It’s on top of the checklist this week

  • 👍2
emontgomery-usgs commented

@kwilcox ,

I see the data from both new datasets- great! But now I can't find the control to show all data instead of whatever approximation is shown by default.
image
All the data I tried to look at appears to be WAY over-simplified in the default view.
Thanks!

kwilcox commented

Thanks, I noticed this as well and pinged @akbstone to take a look. It is likely because this is the staging instance and isn't completely polished for production. I've downloaded the CSV from the stations and the data looks good so if isn't a quick fix I'll just push to production.

emontgomery-usgs commented

OK, thanks.

Another detail is that I'm not sure all the stations for all the experiments are shown:
image
I think they will likely all be displayed when pushed to the existing portal.

kwilcox commented

You are on the W_GMAINE - 1055 page and it is only showing stations within a certain geographic area around that station.

The catalog page for all of the stations shows them all: https://cmgdata.usgsportals.net/?&staging=true&sensor_version=v2#metadata/86/sensor_source/list

If that is confusing maybe we could add a note that the map is only showing nearby stations, or disable the map controls on the station page?

akbstone commented

@emontgomery-usgs @kwilcox - the binning issue is fixed.

  • 😄1
akbstone commented

@emontgomery-usgs @kwilcox

Re. nearby stations on station page, I think a couple things could make this more usable:

  • Limit zoom to the selected zoom and higher
  • On map pan, query for stations within the new bounds
kwilcox commented

Both sound great to me!

emontgomery-usgs commented

@akbstone , @kwilcox
I'd agree- both would help - thanks!

In the last image I put in this thread, there are a ton of other stations from other sources, so I assumed the map would also show everything of ours in the view. Couldn't find out how to remove "other" stations from the view- if there's a way to do that, it would be nice if it were more obvious.

I submitted a new ticket #302 suggesting use of color instead of shape to indicate the site. When dots are dense at certain zooms, it was hard to separate the star from all the dots.

emontgomery-usgs commented

@kwilcox , @akbstone,
Could the data from the 2 new experiments be pushed to the public server some time soon?

Also I went to the link to the staging server in Kyle's post just now and it just cycles and cycles... and doesn't ever return content.
Thanks!

kwilcox commented

@jessicaaustin and I will push these to the portal today, it's happening!

  • 👍1
  • 😄1
  • 🎉1
akbstone commented

@emontgomery-usgs @kwilcox

  • 😄1
emontgomery-usgs commented

@akbstone Thank you! much better!

kwilcox commented

@emontgomery-usgs https://cmgdata.usgsportals.net/?&sensor_version=v2#metadata/86/sensor_source/list

The data is streaming in and will be updating into the portal, some stations will be viewable while some may take a few hours. Searching for individual stations may not work for a few hours since a reindex won't happen until all data is in there.

Also, this is using the new ingestion system so we are all good looking forward!

emontgomery-usgs commented

@kwilcox

Wonderful, thanks! So excited the new ingest system is in place!

If I follow your link above, I see the new datasets in the list. However, if I go to the Main portal page, add the Oceanographic timeseries data to the map, and then zoom to where the Grand Bay data should be, the stations are not displayed. The sites show up on the map on the station page, but not on the map in the data browser. Is this intended?

  • 👍1
emontgomery-usgs commented

@kwilcox @akbstone
The 2 new datasets don't show up in the map view. You can't find them if you're browsing the map, even though they're there if you know how to find the station page.

Thanks for any help!

kwilcox commented

@emontgomery-usgs Brian and I took a look and figured out what was happening.

You can see them here: https://cmgdata.usgsportals.net/?&sensor_version=v2&ls=793b54c9-6055-1fd2-2eba-13ddec139891#map

But notice the "sensor_version=v2" in the URL. Without that are going to be missing from the map because they are on the "new system" and "V2" combines both systems together. The solution is to make "sensor_version=v2" the default which is only held up by a bug in displaying curtain plots (hover isn't working). Brian needs some more time to work that into his schedule, lets touch base mid next week.

emontgomery-usgs commented

@kwilcox, thanks for looking into this. Sounds like you and Brian have made progress. Great! Keep up the good work and I'll ping you next week, if I haven't heard by Thurs.

emontgomery-usgs commented

@kwilcox, OK, here's your reminder. Today I don't see the Western Gulf of Maine, or Grand Bay (Louisiana) data at https://cmgdata.usgsportals.net/#metadata/86/sensor_source/list or https://cmgdata.usgsportals.net/#map. Has the v2 merge happened? Thanks!

emontgomery-usgs commented

@kwilcox, @dnowacki-usgs, @rsignell-usgs

Now we're up to 4 experiments of data thata haven't made it to the portal. Is there any chance that they could be added before I retire on 7/31? I'm trying to tie up loose ends, and it would really be great to have the last four of our data releases visible before I leave :> If you could do what needs to be done to get it onto the IOOS porta,l bypassing the USGS portal, that would be fine too.

Here are the links to the data that is not yet on either portal:
http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/W_GMAINE/catalog.html
http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/GRANDBAY/catalog.html
http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/NRPP_17/catalog.html
http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/MATANZAS/catalog.html

Many thanks!