All sentinel-2 items acquired after 5 July 2018 are missing from the sat-api catalog

[carlodef]

It seems that all sentinel-2 items acquired after 5 July 2018 are missing from the sat-api catalog at https://sat-api/developmentseed.org/search/stac.

This API request:

https://sat-api.developmentseed.org/search/stac?limit=1000&c:id=%22sentinel-2%22&datetime=2018-07-05/2019-01-01

returns no results:

{"type":"FeatureCollection","properties":{"found":0,"limit":1000,"page":1},"features":[]}

while there are many sentinel-2 images acquired after 2018-07-05 that are available on AWS. For instance this one: https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/30/N/TJ/2018/8/16/0/

Is that expected?

[matthewhanson]

@carlodef The problem is that the switch of Sentinel-2 to a requestor pays bucket broke the ingestion of data into sat-api because the metadata was getting ingested from the bucket as well and unfortunately all files (including metadata) are now requestor pays.

We need to update the sat-api ingestion process, hoping this will be fixed in the coming weeks.

[carlodef]

@matthewhanson Ok, thanks for the clarifications. However, according to this post the metadata is still (and will stay) freely accessible at this url, isn't it?

[matthewhanson]

@carlodef yes it is accessible through that URL, but the current ingestion is still trying to read from the bucket, so I've got to update the code to use the new endpoint.

[carlodef]

@matthewhanson Thanks! Is there something I can contribute to help with this update?

[matthewhanson]

Sorry for the delay @carlodef I'm at FOSS4G conf, but should be able to get this fixed after my talks tomorrow.

[carlodef]

Thank you @matthewhanson . I was looking into the code sources to understand where this could be fixed, and ended up in sat-api/packages/sentinel/index.js. Is that correct?

[matthewhanson]

@carlodef That is correct, but there's some additional bugs in sat-api that were introduced with a refactor that recently happened. I've fixed most of those and should be able to deploy a new version today.

[carlodef]

Thank you @matthewhanson, let me know if I can help!

[shiweihappy]

now it is ok

https://sat-api.developmentseed.org/search/stac?datetime=2018-09-06&limit=1&page=1&c:id=sentinel-2

response:

type: "FeatureCollection",
properties: {
    found: 2944,
    limit: 1,
    page: 1
},
...

[carlodef]

A lot of products are still missing. For example:

https://sat-api.developmentseed.org/search/stac?limit=1000&c:id=%22sentinel-2%22&sentinel:product_id=%22S2B_MSIL1C_20180803T094029_N0206_R036_T33QTE_20180803T132643%22

gives

{"type":"FeatureCollection","properties":{"found":0,"limit":1000,"page":1},"features":[]}

while the product S2B_MSIL1C_20180803T094029_N0206_R036_T33QTE_20180803T132643 asked for in the query is available here:

https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/33/Q/TE/2018/8/3/0/productInfo.json

[matthewhanson]

@carlodef It is ingesting new records, but I'm currently backfilling it with the missing data, it should be complete in the next day.