missing NaN values in CloudMaskRequest

Question

missing NaN values in CloudMaskRequest

mdoubko opened this issue 7 years ago · 10 comments

Dear all,

I have noticed that no data in mask and prob outputs of CloudMaskRequest are assigned 0 and 0.0007 value, respectively! Am I doing something wrong?

I have noticed this when using following WCS request over fields where a scenes border occurred (in particular in Haselsdorf, Austria on 21/8/2017:

wcs_bands_request = WcsRequest(data_folder=outdir,
resx='10m',
resy='10m',
layer='NDVI__NORMALIZED_DIFFERENCE_VEGETATION_INDEX_',
maxcc=1.0,
bbox=betsiboka_bbox,
time=timerange,
image_format=MimeType.TIFF_d32f,
custom_url_params={CustomUrlParam.SHOWLOGO: False, CustomUrlParam.ATMFILTER:"None", CustomUrlParam.EVALSCRIPT: bands_script},
instance_id=userID)
all_cloud_masks = CloudMaskRequest(ogc_request=wcs_bands_request, threshold=0.2)

thanks,
Marcela Doubkova

Answer 1 · 2018-04-11T09:00:10.000Z

Dear Marcela,

could you please share the code snippet that reproduces the problem? Like above, but with the definition of the bbox and time_interval.

Thank you and best regards,

Anze

Answer 2 · 2018-04-11T09:29:31.000Z

Hello, polygon is attached. Hopefully this helps. Marcela timerange = ('2015-06-01','2018-04-10') POLYGON = "C:\DATA_raw\shapefiles\Field_Haselsd2_32633.geojson" bands_script = 'return [B01,B02,B04,B05,B08,B8A,B09,B10,B11,B12]' data = pygeoj.load(filepath=POLYGON) feature=data[0] BBOX=feature.geometry.bbox betsiboka_bbox = BBox(bbox=BBOX, crs=CRS.UTM_33N) wcs_bands_request = WcsRequest(data_folder=outdir, resx='10m', resy='10m', layer='NDVI__NORMALIZED_DIFFERENCE_VEGETATION_INDEX_', maxcc=1.0, bbox=betsiboka_bbox, time=timerange, image_format=MimeType.TIFF_d32f, custom_url_params={CustomUrlParam.SHOWLOGO: False, CustomUrlParam.ATMFILTER:"None", CustomUrlParam.EVALSCRIPT: bands_script}, instance_id=userID) dates_WCS_8601=wcs_bands_request.get_dates() #dates_WCS_8601 are in chronological order #convert WCS dates to YMT dates_WCS = [] #dates_WCS are in chronological order for i in range(len(dates_WCS_8601)): dates_WCS.append(module_time.time_convert_from8601_to_YMT(dates_WCS_8601[i])) # The CloudMaskRequest class allows one to retrieve cloud probability maps of an area for all available dates within # a date range, and then efficiently derive binary cloud masks. # This masks is more conservative (as far as I udnerstood, takes into account more of cloud shadows!) all_cloud_masks = CloudMaskRequest(ogc_request=wcs_bands_request, threshold=0.2) cloud_prob_mean = np.empty(len(dates_WCS), dtype=np.float32) index = 0 for prob, mask, data in all_cloud_masks: cloud_prob_mean[index] = np.mean(prob) index = index +1

…

On Wed, Apr 11, 2018 at 11:00 AM, Anze Zupanc ***@***.***> wrote: Dear Marcela, could you please share the code snippet that reproduces the problem? Like above, but with the definition of the bbox and time_interval. Thank you and best regards, Anze — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AiN8IgYB2QS5CLUt1jX8AlaiH56BhLWxks5tncYfgaJpZM4TPmoY> .

Answer 3 · 2018-04-12T07:55:25.000Z

Hi Marcela,

please upgrade your s2cloudless package to the new version 1.1.0. In this latest version you can assign a cloud probability (or mask) value for non valid data pixels. By default it is set to 0 and False, respectively. But if you prefer, you can set it to np.nan or any other value. Example code snippet:

all_cloud_masks.get_probability_masks(non_valid_value=0)

Answer 4 · 2018-04-12T08:47:25.000Z

well thanks but I am using directly CloudMaskRequest. Can you include this functionality also there. cheers, Marcela

…

On Thu, Apr 12, 2018 at 9:55 AM, Anze Zupanc ***@***.***> wrote: Hi Marcela, please upgrade your s2cloudless package to the new version 1.1.0. In this latest version you can assign a cloud probability (or mask) value for non valid data pixels. By default it is set to 0 and False, respectively. But if you prefer, you can set it to np.nan or any other value. Example code snippet: all_cloud_masks.get_probability_masks(non_valid_value=0) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AiN8IlC_0r7Sua_J3R5JqtSjytzyGKaAks5tnwhugaJpZM4TPmoY> .

Answer 5 · 2018-04-12T08:48:40.000Z

This functionality is included in CloudMaskRequest.

Answer 6 · 2018-04-12T08:55:51.000Z

After ugrading S2cloudless and writing: all_cloud_masks = CloudMaskRequest(ogc_request=wcs_bands_request, threshold=0.2,non_valid_value=0) I am getting following error: unexpected keyword argument 'non_valid_value' thanks a lot for your time! Marcela

…

On Thu, Apr 12, 2018 at 10:48 AM, Anze Zupanc ***@***.***> wrote: This functionality is included in CloudMaskRequest. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AiN8IjE-fZqKhuoT650SDLr0_uadEwb1ks5tnxTpgaJpZM4TPmoY> .

Answer 7 · 2018-04-12T08:59:19.000Z

The non valid value is specified in get_probability_masks and get_cloud_masks methods:

all_cloud_masks = CloudMaskRequest(ogc_request=wcs_bands_request, threshold=0.2)

all_cloud_masks.get_probability_masks(non_valid_value=0)
or
all_cloud_masks.get_cloud_masks(non_valid_value=False)

Answer 8 · 2018-04-12T09:56:08.000Z

Thanks, nevertheless I am using cloudmaskrequest in iterative manner. My, hopefully last question, at least fo rnow, is how should I update following code?
THANSK!

all_cloud_masks = CloudMaskRequest(ogc_request=wcs_bands_request, threshold=0.2)

cloud_prob_mean = np.empty(len(dates_WCS), dtype=np.float32) index = 0

for prob, mask, data in all_cloud_masks:
cloud_prob_mean[index] = np.nanmean(prob)
index = index +1

Answer 9 · 2018-04-12T18:21:52.000Z

Two options:

all_cloud_masks = CloudMaskRequest(ogc_request=wcs_bands_request, threshold=0.2)
cloud_prob_mean = np.empty(len(dates_WCS), dtype=np.float32)
index = 0
all_cloud_masks.get_probability_masks(non_valid_value=np.nan)

for prob, mask, data in all_cloud_masks:
cloud_prob_mean[index] = np.nanmean(prob)
index = index +1

Note: cloud probability maps are cached and hence calculated only once.

all_cloud_masks = CloudMaskRequest(ogc_request=wcs_bands_request, threshold=0.2)
cloud_probs = all_cloud_masks.get_probability_masks(non_valid_value=np.nan)
cloud_prob_mean = np.nanmean(cloud_probs, axis=(1,2))

Answer 10 · 2018-04-13T09:52:31.000Z

Perfect. Thanks for all your time!

…

On Thu, Apr 12, 2018 at 8:21 PM, Anze Zupanc ***@***.***> wrote: Two options: 1. ------------------------------ all_cloud_masks = CloudMaskRequest(ogc_request=wcs_bands_request, threshold=0.2) cloud_prob_mean = np.empty(len(dates_WCS), dtype=np.float32) index = 0 all_cloud_masks.get_probability_masks(non_valid_value=np.nan) for prob, mask, data in all_cloud_masks: cloud_prob_mean[index] = np.nanmean(prob) index = index +1 Note: cloud probability maps are cached and hence calculated only once. 1. ------------------------------ all_cloud_masks = CloudMaskRequest(ogc_request=wcs_bands_request, threshold=0.2) cloud_probs = all_cloud_masks.get_probability_masks(non_valid_ value=np.nan) cloud_prob_mean = np.nanmean(cloud_probs, axis=(1,2)) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AiN8ItdYn_fWCF7FuwWBDVjcEyddwaDsks5tn5tAgaJpZM4TPmoY> .