Enhance Point2Grid to support modified quality control settings for smoke/dust AOD data in GOES-16/17 as of April 16, 2024

Question

Enhance Point2Grid to support modified quality control settings for smoke/dust AOD data in GOES-16/17 as of April 16, 2024

Closed this issue 6 months ago · 30 comments

Answer 1 · 2024-04-10T17:00:52.000Z

Without this update, EMC will not be able to operationally verify HRRR and RAP Smoke AOD in EVS v2.0. That is why this update is required. Thank you!

Answer 2 · 2024-04-16T18:51:25.000Z

ABI Algorithm updates of GOES East/West has been delayed to April 17 2024.

Answer 3 · 2024-04-22T20:41:38.000Z

Is the data file with a new algorithm available?
I wonder the meaning of the flags are saved into the ADP file. Here are variable attributes of the old ADP file (generated year 2019)
Filename: goes_16/OR_ABI-L2-ADPC-M6_G16_s20192662141196_e20192662143569_c20192662144526.nc
Variable attribiutes:

        ushort DQF(y, x) ;
                DQF:_FillValue = 65535US ;
                DQF:long_name = "ABI L2+ Aerosol Detection data quality flags" ;
                DQF:standard_name = "status_flag" ;
                DQF:valid_range = 0US, 255US ;
                DQF:flag_masks = 1US, 1US, 2US, 2US, 12US, 12US, 12US, 48US, 48US, 48US, 64US, 64US, 128US, 128US ;
                DQF:flag_values = 0US, 1US, 0US, 2US, 0US, 4US, 12US, 0US, 16US, 48US, 0US, 64US, 0US, 128US ;
                DQF:flag_meanings = "good_smoke_detection_retrieval_qf invalid_smoke_detection_due_to_snow_ice_clouds_or_degraded_source_data_qf good_dust_detection_retrieval_qf invalid_dust_detection_due_to_snow_ice_clouds_or_bad_source_data_qf low_confidence_smoke_detection_qf medium_confidence_smoke_detection_qf high_confidence_smoke_detection_qf low_confidence_dust_detection_qf medium_confidence_dust_detection_qf high_confidence_dust_detection_qf out_of_sun_glint_qf within_sun_glint_qf within_valid_solar_and_satellite_zenith_angle_range_qf outside_valid_solar_or_satellite_zenith_angle_range_qf" ;
                DQF:number_of_smoke_detection_qf_values = 2UB ;

Answer 4 · 2024-04-22T20:53:45.000Z

Our data center are currently down so I lost the pathway to move the data among WCOSS2, ftp server, and local PC system (where I can drop the file to the GitHub issues). Will upload the new datafile after our Data Center is functioned again.

Answer 5 · 2024-04-23T12:36:27.000Z

Recap email sent

Hi, Howard:

I hope my memory helps.

(1) If simply handling total AOD, we do not need to specify the -adp option for point2grid. In this case, the qc flag is read from the same AOD input file and -qc=0,1,2 means AOD quality is "high", "medium", and "low", respectively. I believe this part should remain the same.

(2) If a need to map Smoke/Dust AOD, additional information of the values of smoke/dust detection confidence need to be read from ADP file to determine the status of quality of mapped Smoke/Dust AOD. Because the corresponding value of smoke/dust detection confidence changed in the new algorithm, from your email, the conversion from detection confidence to quality flag may need to be updated to reflect the new values.

I think the logic of conversion needs to be reviewed to reflect the new definition of ADP detection confidence. I recall we madea table for the conversion to high. medium. and low quality Smoke/Dust AOD. The current rule maybe like

AOD qc flag=0 + ADP Smoke detection confidence=12 ==> high quality Smoke AOD.
AOD qc flag=0 + ADP Smoke detection confidence=4 ==> medium quality Smoke AOD.
AOD qc flag=0 + ADP Smoke detection confidence=0 ==> low quality Smoke AOD

AOD qc flag=1 + ADP Smoke detection confidence=12 ==> medium quality Smoke AOD.
AOD qc flag=1 + ADP Smoke detection confidence=4 ==> medium quality Smoke AOD.
AOD qc flag=1 + ADP Smoke detection confidence=0 ==> low quality Smoke AOD

AOD qc flag=2 + ADP Smoke detection confidence=12 ==> low quality Smoke AOD.
AOD qc flag=2 + ADP Smoke detection confidence=4 ==> low quality Smoke AOD.
AOD qc flag=2 + ADP Smoke detection confidence=0 ==> low quality Smoke AOD.

AOD qc flag=0 + ADP Dudt detection confidence=48 ==> high quality Dust AOD.
AOD qc flag=0 + ADP Dust detection confidence=16 ==> medium quality Dust AOD.
AOD qc flag=0 + ADP Dust detection confidence=0 ==> low quality Dust AOD

AOD qc flag=1 + ADP Dust detection confidence=48 ==> medium quality Dust AOD.
AOD qc flag=1 + ADP Dust detection confidence=16 ==> medium quality Dust AOD.
AOD qc flag=1 + ADP Dust detection confidence=0 ==> low quality Dust AOD

AOD qc flag=2 + ADP Dust detection confidence=48 ==> low quality Dust AOD.
AOD qc flag=2 + ADP Dust detection confidence=16 ==> low quality Dust AOD.
AOD qc flag=2 + ADP Dust detection confidence=0 ==> low quality Dust AOD.

If you found the conversion relationship in current is like what I described above, then

======== the new rule ==========

AOD qc flag=0 + ADP Smoke detection confidence=0 ==> high quality Smoke AOD.
AOD qc flag=0 + ADP Smoke detection confidence=4 ==> medium quality Smoke AOD.
AOD qc flag=0 + ADP Smoke detection confidence=8 ==> low quality Smoke AOD

AOD qc flag=1 + ADP Smoke detection confidence=0 ==> medium quality Smoke AOD.
AOD qc flag=1 + ADP Smoke detection confidence=4 ==> medium quality Smoke AOD.
AOD qc flag=1 + ADP Smoke detection confidence=8 ==> low quality Smoke AOD

AOD qc flag=2 + ADP Smoke detection confidence=0 ==> low quality Smoke AOD.
AOD qc flag=2 + ADP Smoke detection confidence=4 ==> low quality Smoke AOD.
AOD qc flag=2 + ADP Smoke detection confidence=8 ==> low quality Smoke AOD.

AOD qc flag=0 + ADP Dudt detection confidence=0 ==> high quality Dust AOD.
AOD qc flag=0 + ADP Dust detection confidence=16 ==> medium quality Dust AOD.
AOD qc flag=0 + ADP Dust detection confidence=32 ==> low quality Dust AOD

AOD qc flag=1 + ADP Dust detection confidence=0==> medium quality Dust AOD.
AOD qc flag=1 + ADP Dust detection confidence=16 ==> medium quality Dust AOD.
AOD qc flag=1 + ADP Dust detection confidence=32 ==> low quality Dust AOD

AOD qc flag=2 + ADP Dust detection confidence=0==> low quality Dust AOD.
AOD qc flag=2 + ADP Dust detection confidence=16 ==> low quality Dust AOD.
AOD qc flag=2 + ADP Dust detection confidence=32 ==> low quality Dust AOD.

Ho-Chun

Answer 6 · 2024-04-23T12:36:37.000Z

By Howard

The DQF variable at the AOD file:

    byte DQF(y, x) ;
            DQF:long_name = "ABI L2+ Aerosol Optical Depth at 550 nm data quality flags" ;
            DQF:standard_name = "status_flag" ;
            DQF:_Unsigned = "true" ;
            DQF:valid_range = 0b, 3b ;
            DQF:flag_values = 0b, 1b, 2b, 3b ;
            DQF:flag_meanings = "high_quality_retrieval_qf medium_quality_retrieval_qf low_quality_retrieval_qf no_retrieval_qf" ;

So "AOD qc flag=0", "AOD qc flag=1", and "AOD qc flag=2" mean the DQF value at AOD file. Is it correct?

Question 1. Filtering out if "AOD qc flag=3". Is it correct?
Question 2. Filtering out if Smoke/Dust detection confidence does not belong to high, medium and low quality. Filtering out with following cases. Is it correct?
Any AOD qc flag + ADP Smoke detection confidence=8
Any AOD qc flag + ADP Dust detection confidence=32

Cheers,
Howard

Answer 7 · 2024-04-23T12:48:32.000Z

@hsoh-u Yes to your question1. It is not clear to me whether your question 2 referred to Smoke or Dust AOD. Simply from the current algorithm, replacing threshold values 0 -> 8, then 12->0 for Smoke AOD. 0->32 then 48-> 0 for Dust AOD. We do not need to re-create the rule. I think we should keep the flexibility of choice for users. Your question seems to indicate the only choice is high quality Smoke AOD or Dust AOD.

ho-Chun

Answer 8 · 2024-04-23T14:31:32.000Z

@JohnHalleyGotway Can you get data from Hera for Howard? That is another pathway for GOES AOD and ADP data transfer.

Answer 9 · 2024-04-23T15:02:03.000Z

Please ignore this comment.

The question 2 is how to handle ADP Smoke confidence=8 and ADP Dust confidence=32 for Baseline algorithm and ADP Smoke confidence=12 and ADP Dust confidence=48 for Enterprise algorithm.
Case 1: when -qc option is not given (MET does not filter out).
Case 2: if ADP Smoke confidence=8 and ADP Dust confidence=32 can be included with -qc 0,1,2,3.

The possible values for Baseline algorithm:

possible ADP Smoke detection confidence: 0, 4, 8, 12.
possible ADP Dust detection confidence: 0, 16, 32 48.
Existing rule for ADP Smoke (0bDDD: binary number)
- 0b0000 = 0 = low
- 0b0100 = 4 = medium
- 0b1000 = 8 (filtering out by -qc option)
- 0b1100 = 8 + 4 = 12 = high.
Existing rule for ADP Dust
- 0b000000 = 0 = low
- 0b010000 =16 = medium
- 0b100000 = 32 (filtering out by -qc option)
- 0b110000 = 32 + 16 = 48 = high.

Answer 10 · 2024-04-23T15:20:27.000Z

Please ignore above comment.

Answer 11 · 2024-04-23T16:50:48.000Z

@hsoh-u Saw your latest post to ignore previous posting. But allow me to provide my comment for your consideration.
"YES" on
So "AOD qc flag=0", "AOD qc flag=1", and "AOD qc flag=2" mean the DQF value at AOD file. Is it correct?

The discussion below specifically points to Point2Grid with -adp option

If no -qc option is given, then qc=0,1,2 high+medium+low is the default, i.e., filtering AOD qc flag = 3 OR ADP Smoke detection confidence =12 (bad_missing_smoke_detection) for SMOKE AOD and ADP Dust detection confidence =48 (bad_missing_dust_detection) for Dust AOD.

One should not allow 3 as an option in point2grid, 3 mean all obs should be filtered out. I do not think any user will choose that option for verification purpose.

AOD qc=3=no retrieval
ADP Smoke detection confidence=12=bad_missing_smoke_detection
ADP Dust detection confidence=48=bad_missing_dust_detection

Available options for -qc

0 high only
1 medium only
2 low only
0,1 high+medium
0,1,2 high+medium+low (default if no -qc is given)

Borrowing your description, I think for the "Enterprise Algorithm"

The possible values for Enterprise Algorithm:

possible ADP Smoke detection confidence: 0, 4, 8, 12.
possible ADP Dust detection confidence: 0, 16, 32 48.
Revised rule for ADP Smoke (0bDDD: binary number)
- 0b0000 = 0 = high
- 0b0100 = 4 = medium
- 0b1000 = 8 = low
- 0b1100 = 8 + 4 = 12 = (filtering out )
Revised rule for ADP Dust
- 0b000000 = 0 = high
- 0b010000 =16 = medium
- 0b100000 = 32 = low
- 0b110000 = 32 + 16 = 48 = (filtering out).

Answer 12 · 2024-04-23T17:03:05.000Z

@Ho-ChunHuang-NOAA Thank you for the comments.

"no retrieval" will be filtered out without -qc option
I agree MET user won't add 3 into -qc option

Answer 13 · 2024-04-25T14:24:46.000Z

The file size is over the limit for add file function.

Please obtain the sample aod and adp file on 04/19/2024

adp_20240419.tar aod_20240419.tar

from

https://www.emc.ncep.noaa.gov/mmb/hchuang/ftp/

Answer 14 · 2024-04-25T14:54:20.000Z

Thank you for the files.
The algorithm can be detected by looking at the DQF:flag_values and DQF:flag_meanings.

DQF:flag_values and DQF:flag_meanings (20190801/OR_ABI-L2-ADPC-M6_G16_s20192131001462_e20192131004235_c20192131005097.nc)

0US: good_smoke_detection_retrieval_qf
1US: invalid_smoke_detection_due_to_snow_ice_clouds_or_degraded_source_data_qf
0US: good_dust_detection_retrieval_qf
2US: invalid_dust_detection_due_to_snow_ice_clouds_or_bad_source_data_qf
0US low_confidence_smoke_detection_qf
4US: medium_confidence_smoke_detection_qf
12US: high_confidence_smoke_detection_qf
0US: low_confidence_dust_detection_qf
16US: medium_confidence_dust_detection_qf
48US: high_confidence_dust_detection_qf
0US: out_of_sun_glint_qf
64US: within_sun_glint_qf
0US: within_valid_solar_and_satellite_zenith_angle_range_qf
128US: outside_valid_solar_or_satellite_zenith_angle_range_qf

DQF:flag_valaues and DQF:flag_meanings (20240419/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc)

2US: low_confidence_ash_detection_qf
1US: medium_confidence_ash_detection_qf
0US: high_confidence_ash_detection_qf
3US: bad_missing_ash_detection_qf
8US: low_confidence_smoke_detection_qf
4US: medium_confidence_smoke_detection_qf
0US: high_confidence_smoke_detection_qf
12US: bad_missing_smoke_detection_qf
32US: low_confidence_dust_detection_qf
16US: medium_confidence_dust_detection_qf
0US: high_confidence_dust_detection_qf
48US: bad_missing_dust_detection_qf
128US: low_confidence_nuc_detection_qf
64US: medium_confidence_nuc_detection_qf
0US: high_confidence_nuc_detection_qf
192US:: bad_missing_nuc_detection_qf

Answer 15 · 2024-04-25T15:25:51.000Z

@hsoh-u Do you mean current Point2Grid (without modification) can automatically detect new threshold value for high/medium/low quality?

Answer 16 · 2024-04-25T15:53:11.000Z

No, the auto detection is not available at the previous release. It's for the next release.

Answer 17 · 2024-04-29T19:47:49.000Z

The data type is changed from ubyte to byte:
from OR_ABI-L2-ADPC-M6_G16_s20192662141196_e20192662143569_c20192662144526.nc:

   ubyte Smoke(y, x) ;
          Smoke:valid_range = 0UB, 1UB ;
   ubyte Dust(y, x) ;
          Dust:valid_range = 0UB, 1UB ;

from OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc

   byte Smoke(y, x) ;
          Smoke:valid_range = 0b, 1b ;
   byte Dust(y, x) ;
          Dust:valid_range = 0b, 1b ;

Answer 18 · 2024-04-29T20:04:04.000Z

Thanks to find out the new changes. NESDIS did not tell us all the changes.

Do you need to read and use the values of
byte Smoke(y, x) ; and byte Dust(y, x)?

I thought all you need from the ADP files is the values read from ushort DQF(y, x)? am I correct?

Answer 19 · 2024-04-29T20:29:57.000Z

MET reads Smoke and Dust variables at ADP file and filters out if the data value is 0 (smoke_absent or dust_absent) before applying QC flags.

  Smoke:flag_values = 0UB, 1UB ;
  Smoke:flag_meanings = "smoke_absent smoke_present" ;
  Dust:flag_values = 0UB, 1UB ;
  Dust:flag_meanings = "dust_absent dust_present" ;

  Smoke:flag_values = 0b, 1b ;
  Smoke:flag_meanings = "smoke_absent smoke_present" ;
  Dust:flag_values = 0b, 1b ;
  Dust:flag_meanings = "dust_absent dust_present" ;

Answer 20 · 2024-04-29T21:14:18.000Z

I see. Thanks for letting me know.

Answer 21 · 2024-05-03T20:24:39.000Z

@hsoh-u I noticed this issue was closed (likely automatically with the merge of the PR into main). I'm thinking it needs to be reopened for the work in develop, unless that's already been done. Can you please confirm and reopen if necessary?

Answer 22 · 2024-05-03T21:09:44.000Z

Hi, Hank: I have question on your statement if #2874 if only high quality is given with-qc 0, all data will be filtered out. There should be 4 records left, with AOD QC=high + ADP QC (For smoke) = high, isn't it? Ho-Chun

…

-- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958

On Fri, May 3, 2024 at 4:25 PM Julie Prestopnik ***@***.***> wrote: @hsoh-u <https://github.com/hsoh-u> I noticed this issue was closed (likely automatically with the merge of the PR into main). I'm thinking it needs to be reopened for the work in develop, unless that's already been done. Can you please confirm and reopen if necessary? — Reply to this email directly, view it on GitHub <#2853 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPHE3HOTTEYUVLH54GYQ3DZAPXBZAVCNFSM6AAAAABF63TCU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJTG4YTGNZQGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Answer 23 · 2024-05-03T21:48:37.000Z

No, it's before applying AOD QC. I will add the count of adjusted confidences for the MET v12 release.

The comment #2874 is for unit test case:

input AOD and ADP files from you:
- 20240419/OR_ABI-L2-AODC-M6_G16_s20241100001171_e20241100003544_c20241100006242.nc
- 20240419/OR_ABI-L2-ADPC-M6_G16_s20241100001171_e20241100003544_c20241100006361.nc
-field 'name="AOD_Smoke"; level="*";' -v 4 -qc 0,1

The debug messages

DEBUG 4:    AOD QC: high=8092 medium=6910, low=47149, no_retrieval=0
DEBUG 4:    ADP QC: high=4 medium=87, low=62050, no_retrieval=10, adjusted=47142

AOD QC values:
- The high confidence count: 8092
- The medium confidence count: 6910
- The low confidence count: 47149
ADP Smoke QC values as raw data:
- The high confidence count: 4
- The medium confidence count: 87
- The low confidence count: 62050
- adjusted: 47142 confidence were reduced (from high to medium/low or from medium to low).

With two inputs above, all high confidences were reduced to medium or low confidence.

Answer 24 · 2024-05-03T21:54:05.000Z

Oops, the adjusted number is not correct (the original low count was added accidentally) . I will correct it.

Answer 25 · 2024-05-03T22:31:11.000Z

Hi, Hank: I thought the logic should be from the user's guide. with -adp option and for Amoke_AOD and Dust_AOD; 0 (high only) : [high AOD QC + High ADP QC] 1 (medium only): [high AOD QC + medium ADP QC]+[medium AOD QC + high ADP QC]+ [medium AODQC+medium ADP QC] 0,1 (high+medium): [high AOD QC + High ADP QC]+[high AOD QC + medium ADP QC]+[medium AOD QC + high ADP QC]+ [medium AODQC+medium ADP QC] You meant the sample files give you all medium quality SMoke_AOD? number of "0" options (high only) is zero but not four? Ho-Chun

…

-- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958

On Fri, May 3, 2024 at 5:54 PM Howard Soh ***@***.***> wrote: Oops, the adjusted number is not correct (the original low count was added accidentally) . I will correct it. — Reply to this email directly, view it on GitHub <#2853 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPHE3HGGLN7JEMJBGMA3KLZAQBRHAVCNFSM6AAAAABF63TCU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJTHAYTCMZYGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Answer 26 · 2024-05-03T22:56:00.000Z

MET follows the logic you described. It's a just log message. The log message will be changed to

DEBUG 4:    AOD QC: high=8092 medium=6910, low=47149, no_retrieval=0
DEBUG 4:    ADP QC: high=0 (4), medium=24 (87), low=62117 (62050), no_retrieval=10
DEBUG 4:    adjusted: high to medium=0, high to low=4, medium to low=63, total=67

high=0 (4): adjusted high count = 0 (raw high count = 4)
medium=24 (87): adjusted medium count = 24 (raw medium count = 87)
low=62117 (62050): adjusted low count = 62117 (raw low count = 62050)

Answer 27 · 2024-05-06T12:39:44.000Z

The numbers mean there are 4 AOD pixels with low AOD DQF read and high smoke confidence ADP DQF read, therefore, the algorithm adjusts AOD QC from HIGH to LOW (as final Smoke_AOD QC), am I correct?

Answer 28 · 2024-05-06T19:08:25.000Z

The numbers mean there are 4 AOD pixels with low AOD DQF read and high smoke confidence ADP DQF read, therefore, the algorithm adjusts AOD QC from HIGH to LOW (as final Smoke_AOD QC), am I correct?

Yes, that's correct

Answer 29 · 2024-06-11T17:48:59.000Z

@Ho-ChunHuang-NOAA The fix was merged into the main_v11.1 bugfix branch and the develop branch three weeks ago.

Answer 30 · 2024-06-11T18:44:43.000Z

Howard, thank you for the update.

Enhance Point2Grid to support modified quality control settings for smoke/dust AOD data in GOES-16/17 as of April 16, 2024

Describe the Enhancement

Time Estimate

Sub-Issues

Relevant Deadlines

Funding Source

Define the Metadata

Assignee

Labels

Milestone and Projects

Define Related Issue(s)

Enhancement Checklist