Error running start_maja.py / Doubts about folder-file structure

Question

Error running start_maja.py / Doubts about folder-file structure

intiluna opened this issue 5 years ago · 10 comments

Platform and versions:
• Ubuntu 18.04.4 LTS
• Python 3.7.6 (default, Jan 8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
• GDAL 2.4.2, released 2019/06/28

Dear Maja Users,

I have some problems with the folder and files structure required to properly run Maja. To start, I want to first process without CAMS. The steps are described below:

1-Downloaded and installed Maja_3.3.2_TM.run

bash MAJA-3.2.2_TM.run –target ~/maja

2.Clone the current repository to get start_maja.py

git clone https://github.com/CNES/Start-MAJA.git

3. Prepare folders.txt

repCode= ~/maja/Start-MAJA
repWork= /media/ernesto/cuatrot/maja_procesos/temporal
repL1 = /media/ernesto/cuatrot/maja_procesos/S2/L1C/L1CCH
repL2 = /media/ernesto/cuatrot/maja_procesos/S2/L2A/L2ACH
repMaja= ~/maja/bin/maja
repCAMS= /media/ernesto/cuatrot/maja_procesos/cams

4. Retrieve Sentinel-2 L1C data
Using peps_download.py, I have downloaded 188 images for 4 tiles (PDU,PDV,PEU,PEV) for the period from 2019-08-01 to 2020-03-20 . An example of the commands used to download images for one tile is presented:

python /media/ernesto/cuatrot/peps_download/peps_download-master/peps_download.py -c S2ST -t 16PDU -o 140 -a /media/ernesto/cuatrot/peps_download/peps_download-master/peps.txt -d 2019-08-01 -f 2020-03-20

Unzip L1C data path=media/ernesto/cuatrot/maja_procesos/S2/L1C/L1CCH

5. Parameters

Folder “userconf”

path=~/maja/Start-MAJA/userconf

The files in this folder are:
MAJAUserConfig_FORMOSAT_MUSCATE_PROTO.xml
MAJAUserConfig_LANDSAT8_MUSCATE_PROTO.xml
MAJAUserConfig_LANDSAT8_MUSCATE.xml
MAJAUserConfig_LANDSAT8.xml
MAJAUserConfig_LANDSAT_MUSCATE_PROTO.xml
MAJAUserConfig_LANDSAT_MUSCATE.xml
MAJAUserConfig_SENTINEL2_GPP.xml
MAJAUserConfig_SENTINEL2_MUSCATE.xml
MAJAUserConfig_SENTINEL2_TM.xml
MAJAUserConfig_SENTINEL2.xml
MAJAUserConfig_SPOT4_MUSCATE_PROTO.xml
MAJAUserConfigSystem.xml
MAJAUserConfig_VENUS.xml

Folder GIPP

path=~/maja/Start-MAJA/GIPP/SENTINEL2_TM
The files in this folder are:
README.md
S2A_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF
S2A_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF
S2A_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF
S2A_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF
S2B_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF
S2B_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF
S2B_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
S2__TEST_GIP_L2SITE_S_ALLSITES_00001_20190626_21000101.EEF

I have a doubt regarding the use of GIPP without CAMS. According to the instructions, if we want to use Parameters without CAMS, we should use the folder SENTINEL2_TM_CAMS:

However, when we read the README.md for http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/SENTINEL2_TM_CAMS. It clearly states that CAMS option enabled. And the README.md for: http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/SENTINEL2_TM, we get the message that CAMS option disabled.

So, I am confused. I assumed that the README files are correct and I should use SENTINEL2_TM when processing without CAMS.

Folder LUT

I downloaded the LUT from the zenodo server and I choose to use: 20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS
path=~/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs

Files in this folder:
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR

6. DTM preparation
After some problems with the code to prepare the dtm. It was suggested to use another branch of the DTM preparation code (https://github.com/CNES/Start-MAJA/tree/reprog-rc1/prepare_mnt) and using this updated code it works fine.

path=~/maja/Start-MAJA/DTM

Since I use 4 tiles I produced 4 DTMs. This folder contains 2 files per tile:
S2__TEST_AUX_REFDE2_16PDU_0001.DBL.DIR S2__TEST_AUX_REFDE2_16PEU_0001.DBL.DIR
S2__TEST_AUX_REFDE2_16PDU_0001.HDR S2__TEST_AUX_REFDE2_16PEU_0001.HDR
S2__TEST_AUX_REFDE2_16PDV_0001.DBL.DIR S2__TEST_AUX_REFDE2_16PEV_0001.DBL.DIR
S2__TEST_AUX_REFDE2_16PDV_0001.HDR S2__TEST_AUX_REFDE2_16PEV_0001.HDR

7. CAMS data

As stated in the beginning currently I want to use MAJA without CAMS but I would like to be able to process with CAMS in the future.

Following instructions from https://github.com/CNES/Start-MAJA/tree/master/cams_download
At this moment I have registered at ECMWF and I have API key.

I do not know what it means with: Create the file '.ecmwfapirc' in your home and type the API key. How you do that. Is it a text in which I hould put the API key? And referring to home folder you mean Home (~) or it means the “cams_download” folder in the Statr-Maja folder?
path= ~/maja/Start-MAJA/cams_download

8. Execute start_maja.py

In the start-maja folder (~/maja/Start-MAJA), I open terminal and run:

python start_maja.py -f ~/maja/Start-MAJA/folders.txt -g ~/maja/Start-MAJA/GIPP/SENTINEL2_TM -l ~/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS -t 16PDU -s ser -d 20190801 -e 202000320

Running the above commands produce the following errors:

What am I missing or doing wrong? Many thanks for your help.
Kind regards,

Answer 1 · 2020-04-01T16:06:01.000Z

Hi, this one is easy, start_maja is meant for python 2.7, and what you observe is typically an error when running it witth python3.
Come on, you're almost there !
Olivier

Answer 2 · 2020-04-01T17:07:01.000Z

Hi Olivier,
Thanks a lot. I want to get there soon :)

I installed a python 2.7 environment with conda:

conda create --name py2 python=2.7
conda activate py2

and the run the code again:

python start_maja.py -f ~/maja/Start-MAJA/folders.txt -g ~/maja/Start-MAJA/GIPP/SENTINEL2_TM -l ~/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS -t 16PDU -s ser -d 20190801 -e 202000320

I get the following error of missing folders/files

However I do have the files.
Here is the tree of folder/files in html for easy visualization:
https://www.dropbox.com/s/t5glnhrht7zcm6f/tree_folders.html?dl=0

Thanks again!!!! :D

Answer 3 · 2020-04-01T17:32:43.000Z

Hi Inti, the message means that some of the folders you defined in folders.txt are not- existing. Could you please check them ?
repCode = ~/maja/bin/Start-MAJA
repMaja = ~/maja/bin/maja

One possibility is that python does not recognise "~", maybe try to replace by the whole path.

(it's now time for me to disconnect, but I can try to answer new questions tomorrow)
Olivier

Answer 4 · 2020-04-02T10:03:36.000Z

Dear Olivier,

I managed to get MAJA script finding correctly the paths and files. It took some tries to realize that when running the start-maja.py we should think of a additive path (folders.txt path + additional path for GIPP, LUT and site name)

Full path to LUT:
"/home/ernesto/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs"
Full path to GIP:"/home/ernesto/maja/Start-MAJA/GIPP/SENTINEL2_TM"
Full path to L1C data: "/media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA"

In my case, the folders.txt is like this now:

repCode= /home/ernesto/maja/Start-MAJA
repWork= /media/ernesto/cuatrot/maja_procesos/temporal
repL1 = /media/ernesto/cuatrot/maja_procesos/S2/L1C
repL2 = /media/ernesto/cuatrot/maja_procesos/S2/L2A
repMaja= /home/ernesto/maja/bin/maja
repCAMS= /media/ernesto/cuatrot/maja_procesos/cams

So, when running the script I just have add the rest of the path like this:
python start_maja.py -f folders.txt -g GIPP/SENTINEL2_TM -l 20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs -t 16PDU -s CHINANDEGA -d 20190801 -e 202000320

Later I run it but I get some others errors:
`
ernesto@ernestomacpro:~/maja/Start-MAJA$ python start_maja.py -f folders.txt -g GIPP/SENTINEL2_TM -l 20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs -t 16PDU -s CHINANDEGA -d 20190801 -e 202000320

2020-04-02 11:48:46,254 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDU____T20190805.DBL.DIR not found
2020-04-02 11:48:46,254 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20190805-_T16PDU_C_V not found
2020-04-02 11:48:46,255 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDU____T20190810.DBL.DIR not found
2020-04-02 11:48:46,255 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20190810-_T16PDU_C_V not found
2020-04-02 11:48:46,258 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDU____T20190815.DBL.DIR not found
2020-04-02 11:48:46,258 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20190815-_T16PDU_C_V not found
....cut some rows just like the above part but for another dates....

2020-04-02 11:48:46,319 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDU____T20191223.DBL.DIR not found
2020-04-02 11:48:46,319 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20191223-_T16PDU_C_V not found
2020-04-02 11:48:46,321 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDU____T20191228.DBL.DIR not found
2020-04-02 11:48:46,321 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20191228-_T16PDU_C_V not found
2020-04-02 11:48:46,321 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2020-04-02 11:48:46,322 - Start-Maja - INFO - => processing date 20190805
2020-04-02 11:48:46,337 - Start-Maja - INFO - dates to process in backward mode :
2020-04-02 11:48:46,337 - Start-Maja - INFO - -- 20190805 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE
2020-04-02 11:48:46,337 - Start-Maja - INFO - -- 20190810 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190810T160911_N0208_R140_T16PDU_20190810T212513.SAFE
2020-04-02 11:48:46,338 - Start-Maja - INFO - -- 20190815 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190815T160829_N0208_R140_T16PDU_20190815T211555.SAFE
2020-04-02 11:48:46,338 - Start-Maja - INFO - -- 20190820 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190820T160901_N0208_R140_T16PDU_20190820T212247.SAFE
2020-04-02 11:48:46,339 - Start-Maja - INFO - -- 20190825 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190825T160829_N0208_R140_T16PDU_20190825T193312.SAFE
2020-04-02 11:48:46,339 - Start-Maja - INFO - -- 20190830 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190830T160901_N0208_R140_T16PDU_20190830T212354.SAFE
2020-04-02 11:48:46,340 - Start-Maja - INFO - -- 20190904 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190904T160829_N0208_R140_T16PDU_20190904T193256.SAFE
2020-04-02 11:48:46,341 - Start-Maja - INFO - -- 20190909 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190909T160901_N0208_R140_T16PDU_20190909T212411.SAFE
2020-04-02 11:48:46,362 - Start-Maja - INFO - #################################
2020-04-02 11:48:46,362 - Start-Maja - INFO - #################################
2020-04-02 11:48:46,362 - Start-Maja - INFO - processing /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE in backward mode
2020-04-02 11:48:46,362 - Start-Maja - INFO - Initialisation mode with backward is longer
2020-04-02 11:48:46,362 - Start-Maja - INFO - MAJA logfile: /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE.log
2020-04-02 11:48:46,362 - Start-Maja - INFO - #################################
2020-04-02T11:48:47.644243 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] vns::Data::ERROR: DataApplicationHandler(0x1a655d0): In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! [vnsDataApplicationHandler.cxx:GetListOfGippFilenames:1699]

2020-04-02T11:48:47.645796 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]

2020-04-02T11:48:47.647014 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] PreProcessing: vns::Data::ERROR: DataApplicationHandler(0x1a655d0): In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! [vnsDataApplicationHandler.cxx:1699] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:129]

2020-04-02T11:48:47.648199 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] [vnsMajaMainProcessor.cxx:main:129]

2020-04-02T11:48:47.649346 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]

2020-04-02 11:48:47,666 - Start-Maja - INFO - #######################################
2020-04-02 11:48:47,666 - Start-Maja - INFO - Error detected, see: /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE.log
2020-04-02 11:48:47,666 - Start-Maja - INFO - #######################################
(py2) ernesto@ernestomacpro:~/maja/Start-MAJA$
`
What is this file type (S2?_OPER_SSC_L2VALD_16PDU____T20190805.DBL.DIR not found)missing?

In my GIP folder I have:

and in my LUT folder:

Thank you for your help,
Inti

Answer 5 · 2020-04-02T17:45:24.000Z

Hi Inti,
The message "In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! " means that the DTM was not found.

It has to be stored in the DTM folder, where you launch start_maja from.
Olivier

Answer 6 · 2020-04-02T20:44:24.000Z

Hi Olivier,
It is strange because I do have a DTM folder inside star-maja folder in which I run the script as it can be seen here:

and inside the DTM folder I have the corresponding DTM files (2 files per tile) created with the preparation tools:

As you can see there are the files required for this tile:
S2__TEST_AUX_REFDE2_16PDU_0001.DBL.DIR
S2__TEST_AUX_REFDE2_16PDU_0001.HDR

What else could it be?
Thank you!

Answer 7 · 2020-04-04T07:52:15.000Z

Hi Olivier,
I have to say that the DTM files were created using the dtm preparation tools suggested by Peter (petket-5) in the previous post:#50 that are located in https://github.com/CNES/Start-MAJA/tree/reprog-rc1/prepare_mnt.

Afterwards, I just moved the created dtm files to the DTM folder inside original start-maja folder (coming from https://github.com/CNES/Start-MAJA). I hope this information can help find the error cause.

Thanks,

Answer 8 · 2020-04-04T09:27:01.000Z

Hi inti,
I checked, and (sorry to have forgotten this part), the DBL.DIR and HDR files are not stored in DTM folder. There is another folder in between:

[hagolle@visu02 Start_Maja]$ ls -1 DTM
S2__TEST_AUX_REFDE2_T11SPC_0001 
S2__TEST_AUX_REFDE2_T29TPH_0001  
S2__TEST_AUX_REFDE2_T31TFJ_0002  
S2__TEST_AUX_REFDE2_T32TNS_0001  
S2__TEST_AUX_REFDE2_T35JPM_0001
[hagolle@visu02 Start_Maja]$ ls DTM/S2__TEST_AUX_REFDE2_T11SPC_0001
S2__TEST_AUX_REFDE2_T11SPC_0001.DBL.DIR  S2__TEST_AUX_REFDE2_T11SPC_0001.HDR

I hope it's the last one !
Olivier

Answer 9 · 2020-04-05T16:59:42.000Z

Hi Olivier,
Thanks, that last hint of the folder in between the DTM and the DTM files (.DBL.DIR .HDR) was what was missing and now it worked fine and I was able to process images for 5 months of one tile since yesterday.

Thanks a lot,
Inti

Answer 10 · 2020-04-05T18:08:00.000Z

I told you you were close to succeed, but that's a good news, thanks for letting us know !
Peter @petket-5 has developped a new version of start_maja, which should be more user friendly. It is in its final tests.