MAJA 4 Errors
jprankl opened this issue · 7 comments
Dear Oliver,
Dear Peter,
thanks for the new Maja release. After some dependency issues (already mentioned in #73 ) I was able to compile it. I can also run startmaja and it produces L2A products, but it always ends with an error. Same behavior of the binary version and my own compiled version. Did I mismatch some parameters or do I have some bad configuration?
I am running Maja on Ubuntu 20.04 in a docker container.
I started from scratch with:
/opt/maja/MAJA-4.2.1/bin/startmaja -f folders.txt -t 33UWP -s test_site -d 2020-01-01 -e 2020-07-01 --type_dem srtm -v --skip_errors --cams
Maja detected the following configuration:
2020-11-22 11:41:10,251 [INFO ] =============This is Start_Maja v4.2.0==============
2020-11-22 11:41:10,251 [DEBUG] Checking config file: folders.txt
2020-11-22 11:41:10,258 [DEBUG] Config file parsed without errors.
2020-11-22 11:41:10,258 [DEBUG] Setting site and tile
2020-11-22 11:41:10,439 [WARNI] 33UWP
2020-11-22 11:41:10,615 [DEBUG] Found site test_site and tile 33UWP
2020-11-22 11:41:10,615 [INFO ] Detecting input products...
2020-11-22 11:41:10,699 [INFO ] 152 L1C product(s) detected for site test_site and tile 33UWP in /data/Sen4Agri/sentinel_products_L1C-test/test_site/T33UWP
2020-11-22 11:41:10,699 [WARNI] No L2A products detected for site test_site and tile 33UWP in /data/Sen4Agri/sentinel_products_L2A_MAJA-test3/test_site/33UWP
Enforce tm. HACK jp, 21.11.2020
2020-11-22 11:41:10,700 [INFO ] Searching for CAMS
2020-11-22 11:41:11,183 [INFO ] ...found 86 CAMS files
2020-11-22 11:41:11,183 [INFO ] Checking GIPP files
2020-11-22 11:41:11,183 [INFO ] Setting up GIPP folder: /data/Sen4Agri/resources_maja4/gipp_maja
2020-11-22 11:41:12,361 [INFO ] Searching for DTM
2020-11-22 11:41:12,364 [INFO ] Found DTM: /data/Sen4Agri/resources_maja4/dtm/S2__TEST_AUX_REFDE2_33UWP_1001.HDR
2020-11-22 11:41:13,668 [INFO ] GIPP Creation succeeded for SENTINEL2_TM_CAMS
2020-11-22 11:41:14,067 [INFO ] 91 workplan(s) successfully created:
DATE | TILE | MODE | L1-PRODUCT | ADDITIONAL INFO
2020-01-01 10:03:19 | 33UWP | BACKWARD | S2B_MSIL1C_20200101T100319_N0208_R122_T33UWP_20200101T110654.SAFE | Backward of 8 products with CAMS
2020-01-03 09:54:01 | 33UWP | NOMINAL | S2A_MSIL1C_20200103T095401_N0208_R079_T33UWP_20200103T102451.SAFE | L2 from previous with CAMS
2020-01-06 10:04:01 | 33UWP | NOMINAL | S2A_MSIL1C_20200106T100401_N0208_R122_T33UWP_20200106T103423.SAFE | L2 from previous with CAMS
2020-01-08 09:53:09 | 33UWP | NOMINAL | S2B_MSIL1C_20200108T095309_N0208_R079_T33UWP_20200108T102330.SAFE | L2 from previous with CAMS
2020-01-09 10:14:01 | 33UWP | NOMINAL | S2A_MSIL1C_20200109T101401_N0208_R022_T33UWP_20200109T104556.SAFE | L2 from previous with CAMS
2020-01-11 10:02:59 | 33UWP | NOMINAL | S2B_MSIL1C_20200111T100259_N0208_R122_T33UWP_20200111T120153.SAFE | L2 from previous with CAMS
...
The following error comes for the first product:
...
2020-11-22 11:46:51 [4092] INFO MAJA 4.2 process_one_product:830 Starting Scientifics processing done.
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:831 L1 Product : Date 2020-01-14 10:17:04.653261
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:832 * p_finalize_backward = False
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:833 * l_WriteL2Products = True
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:834 * p_WriteL2ProductToL2Resolution = False
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:835 * p_write_temporary_l2_product = True
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:836 * p_WriteHeaders = False
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:837 * l_DealingLTC = True
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:838 * l_IgnoreCurrentLTC = False
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:839 -> self._productIsValid = False
Traceback (most recent call last):
File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/launcher/maja.py", line 100, in <module>
main()
File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/launcher/maja.py", line 81, in main
lprocessor.scientific_processing()
File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/processor/l2_backward_processor.py", line 221, in scientific_processing
l_GIPPL2COMMHandler)
File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/processor/l2_backward_processor.py", line 539, in single_product_processing
checking_conditional_clouds)
File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/processor/l2_processor.py", line 840, in process_one_product
LOGGER.debug(" -> p_checking_conditional_clouds = " + str(p_checking_conditional_clouds[0]))
TypeError: 'bool' object is not subscriptable
2020-11-22 11:46:52 [4092] DEBUG MAJA 4.2 __del__:75 Destructing : Stats
...
and for the following products:
...
2020-11-22 13:06:21 [14002] DEBUG MAJA 4.2 __del__:79 RAM : 4388 : 4388 : 0 MB
2020-11-22 13:06:21 [14002] INFO MAJA 4.2 post_processing:542 Starting Postprocessing
2020-11-22 13:06:21 [14002] INFO MAJA 4.2 main:85 Cleaning Temporary datas ...
Traceback (most recent call last):
File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/launcher/maja.py", line 100, in <module>
main()
File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/launcher/maja.py", line 89, in main
["lsof", "+D", str(app_handler.get_working_directory())])
File "/opt/maja/MAJA-4.2.1/lib/python3.7/subprocess.py", line 323, in call
with Popen(*popenargs, **kwargs) as p:
File "/opt/maja/MAJA-4.2.1/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/opt/maja/MAJA-4.2.1/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'lsof': 'lsof'
2020-11-22 13:06:22 [14002] DEBUG MAJA 4.2 free_otb_app:66 Removing anonymous
...
Some detailed log files:
log_verbose_cmd_20201122.txt.tar.gz
S2B_MSIL1C_20200101T100319_N0208_R122_T33UWP_20200101T110654.log.tar.gz
S2A_MSIL1C_20200103T095401_N0208_R079_T33UWP_20200103T102451.log.tar.gz
Thanks for your help.
Johann
Hello, I bumped into the same error about lsof. It seems lsof is a dependency for MAJA. I just installed it ($ yum install -y lsof) and now restarting the task.
EDIT: seems to be working nicely, now
Ahh, yes, thanks. I installed several dependencies, but that I missed.
The second error (I posted above) is gone, the first one still appears, but maybe that's because of initialization on a totally cloudy image.
So I will close it.
Thanks again!
[offtopic] @jprankl is it possible to share your Dockerfile?
Dear @c00kiemon5ter , I did not write a proper Dockerfile, but only used an existing Ubuntu 20.04 container and installed Maja manually. Would be nice to have a Dockerfile:-) For now, I can write down the essential steps I did?
Yes, that would be nice! Given that, I can try to come up with a proper Dockerfile ;)
@c00kiemon5ter I created a separate issue #80 with the steps I did. I hope it helps.
Hey @c00kiemon5ter in case you are still interested, here is a working Dockerfile: https://github.com/rohitshetty/maja-docker