Larch 0.9.75 Data Loss

Question

Larch 0.9.75 Data Loss

Closed this issue a month ago · 11 comments

I updated to the new Larch 0.9.75 yesterday, and I'm having issues reading data files created using this version. I normalised/deglitched and fitted the pre-edge peaks using Larch 0.9.75 and saved this. Upon opening the file today, no spectra are plotted.

I can open and view data files I normalised using the previous version; however, the changes aren't saved if a pre-edge fit is performed and the data points disappear.

Here is the file which doesn't contain information after performing the pre-edge fit:
Test.zip

I used Windows to install it, and I couldn't find the Windows installer for version 0.9.74. If you have this at hand, I would like to try to go back to the old version and see if I still have my data.

This is the command error using the test file provided above:

run in conda environment "base":

C:\ProgramData\xraylarch\Scripts\larix.exe

decode failed:: lfmp4pris '{"class": "Group", "athena_id": "lfmp_4_prist
decode failed:: peak result '{"class": "Group", "energy": {"class": "A
decode failed:: lfmp4pris '{"class": "Group", "athena_id": "lfmp_4_prist
decode failed:: peakresult '{"class": "Group", "energy": {"class": "A
Traceback (most recent call last):
File "C:\ProgramData\xraylarch\Lib\site-packages\larch\wxxas\xasnorm_panel.py", line 586, in onPlotSel
yarray_name = plot_choices[ytitle]
~~~~~~~~~~~~^^^^^^^^
KeyError: 'Normalized μ(E)'
Traceback (most recent call last):
File "C:\ProgramData\xraylarch\Lib\site-packages\larch\wxxas\xasnorm_panel.py", line 586, in onPlotSel
yarray_name = plot_choices[ytitle]
~~~~~~~~~~~~^^^^^^^^
KeyError: 'Normalized μ(E)'
Traceback (most recent call last):
File "C:\ProgramData\xraylarch\Lib\site-packages\larch\wxxas\xasnorm_panel.py", line 586, in onPlotSel
yarray_name = plot_choices[ytitle]
~~~~~~~~~~~~^^^^^^^^
KeyError: 'Normalized μ(E)'

The problem is that I spent about 10 hours yesterday working on merging, deglitching, calibrating, and pre-edge fitting many files. The following log is from a file that contains 37 groups that were all normalised/fitted, etc. Has all of this information been lost?

run in conda environment "base":

C:\ProgramData\xraylarch\Scripts\larix.exe

decode failed:: _10__merge_2 '{"class": "Group", "_tmp": {"class": "Arr
decode failed:: _10__merge_2_1 '{"class": "Group", "_tmp": {"class": "Arr
decode failed:: _10__merge_2_2 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: _10__merge_2_3 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: a114mnoda '{"class": "Group", "_100": {"class": "Arr
decode failed:: a114mnoda_1 '{"class": "Group", "_100": {"class": "Arr
decode failed:: a114mnoda_2_1 '{"class": "Group", "_100": {"class": "Arr
decode failed:: b109lfpmn '{"class": "Group", "_100": {"class": "Arr
decode failed:: b113lfpmn '{"class": "Group", "_100": {"class": "Arr
decode failed:: b115mn2o3 '{"class": "Group", "_100": {"class": "Arr
decode failed:: b115mn2o3_2_1 '{"class": "Group", "_100": {"class": "Arr
decode failed:: b117mno2d '{"class": "Group", "_100": {"class": "Arr
decode failed:: c111lfpmn '{"class": "Group", "_100": {"class": "Arr
decode failed:: c112lfpmn '{"class": "Group", "_100": {"class": "Arr
decode failed:: c116mn3o4 '{"class": "Group", "_100": {"class": "Arr
decode failed:: cmsti '{"class": "Group", "athena_id": "cmsti", "ath
decode failed:: d84lfmp2e '{"class": "Group", "_100": {"class": "Arr
decode failed:: demur '{"class": "Group", "athena_id": "demur", "ath
decode failed:: dltaz '{"class": "Group", "athena_id": "dltaz", "ath
decode failed:: e110lfpmn '{"class": "Group", "_100": {"class": "Arr
decode failed:: g85lfmp2p '{"class": "Group", "_100": {"class": "Arr
decode failed:: g86mnfoil '{"class": "Group", "_100": {"class": "Arr
decode failed:: iydqw '{"class": "Group", "athena_id": "iydqw", "ath
decode failed:: lfmp_4__2_5 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4__3 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4__4 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4__5 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4_ocv_lipf6 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4_ocv_litfsi '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4_soc_2 '{"class": "Group", "_tmp": {"class": "Arr
decode failed:: lfmp_4_soc_2_1 '{"class": "Group", "_tmp": {"class": "Arr
decode failed:: peakresult '{"class": "Group", "energy": {"class": "A
decode failed:: rfppt '{"class": "Group", "athena_id": "rfppt", "ath
decode failed:: uiutz '{"class": "Group", "athena_id": "uiutz", "ath
decode failed:: wfwzi '{"class": "Group", "athena_id": "wfwzi", "ath
decode failed:: wilrk_1 '{"class": "Group", "athena_id": "wilrk", "ath
decode failed:: wkvny '{"class": "Group", "athena_id": "wkvny", "ath
decode failed:: yzozm '{"class": "Group", "athena_id": "yzozm", "ath
decode failed:: _10__merge_2 '{"class": "Group", "_tmp": {"class": "Arr
decode failed:: _10__merge_2_1 '{"class": "Group", "_tmp": {"class": "Arr
decode failed:: _10__merge_2_2 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: _10__merge_2_3 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: a114mnoda '{"class": "Group", "_100": {"class": "Arr
decode failed:: a114mnoda_1 '{"class": "Group", "_100": {"class": "Arr
decode failed:: a114mnoda_2_1 '{"class": "Group", "_100": {"class": "Arr
decode failed:: b109lfpmn '{"class": "Group", "_100": {"class": "Arr
decode failed:: b113lfpmn '{"class": "Group", "_100": {"class": "Arr
decode failed:: b115mn2o3 '{"class": "Group", "_100": {"class": "Arr
decode failed:: b115mn2o3_2_1 '{"class": "Group", "_100": {"class": "Arr
decode failed:: b117mno2d '{"class": "Group", "_100": {"class": "Arr
decode failed:: c111lfpmn '{"class": "Group", "_100": {"class": "Arr
decode failed:: c112lfpmn '{"class": "Group", "_100": {"class": "Arr
decode failed:: c116mn3o4 '{"class": "Group", "_100": {"class": "Arr
decode failed:: cmsti '{"class": "Group", "athena_id": "cmsti", "ath
decode failed:: d84lfmp2e '{"class": "Group", "_100": {"class": "Arr
decode failed:: demur '{"class": "Group", "athena_id": "demur", "ath
decode failed:: dltaz '{"class": "Group", "athena_id": "dltaz", "ath
decode failed:: e110lfpmn '{"class": "Group", "_100": {"class": "Arr
decode failed:: g85lfmp2p '{"class": "Group", "_100": {"class": "Arr
decode failed:: g86mnfoil '{"class": "Group", "_100": {"class": "Arr
decode failed:: iydqw '{"class": "Group", "athena_id": "iydqw", "ath
decode failed:: lfmp_4__2_5 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4__3 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4__4 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4__5 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4_ocv_lipf6 '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4_ocv_litfsi '{"class": "Group", "atsym": "Mn", "callargs":
decode failed:: lfmp_4_soc_2 '{"class": "Group", "_tmp": {"class": "Arr
decode failed:: lfmp_4_soc_2_1 '{"class": "Group", "_tmp": {"class": "Arr
decode failed:: peakresult '{"class": "Group", "energy": {"class": "A
decode failed:: rfppt '{"class": "Group", "athena_id": "rfppt", "ath
decode failed:: uiutz '{"class": "Group", "athena_id": "uiutz", "ath
decode failed:: wfwzi '{"class": "Group", "athena_id": "wfwzi", "ath
decode failed:: wilrk_1 '{"class": "Group", "athena_id": "wilrk", "ath
decode failed:: wkvny '{"class": "Group", "athena_id": "wkvny", "ath
decode failed:: yzozm '{"class": "Group", "athena_id": "yzozm", "ath
Traceback (most recent call last):
File "C:\ProgramData\xraylarch\Lib\site-packages\larch\wxxas\xasnorm_panel.py", line 586, in onPlotSel
yarray_name = plot_choices[ytitle]
~~~~~~~~~~~~^^^^^^^^
KeyError: 'Normalized μ(E)'

Answer 1 · 2024-03-26T13:21:52.000Z

@rparm121 very sorry for the trouble. I'll have to investigate, but this looks serious. I went back to "0.9.74" as the "announced released" so that automatic updates will not advertise 0.9.75.

To rollback, you should be able to open a command shell and do

C:\ProgramData\xraylarch\Scripts\pip.exe install xraylarch==0.9.74

Answer 2 · 2024-03-26T14:23:00.000Z

@newville

Brilliant. I have successfully rolled back, and all the data is there. Hopefully, the fix will be easy to get around. Thank you for all your help. Ps. The new plot display/features look great!

Answer 3 · 2024-03-28T18:55:46.000Z

@rparm121 I think I have this working better in the master branch. I saw two separate problems with your example (so thanks twice!)

a) there was some weird problem about a group that ended with a "_1" that may have been deleted during the session but somehow got partly saved in the Session file, and that could not be restored and used. I believe that group/spectrum was deleted, so that it should have not been saved in the first place, and then seen that it should not be restored. The symptom that the "Normalized μ(E)" array could not be plotted was really due to this (but also that error can be avoided too).

b) restoring a saved Pre-edge Peak fit just plain failed. It was worse than that, as saving the Pre-edge Peak Fits failed, writing data to the session that could not be properly restored. That's now fixed, and saving/restoring Pre-edge Peak Fits now works.

Sorry for the trouble and loss of time/fits. I'll try to get 0.9.76 released that fixes these (and other problems!) within a week.

Answer 4 · 2024-04-26T13:53:45.000Z

@newville is this fixed in 0.9.76?

Answer 5 · 2024-04-26T21:23:57.000Z

Yes, or at least "I think so". It works for me with your data. But if you have problems with it, let me know.

Answer 6 · 2024-05-03T17:14:51.000Z

We have been using Larix for FEFF-based EXAFS fitting and seem to have a similar problem in 0.9.76. Loaded a spectrum from an Athena PRJ. Processed data. Run a FEFF calculation. Fitted. Saved LARIX file. That was all fine and it reopened. After reopening, we made some changes to fitting and decided to save a new LARIX file because we are refining a model that we eventually want to apply to multiple datasets. After saving that new file, LARIX was closed. When reopening, the data (a spectrum) were missing. Reverting back to 0.9.74 fixed the problem.

Answer 7 · 2024-05-03T20:06:36.000Z

@catalano-wustl Sorry for the trouble and thanks for the report -- I'll investigate.

Answer 8 · 2024-05-04T00:03:17.000Z

@maurov @catalano-wustl I think I have found and fixed this. The good news (if any) is that the saved projects are probably actually fine and will be restorable. I'll try to get out a fix soon.

Answer 9 · 2024-05-04T21:22:43.000Z

Thanks! The file works fine in 0.9.74, so I think it will be fine in the updated version. We are enjoying FEFF-based EXAFS fitting in Larix.

Answer 10 · 2024-05-06T06:35:38.000Z

@newville I confirm this is fixed master.

Answer 11 · 2024-05-23T07:27:20.000Z

This is fixed in 0.9.78