Missing export to IBFV2.00 INTERMAGNET Baseline Format
mlnorthernswe opened this issue · 11 comments
I can't see how I can get IBFV2.00 file that is used in INTERMAGNET yearly DVD: https://www.intermagnet.org/data-donnee/formats/ibfv200-eng.php
It should be suitable to create that file when exporting to IAF format as those files are used in mentioned INTERMAGNET yearly DVD.
The README.IMO file is created already when exporting to IAF.
Also https://www.intermagnet.org/data-donnee/formats/iyfv101-eng.php should also be created when exporting to IAF as that file is also used in the INTERMAGNET yearly DVD.
Both files can be created using MagPy. They are not created automatically as they are based on different sources (Di values plus adopted baseline, BLV) or require a distinct coverage (IYFV).
Regarding IYFV
When i.e. IAF exports are finished or any other data file containing one year of one-minute data is available: Open the full year of one-minute data - got to menu file -> export.
Within the "export" window select an appropriate directory and choose the "IYVF" format from the dropdown menu. Do NOT change any further export options. The suggested file name will look like YEARMEANNone.IMO.
If (a) a file called YEARMEAN.IMO is already existing within the selected directory a new line like
2022.500 4 47.2 64 26.6 21071 20997 1758 44062 48841 A XYZG
will be appended at the correct position.
If (b) such YEARMEAN file is not yet existing, it will be newly created making use of information from the meta panel.
Please note: IAF contain data rounded to 0.1 nT. I usually prefer to create the IYVF input from my full resolution one-minute data product which I typically store as an CDF structure.
Creating input lines for quiet days and disturbed days is currently only possible using backend methods.
Regarding BLV
This should be possible by using your analyzed basevalue data. Store this data after DI analysis, eventually already with the adopted baseline (using "ANALYSIS" -> "FIT"). When exporting such data you only will have three format options (BLV, PYCDF, and PYSTR). I typically keep a PYSTR (ascii) file which contains all underlying data from the DI analysis. Unfortunately, BLV is not yet working properly in the XMAGPY frontend - I get some permission denied error. I will dig into this and come back to you. (I typically use a quasi-automated backend script to create all my IM and IAGA dissimination files from my one-second definitive results).
Thank you for the answer.
I wanted to export to IYFV with no full year data. It sounds unlogical but I use an software (imcdview: https://geomag.bgs.ac.uk/data_service/intermagnet/home.html) that use that file and I check data after each month we have cleaned the data (hence I don't have full year data).
Regarding BLV I did as you suggested in xmagpy. However the scalar data is not included in that file (it's marked with gaps (99999.00)) and hence delta F also is filled with gaps (888.00). How can I get scalar data included?
It looks like this:
001 10481.50 31.86 52013.86 99999.00 888.00 c
002 10481.51 31.89 52013.86 99999.00 888.00 c
003 10481.52 31.90 52013.86 99999.00 888.00 c
004 10481.53 31.91 52013.87 99999.00 888.00 c
005 10481.54 31.92 52013.87 99999.00 888.00 c
006 10481.56 31.93 52013.87 99999.00 888.00 c
I did a number of changes to solve these issues. Firstly, I corrected the IYFV export scheme and, secondly, I included a number of options to XMagPy which were previously only accessible using backend methods. All updates will be included in version 1.1.3. Details below
IYVF:
- in old magpy versions there was a hard criteria that yearly mean values will only be calculated if a data for more than 90% of the days is available. I have removed that now (in 1.1.3). If data coverage is less than 90% (and neither Q or D are selected) then the data set is marked as I (incomplete).
- starting with version 1.1.3: when exporting data to IYFV using XMagPy an additional window will pop up (after the general export options window) with IYFV specific options. Here you can selected what kind of data you are exporting (A, Q, D, I).
BLV:
- adopted scalar baseline: in order to fit and export such data you need to select this column (panel "Data" -> "Select columns" -> activate "dF" in addition to the default selected). Then fit and finally export.
- starting with 1.1.3, an additional BLV specific window will open after the general export option window. Here you may select the year covered by the BLV file and you can select the fitting method used for the adopted baseline. Currently supported are the default fit, making use of the general fitting function as outlined above, but you may also select just to use the mean or median of the provided scalar values within the basevalue data. A completely different complex fit is still only possible within the backend.
- representative delta F values making use of a complementary definitive data set can now (1.1.3) also be included into BLV using the XMagPy frontend. The BLV specific export window allows you to select a previously opened data source for such data. In order to use it: Firstly, open one-year (or less if not available) of definitive one-minute data. Secondly, open basevalue data, eventually perform fits and then export to BLV. The BLV specific export will now allow for choosing the initially opened definitive one-minute data within a dropdown and include daily averages of delta F's into the BLV data. Please note: as daily means are obtained, BLV export might need a while.
Changes are currently available within the develop branch and can be found here:
8e07a28
bde5cdf
d5453a5
As soon as 1.1.3 is tagged, and provided that the issues are solved, I will close this one.
It worked to install the develop branch with:
pip3 install git+https://github.com/geomagpy/magpy.git@develop
But xmagpy shows still version 1.1.2 in About. Is it correct?
In venv I managed to install develop branch correctly so now I can test that branch.
No, develop version number should be something like1.1.3b00x. Good that you were able to install it using venv.
I have tested a little. But I get the same result as before. I do like the following:
- I load the absolute measurements which are in MAGPYNEWABS-format from here: https://www2.irf.se/maggraphs/absolute_measurements/ (I choose 2022-12-29 to 2023-04-11). Vario/Scalar from here: https://www2.irf.se/maggraphs/vario_scalar/.
- I press "Analyze" (it should be good with feedback in xmagpy how far the analyze has reached (i.e what absolute measurement that is processed))
- I go the "Data" tab. As our Magnetometer measures XYZ I convert to XYZ by selecting xyz. I select columns X, Y, Z, H-base, D-base, Z-base and dF.
- I go back to "Analyzis" tab. Press "Fit". Choose the default settings and press "Apply".
- I go to "File" tab and export data by choosing "BLV".
- Choose Year: 2023. Adopted scalar F difference: default. Representative delta F source (one-minute): None. Apply.
The resulting file can be found here: https://www2.irf.se/maggraphs/absolute_measurements/KIR2023.blv
I also wonder about MAGPYNEWABS-format. In that format you can't present the "pillar difference" (baseline scalar F) value as it's added to the PPM values. In https://intermagnet.github.io/docs/Technical-Manual/technical_manual.pdf page 115 the IBF format is described and there that value should be presented but as that value is not included in MAGPYNEWABS I don't think that it's possible or am I wrong?
Just some quick clarification questions:
- The procedure seems ok for me. Is our magnetometer oriented along magnetic or geographic coordinates? Geographic I guess as you say you measure XYZ and not HEZ. The DI analysis method (DTU scheme) is actually optimized for HEZ variometer orientation, particularly when using residual methods, as those residuals are measured along the E axis. I still have to check the sensitivity/validity for other setups
- I will need to verify the file contents/sources for scalar baselines. It seems as if your differences are particularly small, below the 2 digit BLV resolution, eventually related to the fact that pillar diffs are already included in your ppms. Hope I can reply soon with some more detail.
Are you attending the IAGA workshop for some in situ discussion?
Yes, our magnetometer is aligned along geographic coordinates XYZ (of course X to true north, and Y west and Z down).
I understand now what you mean with "small differences in scalar baselines". The scalar baseline is the pillar difference that we measure a couple of times each year. Between those measurements that baseline is you can say a constant, therefore the small differences. See page 115-116 in https://intermagnet.github.io/docs/Technical-Manual/technical_manual.pdf that defines IBFV2.00 format. See also page 17 about "CALCULATION OF THE
DIFFERENCE BETWEEN F PILLAR AND ABSOLUTE PILLAR". As pillar diff is already included in MAGPYNEWABS format the scalar baseline will be 0 in IBFV2.00 or am I wrong? Therefore I miss scalar baseline in MAGPYNEWABS format.
This is our own BLV file if you want to compare: https://www.irf.se/observatory/mag/intermagnet_maglinux/cd/MAG2023/KIR/KIR2023.BLV
But we are using a different spline method (monotonic spline (Steffen method: https://adsabs.harvard.edu/full/1990A%26A...239..443S) with the knots at the absolute measurements time points (bad absolute measurements filtered away)). I wanted that the baseline should contain the good absolute measurements as at those points the quality of data is best (dF->0 etc.), but at the same time the interpolation should be aware of trends from earlier and later absolute measurements which the Steffen method is suitable for.
No, not in place anyway, I will attend IAGA workshop. If there are online sessions I maybe can attend.
For now I have updated the README with detailed information on the DI-flux analysis and basevalue adoption technique as currently used by MagPy. In this section (https://github.com/geomagpy/magpy/tree/develop#2117-details-on-di-flux-analysis-and-calculation-of-basevalues) you will find basically all details and also a short section on scalar values and how they are incorporated into the result. In your case, the scalar values come from a separate pier. They are however also included into your individual DI data structure files. Thus , the analysis routine treats them as measurements at the same pier. Additionally, scalar data, corrected for pier diff, is provided from an external source. The routine calculates a pier difference between external data and the ones provided within the DI struct, which are identical and therefore pier diff is zero.
As I understand so far for your data source, the DI data structures should not contain F values, as you don't have measurements at the DI-flux pier. You should only provide the raw data from your scalar source (vario/scalar button on DI panel) and provide the pier difference at the field dideltaF within the Analysis parameter window on the DI panel. This way, a correct BLV output will be obtained and delta F's as provided will be correctly stored. I hope the extension of the README helps. I also found an issue with baseline adoption if non-quasi absolute variations are used for DI analysis and corrected that. Furthermore I am working on a solution to also get XYZ BLV files in case of XYZ variation data (see also issue #123). Updates are available on develop branch and will be part of 1.1.3.
Thanks for the thorough explanation. In the data in vario/scalar in DI panel I have already added pillar difference so dideltaF should be 0 there. I got however a rather different value in Y baseline compared to own calculations. D and I values are the same you can say for the absolute measurements (some decimals apart). I will test again if I get the same result and come back.
Just uploaded a new update into the development branch. The new version fully supports XYZ and HEZ variation data (and also basevalue outputs in XYZ). Further details are here #123 (comment)