Parsing Error for high precision double values

Question

Parsing Error for high precision double values

Closed this issue 4 years ago · 1 comments

We ran into an issue using luciphor2 with parsing high precision double values in the mzML files:

luciphor2 failed with:

umich.ms.fileio.exceptions.FileParsingException: java.lang.NumberFormatException: Too many digits - Overflow

It seems that the "javolution" library is not able to parse doubles which have more than 19 digits.

https://github.com/chhh/MSFTBX/blob/master/MSFileToolbox/src/main/java/umich/ms/fileio/filetypes/mzml/MZMLMultiSpectraParser.java#L286

Error:

Exception in thread "main" umich.ms.fileio.exceptions.FileParsingException: java.lang.NumberFormatException: Too many digits - Overflow
	at umich.ms.fileio.filetypes.xmlbased.AbstractXMLBasedDataSource.parseScan(AbstractXMLBasedDataSource.java:679)
	at lucxor.globals.read_mzML(globals.java:652)
	at lucxor.globals.read_in_spectra(globals.java:622)
	at lucxor.LucXor.main(LucXor.java:71)
Caused by: java.lang.NumberFormatException: Too many digits - Overflow
	at javolution.text.TypeFormat.parseDouble(TypeFormat.java:518)
	at javolution.text.TypeFormat.parseDouble(TypeFormat.java:579)
	at javolution.text.CharArray.toDouble(CharArray.java:481)
	at umich.ms.fileio.filetypes.mzml.MZMLMultiSpectraParser.tagCvParamStart(MZMLMultiSpectraParser.java:586)
	at umich.ms.fileio.filetypes.mzml.MZMLMultiSpectraParser.call(MZMLMultiSpectraParser.java:195)
	at umich.ms.fileio.filetypes.mzml.MZMLMultiSpectraParser.call(MZMLMultiSpectraParser.java:57)
	at umich.ms.fileio.filetypes.xmlbased.AbstractXMLBasedDataSource.parseScan(AbstractXMLBasedDataSource.java:669)
	... 3 more

Answer 1 · 2020-04-08T16:26:39.000Z

Hi Oliver,

Do you really need 20 decimal digits? Fixing that might be easier.
I think this is the correct behavior - an IEEE 754 double can only have 15-17 digits of precision (it's a complicated topic overall). Parsing something with 19 digits would be loss of precision and require a specialized numeric type.
The exception you see is thrown deliberately, not just by neglect:
https://github.com/chhh/javolution-msftbx/blob/1ae76375165dbf89b243533863b43de0c994422c/src/main/java/javolution/text/TypeFormat.java#L518
Just in case, made a test to see if it's misbehaving, looks to be working fine:

[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 12 decimal places, 1 decimal digits: 0.00000000001
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.0E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 13 decimal places, 2 decimal digits: 0.000000000012
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.2E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 14 decimal places, 3 decimal digits: 0.0000000000123
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.23E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 15 decimal places, 4 decimal digits: 0.00000000001234
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.234E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 16 decimal places, 5 decimal digits: 0.000000000012345
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.2345E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 17 decimal places, 6 decimal digits: 0.0000000000123456
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.23456E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 18 decimal places, 7 decimal digits: 0.00000000001234567
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.234567E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 19 decimal places, 8 decimal digits: 0.000000000012345678
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.2345678E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 20 decimal places, 9 decimal digits: 0.0000000000123456789
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.23456789E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 21 decimal places, 10 decimal digits: 0.00000000001234567891
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.234567891E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 22 decimal places, 11 decimal digits: 0.000000000012345678912
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.2345678912E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 23 decimal places, 12 decimal digits: 0.0000000000123456789123
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.23456789123E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 24 decimal places, 13 decimal digits: 0.00000000001234567891234
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.234567891234E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 25 decimal places, 14 decimal digits: 0.000000000012345678912345
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.2345678912345E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 26 decimal places, 15 decimal digits: 0.0000000000123456789123456
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.23456789123456E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 27 decimal places, 16 decimal digits: 0.00000000001234567891234567
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.234567891234567E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 28 decimal places, 17 decimal digits: 0.000000000012345678912345678
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.2345678912345678E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 29 decimal places, 18 decimal digits: 0.0000000000123456789123456789
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.2345678912345678E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 30 decimal places, 19 decimal digits: 0.00000000001234567891234567891
[Test worker] DEBUG javolution.text.CharArrayTest - Parsed double: 1.2345678912345678E-11
[Test worker] DEBUG javolution.text.CharArrayTest - Trying to convert a digit with 31 decimal places, 20 decimal digits: 0.000000000012345678912345678912
[Test worker] DEBUG javolution.text.CharArrayTest - Conversion failed
java.lang.NumberFormatException: Too many digits - Overflow
	at javolution.text.TypeFormat.parseDouble(TypeFormat.java:518)
	...