System.FormatException: Input string was not in a correct format
DaveSkender opened this issue · 17 comments
The problem
We're still seeing users struggle with locale related conversion problems for quotes. It's certainly due to a misalignment of environment settings; however, I think there's more we can do to better understand it and possibly find more solutions to either mitigate the problem or to help users know exactly what they need to change when it does occur, maybe a self-test that they can run locally.
See original post by @ArtemHachaturov in DaveSkender/Stock.Indicators#1194 for more information and source quotes.
I think we're going to need to setup a rather elaborate set of tests using different locale settings on a GitHub Actions runner to better understand exactly what's happening. It's difficult to test on a local computer, and I'm not 100% confident I know what's happening. ru-Ru is a good test case to start with.
Any one of these can be a problem if they're not aligned on locale for both date/time and numerics:
- source quotes format
- environment/machine locale
- dotnet native locale
- python native locale
- Python.NET locale handling
- pandas dataFrame locale handling
- DLL compiled default locale handling
There's also this <NeutralLanguage>en-US</NeutralLanguage> tag hard-coded into the .NET library, but unsure if it's a problem.
Yes, I agree with you, of course, but I just want to solve my problem
Yes, I agree with you, of course, but I just want to solve my problem
If locale were compatible in all of the components listed above in your environment it should work. Until I run some more tests with all possible scenarios, I really can't tell which compatibility scenarios are the most problematic ... I honestly don't know where your implementation is failing.
It works with all the default settings that the GitHub Actions use, which is likely en_US.UTF-8.
Yeah we need to have some tests for this. I'll have a look when I get back home (I'm in workshop now)
After quick search, I found that we might convert Decimal into str incorrectly. Looks like just using str() is not affected by locale setting.
See here: https://stackoverflow.com/questions/31910741/how-can-i-locale-format-a-python-decimal-and-preserve-its-precision
Hi, @ArtemHachaturov . Thanks for your detail report. As I mentioned above, it looks like the locale setting does not apply in converting into str
Could you give me the results from the code below? That would be very helpful to find the reason
from decimal import Decimal
value = Decimal(12.345)
print(value) # case 1
print('{:n}'.format(value)) # case 2
print(format(value, 'n')) # case 3
from stock_indicators._cslib import CsDecimal
print(CsDecimal.Parse(format(value, 'n'))) # case 4Hi, @ArtemHachaturov . Thanks for your detail report. As I mentioned above, it looks like the
localesetting does not apply in converting intostrCould you give me the results from the code below? That would be very helpful to find the reasonfrom decimal import Decimal value = Decimal(12.345) print(value) # case 1 print('{:n}'.format(value)) # case 2 print(format(value, 'n')) # case 3 from stock_indicators._cslib import CsDecimal print(CsDecimal.Parse(format(value, 'n'))) # case 4
Thank you very much to you and your team for your feedback and for your responsiveness in solving this problem, I really appreciate it.
Here is the output of this code
12.3450000000000006394884621840901672840118408203125
12.3450000000000006394884621840901672840118408203125
12.3450000000000006394884621840901672840118408203125
Traceback (most recent call last):
File "e:\Binance Bot\python\14.py", line 8, in
print(CsDecimal.Parse(format(value, 'n'))) # case 4
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
System.FormatException: Input string was not in a correct format. at System.Number.ThrowOverflowOrFormatException(ParsingStatus
status, TypeCode type)
at System.Decimal.Parse(String s)
12.3450000000000006394884621840901672840118408203125
12.3450000000000006394884621840901672840118408203125
12.3450000000000006394884621840901672840118408203125
Traceback (most recent call last):
File "e:\Binance Bot\python\14.py", line 8, in
print(CsDecimal.Parse(format(value, 'n'))) # case 4
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
System.FormatException: The input string '12.3450000000000006394884621840901672840118408203125' was not in a correct format.
at System.Number.ThrowFormatException[TChar](ReadOnlySpan1 value) at System.Number.ParseDecimal[TChar](ReadOnlySpan1 value, NumberStyles styles,
NumberFormatInfo info)
at System.Decimal.Parse(String s)
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj,
Span`1 copyOfArgs, BindingFlags invokeAttr)
There is a small advance, the values fall into quotes
import pandas as pd
from stock_indicators import indicators, Quote
from babel.numbers import format_decimal
df = pd.read_csv('quotes.csv', parse_dates=['Time'])
def format_with_babel(number, locale='ru_RU'):
return format_decimal(number, locale=locale)
df['Open'] = df['Open'].apply(format_with_babel)
df['High'] = df['High'].apply(format_with_babel)
df['Low'] = df['Low'].apply(format_with_babel)
df['Close'] = df['Close'].apply(format_with_babel)
df['Volume'] = df['Volume'].apply(lambda x: format_with_babel(x, locale='ru_RU'))
quotes = [
Quote(date, open, high, low, close, volume)
for date, open, high, low, close, volume
in zip(df['Time'], df['Open'], df['High'], df['Low'], df['Close'], df['Volume'])
]
print(quotes)
#[<stock_indicators.indicators.common.quote.Quote object at 0x0000023D856F24C0>, <stock_indicators.indicators.common.quote.Quote object at 0x0000023D88E87080>....
print("Time | Open | High | Low | Close | Volume")
for q in quotes:
print(f"{q.date:%Y-%m-%d} | {q.open} | {q.high} | {q.low} | {q.close} | {q.volume})")
#But at the end there is an error
Time | Open | High | Low | Close | Volume
Traceback (most recent call last):
File "e:\Binance Bot\python\12.py", line 34, in
print(f"{q.date:%Y-%m-%d} | {q.open} | {q.high} | {q.low} | {q.close} | {q.volume})")
^^^^^^
File "C:\Users\russt\AppData\Local\Programs\Python\Python311\Lib\site-packages\stock_indicators\indicators\common\quote.py", line 22, in _get_open
return to_pydecimal(quote.Open)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\russt\AppData\Local\Programs\Python\Python311\Lib\site-packages\stock_indicators_cstypes\decimal.py", line 36, in to_pydecimal
return PyDecimal(str(cs_decimal))
^^^^^^^^^^^^^^^^^^^^^^^^^^
decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>]
@ArtemHachaturov
Ohhhh! Thank you. That was very helpful to figure out what's the problem.
Could you give me more information? It looks like your python runtime is using the locale which uses period(.) as a decimal point while your DotNet runtime is using comma(,) as a decimal point.
If you can give me the results after running the below, It'll be very appreciated.
On Python:
import locale
print(locale.getlocale(locale.LC_ALL))
print(locale.localeconv())On your terminal:
localeDear friends, I am glad to inform you that everything worked out for me and now everything works like a clock.
I had only one locale, this was RU, I understood that I still needed to install the language pack us, but for certain reasons I was unable to do this until I completely demolished the system and installed another Windows, where I also tried to download it manually the language pack was installed but was not applied until I used the following commands
- cmd
control intl.cpl,, /f:"%SystemRoot%\System32\LangRes.dll",/s:"0x00000409"
-PowerShell
Set-WinSystemLocale en-US
Set-Culture en-US
and restarting the computer
Then I used check current locale
import locale
current_locale = locale.getdefaultlocale()[0]
print(current_locale) #en_US
I realized that everything should work!
Everything looks amazing
df = pd.read_csv('quotes.csv', parse_dates=['Time'])
quotes = [
Quote(date, open, high, low, close, volume)
for date, open, high, low, close, volume
in zip(df['Time'], df['Open'], df['High'], df['Low'], df['Close'], df['Volume'])
]
print("Time | Open | High | Low | Close | Volume")
for q in quotes:
print(f"{q.date:%Y-%m-%d} | {q.open} | {q.high} | {q.low} | {q.close} | {q.volume})")
2018-07-27 | 275.57 | 275.68 | 272.34 | 273.35 | 79050080)
2018-07-30 | 273.44 | 273.61 | 271.35 | 271.92 | 65624404)
2018-07-31 | 272.76 | 273.93 | 272.34 | 273.26 | 70594928)
2018-08-01 | 273.49 | 274.04 | 272.1 | 272.81 | 55443260)
2018-08-02 | 271.38 | 274.48 | 271.15 | 274.29 | 65298924)
2018-08-03 | 274.43 | 275.52 | 274.23 | 275.47 | 55527740)
2018-08-06 | 275.51 | 276.82 | 275.08 | 276.48 | 40564136)
2018-08-07 | 277.21 | 277.81 | 277.06 | 277.39 | 44471960)
Thank you very much to you and your team, I wish you great success and prosperity
I suspect this essentially reset your PC environment locale to en_US, which would certainly solve your blocking issue; however, we’re still going to try and get it working better on other locales.
I am able to reproduce the symptom. Resetting system locale is clearly not an option in my case.
Indeed the problem is at
CsDecimal.Parse(str(decimal))
import locale print(locale.localeconv()) print(locale.getlocale(locale.LC_ALL))
{'int_curr_symbol':` '', 'currency_symbol': '', 'mon_decimal_point': '', 'mon_thousands_sep': '', 'mon_grouping': [], 'positive_sign': '', 'negative_sign': '', 'int_frac_digits': 127, 'frac_digits': 127, 'p_cs_precedes': 127, 'p_sep_by_space': 127, 'n_cs_precedes': 127, 'n_sep_by_space': 127, 'p_sign_posn': 127, 'n_sign_posn': 127, 'decimal_point': '.', 'thousands_sep': '', 'grouping': []}
File "", line 1, in
File "C:\Program Files\Python311\Lib\locale.py", line 608, in getlocale
raise TypeError('category LC_ALL is not supported')
@SnowyOw1 Thank you for reporting. We are fixing it now here. Could you try with this patched version? (it's still in reviewing yet but about to be merged.)
pip uninstall stock-indicators
pip install git+https://github.com/LeeDongGeon1996/Stock.Indicators.Python.git@fix/apply-locale+and If you let me know your locale setting on your machine, it'll be very appreciated.
Patched version released on test PyPI. Please have a test and any feedback appreciated.
pip install -i https://test.pypi.org/simple/ stock-indicators==1.3.1.dev9@SnowyOw1 Thank you for reporting. We are fixing it now here. Could you try with this patched version? (it's still in reviewing yet but about to be merged.)
+and If you let me know your
localesetting on your machine, it'll be very appreciated.
I hereby confirrm the fix is working. I didn't apply the full patch but monkey-patched sources.
My locale is
PS C:\> Get-WinSystemLocale
LCID Name DisplayName
---- ---- -----------
1049 ru-RU Russian (Russia)
PS C:\> (Get-WinSystemLocale).NumberFormat
CurrencyDecimalDigits : 2
CurrencyDecimalSeparator : ,
IsReadOnly : False
CurrencyGroupSizes : {3}
NumberGroupSizes : {3}
PercentGroupSizes : {3}
CurrencyGroupSeparator :
CurrencySymbol : ₽
NaNSymbol : не число
CurrencyNegativePattern : 8
NumberNegativePattern : 1
PercentPositivePattern : 1
PercentNegativePattern : 1
NegativeInfinitySymbol : -∞
NegativeSign : -
NumberDecimalDigits : 2
NumberDecimalSeparator : ,
NumberGroupSeparator :
CurrencyPositivePattern : 3
PositiveInfinitySymbol : ∞
PositiveSign : +
PercentDecimalDigits : 2
PercentDecimalSeparator : ,
PercentGroupSeparator :
PercentSymbol : %
PerMilleSymbol : ‰
NativeDigits : {0, 1, 2, 3...}
DigitSubstitution : None
Thank you for confirming. This patch has been published in v1.3.1