Today Error Updating
Closed this issue · 19 comments
Hello Benny,
Today getting error while updating.
(.venv) saurabhgarg@MacBook-Pro src % python3 init.py
sh: color: command not found
INFO: 20-09-2024 19:07 - main - Downloading Files
INFO: 20-09-2024 19:07 - defs.defs - Starting Data Sync
INFO: 20-09-2024 19:07 - defs.defs - EOD sync complete
INFO: 20-09-2024 19:07 - defs.defs - ### Nifty PE at 23.98 ###
INFO: 20-09-2024 19:07 - defs.defs - Index sync complete.
INFO: 20-09-2024 19:07 - defs.defs - Making adjustments for splits and bonus
CRITICAL: 20-09-2024 19:07 - root - Adjustment Error - Context PHOENIXLTD - Bonus - 20-Sep-2024
ERROR: 20-09-2024 19:07 - main - Error while making adjustments.
All adjustments have been discarded.
Traceback (most recent call last):
File "/Users/saurabhgarg/shivshakti/eod2/src/init.py", line 125, in
defs.adjustNseStocks()
File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 833, in adjustNseStocks
raise e
File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 813, in adjustNseStocks
commit = makeAdjustment(
^^^^^^^^^^^^^^^
File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 656, in makeAdjustment
df = pd.read_csv(file, index_col="Date", parse_dates=["Date"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
return _read(filepath_or_buffer, kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 626, in _read
return parser.read(nrows)
^^^^^^^^^^^^^^^^^^
File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1923, in read
) = self._engine.read( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 234, in read
chunks = self._reader.read_low_memory(nrows)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "parsers.pyx", line 838, in pandas._libs.parsers.TextReader.read_low_memory
File "parsers.pyx", line 905, in pandas._libs.parsers.TextReader._read_rows
File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows
File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status
File "parsers.pyx", line 2061, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 9 fields in line 4312, saw 17
INFO: 20-09-2024 19:07 - defs.defs - Rolling back changes from 2024-09-20: /Users/saurabhgarg/shivshakti/eod2/src/eod2_data/daily
INFO: 20-09-2024 19:07 - defs.defs - Rollback successful
Thanks
Saurabh
#metoo same error :|
I had a slightly different error, a keyerror. Can you send me Phoenixltd.csv file. I will have a look and let you know
On 20 September 2024 13:41:49 UTC, Saurabh @.> wrote: Hello Benny, Today getting error while updating. (.venv) @. src % python3 init.py sh: color: command not found INFO: 20-09-2024 19:07 - main - Downloading Files INFO: 20-09-2024 19:07 - defs.defs - Starting Data Sync INFO: 20-09-2024 19:07 - defs.defs - EOD sync complete INFO: 20-09-2024 19:07 - defs.defs - ### Nifty PE at 23.98 ### INFO: 20-09-2024 19:07 - defs.defs - Index sync complete. INFO: 20-09-2024 19:07 - defs.defs - Making adjustments for splits and bonus CRITICAL: 20-09-2024 19:07 - root - Adjustment Error - Context PHOENIXLTD - Bonus - 20-Sep-2024 ERROR: 20-09-2024 19:07 - main - Error while making adjustments. All adjustments have been discarded. Traceback (most recent call last): File "/Users/saurabhgarg/shivshakti/eod2/src/init.py", line 125, in defs.adjustNseStocks() File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 833, in adjustNseStocks raise e File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 813, in adjustNseStocks commit = makeAdjustment( ^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 656, in makeAdjustment df = pd.read_csv(file, index_col="Date", parse_dates=["Date"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv return _read(filepath_or_buffer, kwds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 626, in _read return parser.read(nrows) ^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1923, in read ) = self._engine.read( # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 234, in read chunks = self._reader.read_low_memory(nrows) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "parsers.pyx", line 838, in pandas._libs.parsers.TextReader.read_low_memory File "parsers.pyx", line 905, in pandas._libs.parsers.TextReader._read_rows File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status File "parsers.pyx", line 2061, in pandas._libs.parsers.raise_parser_error pandas.errors.ParserError: Error tokenizing data. C error: Expected 9 fields in line 4312, saw 17 INFO: 20-09-2024 19:07 - defs.defs - Rolling back changes from 2024-09-20: /Users/saurabhgarg/shivshakti/eod2/src/eod2_data/daily INFO: 20-09-2024 19:07 - defs.defs - Rollback successful Thanks Saurabh -- Reply to this email directly or view it on GitHub: #184 You are receiving this because you are subscribed to this thread. Message ID: @.***>
Regards, Benny Thadikaran
.venv) saurabhgarg@MacBook-Pro daily % tail -10 phoenixltd.csv
2024-09-06,1827.45,1895.95,1795.6,1803.4,738915,77559.0,9.53,375385.0
2024-09-09,1817.6,1837.2,1731.3,1748.9,247675,37562.0,6.59,152866.0
2024-09-10,1762.45,1806.0,1762.0,1782.9,196083,40000.0,4.9,98060.0
2024-09-11,1793.5,1804.5,1715.0,1719.85,191423,28819.0,6.64,122336.0
2024-09-12,1735.0,1746.45,1672.0,1682.1,238262,29898.0,7.97,153350.0
2024-09-13,1697.0,1699.9,1645.2,1671.4,543217,60139.0,9.03,352718.0
2024-09-16,1689.45,1775.0,1647.8,1766.1,358166,42577.0,8.41,168362.0
2024-09-17,1777.45,1789.7,1739.6,1746.35,251446,39562.0,6.36,126985.0
2024-09-18,1759.9,1782.5,1680.6,1722.85,228944,40642.0,5.63,109858.0
2024-09-19,1745.0,1777.5,1686.85,1765.9,263004,49066.0,5.36,122032.02024-09-20,1820.0,1950.0,1722.05,1844.05,1296237,110874,11.69,5117482024-09-20,1820.0,1950.0,1722.05,1844.05,1296237,110874,11.69,511748
(.venv) saurabhgarg@MacBook-Pro daily %
You are correct the issue is in this file.
Thanks
Saurabh
phoenixltd.csv
Attached is the file.
Also there was issue on 18 also. Uploaing error.log for your review.
Thanks
Saurabh
Benny,
I checked other files also (HDFCBANK/Reliance), seems all are corrupt and need to remove 18 and 19th data and resync it.
Thanks
Saurabh
.venv) saurabhgarg@MacBook-Pro defs % python3 diagnostic.py
File or Pandas exceptions
SETFNN50.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 2264, saw 33\n')
INTLCONV.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 748, saw 33\n')
DIAMONDYD.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 1722, saw 33\n')
AEROFLEX.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 263, saw 33\n')
BHARTIARTL.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 4524, saw 33\n')
SBIETFPB.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 970, saw 33\n')
INFINIUM_SME.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 353, saw 33\n')
BAWEJA_SME.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 156, saw 33\n')
HDFCNEXT50.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 523, saw 33\n')
PRIVISCL.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 2320, saw 33\n')
GABRIEL.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 4814, saw 33\n')
AVROIND.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 853, saw 33\n')
3PLAND.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 3255, saw 33\n')
ZENTEC.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 2345, saw 33\n')
TATASTEEL.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 4690, saw 33\n')
SENCO.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 296, saw 33\n')
OMFURN_SME.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 571, saw 33\n')
BLBLIMITED.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 5436, saw 33\n')
RELAXO.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 3280, saw 33\n')
LIQUIDADD.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 117, saw 33\n')
CELLO.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 219, saw 33\n')
HDFCVALUE.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 490, saw 33\n')
INFOMEDIA.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 3483, saw 33\n')
LUPIN.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 5722, saw 33\n')
SHARIABEES.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 3498, saw 33\n')
JITFINFRA.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 1867, saw 33\n')
BFINVEST.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 3372, saw 33\n')
NIFTY PHARMA.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 2191, saw 33\n')
NIFTY MIDSMALLCAP 400.CSV: ParserError('Error tokenizing data. C error: Expected 9 fields in line 2095, saw 33\n')
(.venv) saurabhgarg@MacBook-Pro defs %
I have fixed a KeyError when verifying adjustment for bonus in UEL share. It was temporarily suspended, so today's data was not available. So trying to access today's Close results in the error.
Going forward this will simply throw a warning to the user like below. It means you need to check the charts to see if the adjustment is correct.
WARNING: 20-09-2024 23:40 - defs.defs - Unable to verify adjustment on UEL - Please confirm manually.
Now this error you mentioned - I feel like I've already dealt with this issue before. Can you confirm the EOD2 you're using with py init.py -v
Also if you open src/defs/defs.py
and search for def updateNseSymbol
. Scroll down a few lines below and you should see the below line. Are you missing the \n
at the end? That has to be the cause of the issue.
text += bytes(
f"{dates.pandasDt},{open},{high},{low},{close},{volume},{trdCnt},{avgTrdCnt},{dq}\n",
encoding="utf-8",
)
If you cant find this, just rename defs.py
to defs.txt
and drop the file to me here.
Just make sure you have the latest version by running git pull
in the EOD2 folder.
To resolve the corrupted data. Open eod2_data folder in your terminal and run the following commands.
git checkout main
git reset --hard HEAD
git clean -f
Now run init.py
and let me know if the issue persists.
I will push the other update tomorrow morning.
Now this error you mentioned - I feel like I've already dealt with this issue before. Can you confirm the EOD2 you're using with
py init.py -v
Also if you open
src/defs/defs.py
and search fordef updateNseSymbol
. Scroll down a few lines below and you should see the below line. Are you missing the\n
at the end? That has to be the cause of the issue.text += bytes( f"{dates.pandasDt},{open},{high},{low},{close},{volume},{trdCnt},{avgTrdCnt},{dq}\n", encoding="utf-8", )If you cant find this, just rename
defs.py
todefs.txt
and drop the file to me here.Just make sure you have the latest version by running
git pull
in the EOD2 folder.To resolve the corrupted data. Open eod2_data folder in your terminal and run the following commands.
git checkout main
git reset --hard HEAD
git clean -f
Now run
init.py
and let me know if the issue persists.I will push the other update tomorrow morning.
Attached is the file. This code is there in the file I have.
I keep updated the repo once a week.
Thanks
Saurabh
Benny,
I cloned the repo again and i see the same issue on 19 Sep file.
Its not putting "/n" at the end of the file.
(.venv) saurabhgarg@MacBook-Pro eod2_data % sh test.sh daily/reliance.csv
No newline at the end of the file.
If I remove the 19th entry it has a new line.
(.venv) saurabhgarg@MacBook-Pro eod2_data % sh test.sh daily/reliance.csv
The file ends with a newline.
(.venv) saurabhgarg@MacBook-Pro eod2_data %
Thanks
Saurabh
I have gone through your error logs and the CSV file you gave me. From the logs, this error originated on 18th July 2024 (below is the relevant bits)
2024-07-18 20:12:30,872 - __main__ - ERROR - Error while making adjustments.
TypeError: cannot do positional indexing on DatetimeIndex with these indexers [slice(737, 739, None)] of type slice
2024-07-18 20:16:02,170 - __main__ - ERROR - Error while making adjustments.
pandas.errors.ParserError: Error tokenizing data. C error: Expected 9 fields in line 739, saw 17
The slice indicates duplicate indexes (dates) in one of the CSV files. It can happen, if an error occurred and the rollback itself failed ( So the row was never removed). Trying to sync again will result in the same row being added again.
I am unable to reproduce this error. I cloned the repo from scratch and ran init.py
. Only error was the latest KeyError. Your error is likely originating from an older version of EOD2.
Here is what i will do:
- KeyError issue is fixed.
- Add additional try - catch blocks and with more logging context like the current EOD2 version, sync date, symbol name etc. This should make catching and fixing errors faster and easier.
- Handle the TimeoutError with a warning log asking the user to try again. It is just a network error, no need for an error trace.
I will post the update soon and let you know.
Update 7.0.3 has been released. My suggestion to delete your existing local repo and perform a fresh install of EOD2. (Backup your user.json and data folder if you have custom config or line data)
Below you can see I have added the EOD2 and NSE version and last update date before file execution.
WARNING: 2024-09-21 17:02:16,900 - defs.defs - Unable to verify adjustment on UEL - Please confirm manually. - 2024-09-20 00:00:00+05:30 - EOD2 v7.0.2 - NSE v1.0.0 - 2024-09-18T00:00:00+05:30
I cloned the repo and made a successful sync
INFO: 21-09-2024 17:02 - __main__ - Downloading Files
INFO: 21-09-2024 17:02 - defs.defs - Starting Data Sync
INFO: 21-09-2024 17:02 - defs.defs - EOD sync complete
INFO: 21-09-2024 17:02 - defs.defs - ### Nifty PE at 23.64 ###
INFO: 21-09-2024 17:02 - defs.defs - Index sync complete.
INFO: 21-09-2024 17:02 - defs.defs - Making adjustments for splits and bonus
INFO: 21-09-2024 17:02 - defs.defs - NDL: face value split (sub-division) - from rs 10/- per share to re 1/- per share
INFO: 21-09-2024 17:02 - defs.defs - SAKSOFT: bonus 1:4
INFO: 21-09-2024 17:02 - __main__ - 19 Sep 2024: Done
----------------------------------------------------
INFO: 21-09-2024 17:02 - defs.defs - Updating NSE EQUITY actions
INFO: 21-09-2024 17:02 - defs.defs - Updating NSE SME actions
INFO: 21-09-2024 17:02 - __main__ - Downloading Files
INFO: 21-09-2024 17:02 - defs.defs - Starting Data Sync
INFO: 21-09-2024 17:02 - defs.defs - EOD sync complete
INFO: 21-09-2024 17:02 - defs.defs - ### Nifty PE at 23.98 ###
INFO: 21-09-2024 17:02 - defs.defs - Index sync complete.
INFO: 21-09-2024 17:02 - defs.defs - Making adjustments for splits and bonus
INFO: 21-09-2024 17:02 - defs.defs - UEL: bonus 1:4
INFO: 21-09-2024 17:02 - defs.defs - RITES: bonus 1:1
INFO: 21-09-2024 17:02 - defs.defs - PHOENIXLTD: bonus 1:1
INFO: 21-09-2024 17:02 - defs.defs - MINDTECK: bonus 1:4
INFO: 21-09-2024 17:02 - defs.defs - AXITA: bonus 1:3
WARNING: 21-09-2024 17:02 - defs.defs - Unable to verify adjustment on UEL - Please confirm manually. - 2024-09-20 00:00:00+05:30
INFO: 21-09-2024 17:02 - defs.defs - SOLEX_sme: bonus 1:4
INFO: 21-09-2024 17:02 - __main__ - 20 Sep 2024: Done
----------------------------------------------------
INFO: 21-09-2024 17:02 - defs.defs - All Up To Date. Check again after 07:00PM for today's EOD data
The warning message on UEL
can be ignored for now. I need to check the charts to verify the adjustment was correct.
Issues will Ambaniorgo
and RPPL
have already been fixed and published on eod2_data
.
aye aye ben !
Update 7.0.3 has been released. My suggestion to delete your existing local repo and perform a fresh install of EOD2. (Backup your user.json and data folder if you have custom config or line data)
Below you can see I have added the EOD2 and NSE version and last update date before file execution.
WARNING: 2024-09-21 17:02:16,900 - defs.defs - Unable to verify adjustment on UEL - Please confirm manually. - 2024-09-20 00:00:00+05:30 - EOD2 v7.0.2 - NSE v1.0.0 - 2024-09-18T00:00:00+05:30
I cloned the repo and made a successful sync
INFO: 21-09-2024 17:02 - __main__ - Downloading Files INFO: 21-09-2024 17:02 - defs.defs - Starting Data Sync INFO: 21-09-2024 17:02 - defs.defs - EOD sync complete INFO: 21-09-2024 17:02 - defs.defs - ### Nifty PE at 23.64 ### INFO: 21-09-2024 17:02 - defs.defs - Index sync complete. INFO: 21-09-2024 17:02 - defs.defs - Making adjustments for splits and bonus INFO: 21-09-2024 17:02 - defs.defs - NDL: face value split (sub-division) - from rs 10/- per share to re 1/- per share INFO: 21-09-2024 17:02 - defs.defs - SAKSOFT: bonus 1:4 INFO: 21-09-2024 17:02 - __main__ - 19 Sep 2024: Done ---------------------------------------------------- INFO: 21-09-2024 17:02 - defs.defs - Updating NSE EQUITY actions INFO: 21-09-2024 17:02 - defs.defs - Updating NSE SME actions INFO: 21-09-2024 17:02 - __main__ - Downloading Files INFO: 21-09-2024 17:02 - defs.defs - Starting Data Sync INFO: 21-09-2024 17:02 - defs.defs - EOD sync complete INFO: 21-09-2024 17:02 - defs.defs - ### Nifty PE at 23.98 ### INFO: 21-09-2024 17:02 - defs.defs - Index sync complete. INFO: 21-09-2024 17:02 - defs.defs - Making adjustments for splits and bonus INFO: 21-09-2024 17:02 - defs.defs - UEL: bonus 1:4 INFO: 21-09-2024 17:02 - defs.defs - RITES: bonus 1:1 INFO: 21-09-2024 17:02 - defs.defs - PHOENIXLTD: bonus 1:1 INFO: 21-09-2024 17:02 - defs.defs - MINDTECK: bonus 1:4 INFO: 21-09-2024 17:02 - defs.defs - AXITA: bonus 1:3 WARNING: 21-09-2024 17:02 - defs.defs - Unable to verify adjustment on UEL - Please confirm manually. - 2024-09-20 00:00:00+05:30 INFO: 21-09-2024 17:02 - defs.defs - SOLEX_sme: bonus 1:4 INFO: 21-09-2024 17:02 - __main__ - 20 Sep 2024: Done ---------------------------------------------------- INFO: 21-09-2024 17:02 - defs.defs - All Up To Date. Check again after 07:00PM for today's EOD data
The warning message on
UEL
can be ignored for now. I need to check the charts to verify the adjustment was correct.Issues will
Ambaniorgo
andRPPL
have already been fixed and published oneod2_data
.
Done as instructed. Further processing the same.
Will update.
Thanks
Saurabh
All set Benny. Thank you.
@rohit1409 I suppose this has resolved for you too. Closing this issue for now.
Yup, Thanks Benny.. you are very supportive as ever !