BennyThadikaran/eod2

Error While Syncing Data

cotton1234 opened this issue · 4 comments

Hello,

The program was running file till now but today (06/03/2023) the issue is coming in cleanup of the files.

in
Downloading Files
Starting Data Sync
EOD sync complete
Delivery Data updated
Delivery sync complete
Index sync complete.
Makings adjustments for splits and bonus
Cleaning up files
Traceback (most recent call last):
File "/Users/saurabhgarg/shivshakti/eod2/init.py", line 51, in
defs.cleanup([bhavFile])
File "/Users/saurabhgarg/shivshakti/eod2/defs.py", line 553, in cleanup
lastUpdated = datetime.strptime(getLastDate(entry.path), fmt)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/_strptime.py", line 349, in _strptime
raise ValueError("time data %r does not match format %r" %
ValueError: time data '' does not match format '%Y-%m-%d'

Since the lastupdate.txt file get updated in the end after cleanup so running the prog will make it double entries in daily and delivery folder.

Thanks
Saurabh

Its seems like your lastupdate.txt is using the wrong date format. Change it to reflect 2023-03-03T00:00:00.

First thing, take a backup of daily and delivery folder.
I have attached a clean up script drop_last.txt. Rename the extension to drop_last.py and move it to the same folder as init.py.

Edit the date in the fourth line to whatever date you need removed. Run the file. It will remove lines with the specific date in both daily and delivery folder.

Check if everything is fine and run your init.py. Let me know if this works for you.

drop_last.txt

Its seems like your lastupdate.txt is using the wrong date format. Change it to reflect 2023-03-03T00:00:00.

First thing, take a backup of daily and delivery folder. I have attached a clean up script drop_last.txt. Rename the extension to drop_last.py and move it to the same folder as init.py.

Edit the date in the fourth line to whatever date you need removed. Run the file. It will remove lines with the specific date in both daily and delivery folder.

Check if everything is fine and run your init.py. Let me know if this works for you.

drop_last.txt

Thanks Benny for quick response.

Running the file is giving the error for cleaning the folders.

saurabhgarg@Saurabhs-MacBook-Pro eod2 % python3 drop_last.py
Traceback (most recent call last):
File "/Users/saurabhgarg/shivshakti/eod2/drop_last.py", line 8, in
df = read_csv(entry.path, index_col='Date',
File "/Users/saurabhgarg/Library/Python/3.9/lib/python/site-packages/pandas/util/_decorators.py", line 211, in wrapper
return func(*args, **kwargs)
File "/Users/saurabhgarg/Library/Python/3.9/lib/python/site-packages/pandas/util/_decorators.py", line 331, in wrapper
return func(*args, **kwargs)
File "/Users/saurabhgarg/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
return _read(filepath_or_buffer, kwds)
File "/Users/saurabhgarg/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/readers.py", line 605, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/Users/saurabhgarg/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/readers.py", line 1442, in init
self._engine = self._make_engine(f, self.engine)
File "/Users/saurabhgarg/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/readers.py", line 1753, in _make_engine
return mapping[engine](f, **self.options)
File "/Users/saurabhgarg/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 157, in init
self._set_noconvert_columns()
File "/Users/saurabhgarg/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 211, in _set_noconvert_columns
noconvert_columns = self._set_noconvert_dtype_columns(
File "/Users/saurabhgarg/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/base_parser.py", line 675, in _set_noconvert_dtype_columns
noconvert_columns.add(_set(k))
File "/Users/saurabhgarg/Library/Python/3.9/lib/python/site-packages/pandas/io/parsers/base_parser.py", line 652, in _set
x = col_indices[names.index(x)]
ValueError: 'Date' is not in list


saurabhgarg@Saurabhs-MacBook-Pro eod2 % cat drop_last.py
from pandas import read_csv
from os import scandir

dt = '2023-03-06'

Thanks
Saurabh

ValueError: 'Date' is not in list -> this is saying your csv file doesn't have a 'Date' column in it.

Can you open any csv file in daily folder and confirm the column names. They should be like - Date | Open | High | Low | Close | Volume

Manually editing the lastupdate.txt leaves a line ending character \n at the end of the date. This will throw an error in getLastUpdated function. My latest scripts strips this line ending character before converting to python datetime.

You're likely using a very old version of my script. If that's the case it might explain your other issue with adjustments. (though i need to confirm that)

I have made several major changes in the script recently including some error fixes in stock adjustments function.
It is far easier to just download the latest repo. The data is fully updated till yesterday.

I'll keep this issue open till Saturday so you can check everything is working fine.

I re-cloned the same and its working fine.

Thanks for your help.

Saurabh