init_oca
Closed this issue · 10 comments
So having finally got some races to organise, I need once again to be able to pre-load riders into the database. Under python 2.7 I was using the init_oca process. I think it's been a victim of the python 3 upgrade as it no longer works. I've ploughed though the str/string issue, the optione/options one, but I'm still hitting more issues with the move to utf8. The load is now failing as follows:
command:
python3 /RaceDB/manage.py init_oca /racedb-data/lcca_2020_Race1_oca_entries_20201215190657.csv
error:
RaceDBVersion=v3.0.35-private databaseEngine="django.db.backends.postgresql" databaseName="racedb" ServerTimeZone="Europe/London" python="3.7.9 (default, Nov 18 2020, 05:33:08) [GCC 8.3.0]" logFileName="/racedb-data/RaceDB-log.txt" fix_bad_license_codes and emergency contacts... Traceback (most recent call last): File "/RaceDB/manage.py", line 38, in <module> execute_from_command_line(sys.argv) File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line utility.execute() File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv self.execute(*args, **cmd_options) File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute output = self.handle(*args, **options) File "/RaceDB/core/management/commands/init_oca.py", line 16, in handle core.init_oca.init_oca( options['fname'] ) File "/RaceDB/core/init_oca.py", line 143, in init_oca fields = utils.getHeaderFields( [unescape(v.strip()) for v in row] ) File "/RaceDB/core/utils.py", line 133, in getHeaderFields fields = [v.replace('-','_').replace('#','').strip().replace('4', 'four').replace(' ','_') for v in fields] File "/RaceDB/core/utils.py", line 133, in <listcomp> fields = [v.replace('-','_').replace('#','').strip().replace('4', 'four').replace(' ','_') for v in fields] TypeError: a bytes-like object is required, not 'str'
Thanks
Chris
...and as I carried on hacking..
utils.py, line 133 I changed to this and it appeared to now work:
fields = [v.decode('utf-8').replace('-','_').replace('#','').strip().replace('4', 'four').replace(' ','_') for v in fields]
Not sure if that'r really the 'right' answer but it seems to work for me at least
If you are so inclined, submit a merge request. That mean, fork the code, make your changes, make it work, and send a merge request back. We can then accept it, and it will be done for everyone.
(of course, that also means any further hacking can be submitted in the same fashion)
Just make sure if you do all that you are working on the "dev" branch...merge request must be submitted against the dev branch otherwise things will get messed up.
ok will do. I'm currently building a new image locally - it takes the best part of an hour which seems a bit crazy (building for ARM32V7). I suspect I'm possibly alone in using this architecture though so not sure if worth adding a pipeline for it.
It's just python. The platform shouldn't be relevant. CrossMgr might pose an issue if the GUI lib isn't available, but RaceDB should work fine. I would suggesting looking at any of the OrangePI boards....they run much faster than any of the RPi ones....RPi 4 included.
ARM could be a platform Github supports for Linux. Check into that. It wouldn't be hard to create another platform for RaceDB container (or for a build on docker hub directly).
Using the standard build I was getting platform related errors - it was only a one line change in the Dockerfile and works fine otherwise.
If you can figure out how to make the same "code" build on both (all?) platforms, submit a merge request with those changes too. It's good to spread the knowledge around.
Never rains but it pours...
So after (now successfully) loading the oca format file, I load the event entries which are xlsx. It seems that since 11th December, xlrd no longer supports xlsx files:
https://groups.google.com/g/python-excel/c/IRa8IWq_4zk/m/Af8-hrRnAgAJ
So I need to either pin versions or change my file to xls
Yes, xlrd dropping .xlsx format caught me off-guard.
Regardless, I converted all of RaceDB to use a new module, openpyxl, which supports .xlsx files (and doesn't support .xls files).
So, RaceDB only support .xlsx files going forward. This should be OK as Microsoft is dropping support for .xls files too.
Let me know if everything is working correctly.
Presumed fixed. Closing.