sqlite3 error database disk image is malformed (conda on Ubuntu)

Question

sqlite3 error database disk image is malformed (conda on Ubuntu)

fol-debug opened this issue 5 years ago · 21 comments

Hi,

I'm getting a lot of errors on the database itself while running run_airsenal_predictions. Right now it says "Filling history dataframe for xxx: 218/218" so it seems that it maybe isn't getting all of the players. Also, running the optimization, it shows errors because of this when running strategies. So right now it only runs on one thread; dropped out everything else.

The prediction-table seems correctly filled out to me so I'm not really sure where to start looking for the faults.

One of the errors also says "file is not a database".

EDIT: I see now that the fixture-table is empty on player_id; maybe this is where the error stems from?

EDIT: Running a PRAGMA integrity_check turns out OK

EDIT: This is on the fix/unique_violation_fix branch.

Answer 1 · 2019-08-27T12:04:36.000Z

With the fix/unique_violation_fix branch installed, and ensuring you have FPL_TEAM_ID set with your team ID either as an environment variable or in the file airsenal/data/FPL_TEAM_ID, could you run each of the following steps in order and copy any errors you get into a comment here?

> rm /tmp/data.db
> setup_airsenal_database
> update_airsenal_database
> run_airsenal_predictions --weeks_ahead 3
> run_airsenal_optimization --weeks_ahead 3

Answer 2 · 2019-08-27T12:09:51.000Z

Yep, will do tomorrow morning. tir. 27. aug. 2019, 14:04 skrev Jack Roberts <notifications@github.com>:

…

With the fix/unique_violation_fix branch installed, and ensuring you have FPL_TEAM_ID set with your team ID either as an environment variable or in the file airsenal/data/FPL_TEAM_ID, could you run each of the following steps and copy any errors you get into a comment here? > rm /tmp/data.db > setup_airsenal_database > update_airsenal_database > run_airsenal_predictions --weeks_ahead 3 > run_airsenal_optimization --weeks_ahead 3 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#81?email_source=notifications&email_token=AM7YG5T4FW323NJHVYJARHLQGUJ5LA5CNFSM4IP6J6T2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5HP3HA#issuecomment-525270428>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AM7YG5SGU6KQIDNVKOAULBLQGUJ5LANCNFSM4IP6J6TQ> .

Answer 3 · 2019-08-28T07:42:01.000Z

Heres the beginning of run_airsenal_predictions --weeks_ahead 3

https://pastebin.com/5d5TS7sW

Answer 4 · 2019-08-28T08:10:18.000Z

run_airsenal_optimization --weeks_ahead 3
https://pastebin.com/p1Gyxy4U

Answer 5 · 2019-08-28T08:14:57.000Z

Ok thanks I'll have a look over at some point. Just to confirm - the first 3 steps rm /tmp/data.db; setup_airsenal_database; update_airsenal_database don't give you any errors?

Answer 6 · 2019-08-28T08:17:06.000Z

No, those are all good. I did however get an error while setting up the enviroment, building the bpl.wheel, but it continued and showed no further errors.

Answer 7 · 2019-08-28T08:27:55.000Z

Heres the fully run optimization:
https://pastebin.com/m4ReUdFQ

Answer 8 · 2019-08-28T09:43:15.000Z

No, those are all good. I did however get an error while setting up the enviroment, building the bpl.wheel, but it continued and showed no further errors.

This might be the problem. There are differences between how bpl is installed on master and the fix/unique_violation_fix branch at the moment, let's see whether you still have problems once the two have been merged (hopefully later this week).

Answer 9 · 2019-08-28T09:44:06.000Z

Hmm. Could I maybe set up the database with the fix, and then use the same database for the master, meanwhile?

Answer 10 · 2019-08-28T10:00:58.000Z

If you're keen to get something working you can try all this (install from master but run off fix branch):

Clean up previous environment:

> rm /tmp/data.db
> rm -r <PATH_TO_AIRSENAL_DIRECTORY>
> conda deactivate
> conda env remove -n airsenalenv

Install AIrsenal to new environment (note -e argument to pip install):

> conda create -n airsenalenv python=3.7
> conda activate airsenalenv
> conda install -c psi4 gcc-5
> git clone https://github.com/alan-turing-institute/AIrsenal.git
> cd AIrsenal
> pip install -e .

If that doesn't give you build errors due to bpl then run:

> git checkout remotes/origin/fix/unique_playerid_violation
> setup_airsenal_database
> update_airsenal_database
> run_airsenal_predictions --weeks_ahead 3
> run_airsenal_optimization --weeks_ahead 3

Answer 11 · 2019-08-28T10:37:09.000Z

Still gives me an error when 'run_airsenal_predictions --weeks_ahead 3'
http://pastebin.com/XvT2ikKA

EDIT: Thanks a lot for helping out, by the way. I really appreciate it.

Answer 12 · 2019-08-28T16:03:43.000Z

Did bpl install successfully with no build errors with those instructions?

Answer 13 · 2019-08-28T17:00:27.000Z

Yes.

I'm starting to think maybe my setup is wrong.

I've tried both Ubuntu and Debian on both VirtualBox and Hyper-V. Do you have any other suggestions on what I can try?

I've also had zero success on Windows; can't seem to find a compatible compiler.

Answer 14 · 2019-08-29T13:25:23.000Z

It's mostly been developed on Macs and we have already seen some issues on Linux relating to the pystan package, e.g. #66

Answer 15 · 2019-08-29T15:55:42.000Z

Ahhh. I see. That would explain the issues I have. I'll see if I can find a workaround :)

Answer 16 · 2019-08-30T15:15:23.000Z

It worked immediately when trying to install it on a MacBook. Thanks!

Answer 17 · 2020-07-10T13:34:52.000Z

Reopening this issue as I had the same problem with an Ubuntu 18.04 Azure VM. Also tried in an Ubuntu docker container on my Mac, which ran fine. Not sure what causes it, maybe related to trying to access the database from multiple processes.

setup_airsenal_database and update_run_airsenal_database run fine (and give a database that seems correct), but run_airsenal_predictions gives the error.

Answer 18 · 2020-07-12T21:06:19.000Z

On the same Azure VM doing the following led to a working installation:

Install gcc, sqlite3 and pip3 with apt.
Install airsenal to the system python - not in a conda environment.

So might be a conda issue, possibly something r.e. conda and sqlite and/or gcc.

Answer 19 · 2020-09-09T22:41:15.000Z

On the same Azure VM doing the following led to a working installation:

Install gcc, sqlite3 and pip3 with apt.

Install airsenal to the system python - not in a conda environment.

So might be a conda issue, possibly something r.e. conda and sqlite and/or gcc.

These steps worked to fix this on Ubuntu18.04 in windows subsytem for Linux too

Answer 20 · 2020-09-10T14:17:56.000Z

Thanks for letting us know @Tdarnell - we should try to track down what's causing this.

Answer 21 · 2020-09-16T12:29:24.000Z

Bit of searching around suggests this could be caused by a mismatch between sqlite3 versions, e.g. between the system version and the version used by python (in the conda env). Or related to managing connections across threads with multiprocessing/sqlalchemy/sqlite3 - that may make more sense as the (single thread) database setup and update work, but the (multithreaded) prediction fails.