Missing 2017 and 2018 schedule data

Question

Missing 2017 and 2018 schedule data

Couragyn opened this issue 7 years ago · 17 comments

I am missing schedule dates from 2017 up. I have seen this issue elsewhere, but the solutions suggested are not working for me as these files are not in scripts:

update_sched.py --year 2017
nflgame-update-schedule --year 2017

nfldb-update has a '--update-schedules' flag, but it does not add in the missing years.

Any ideas?

Answer 1 · 2018-07-09T19:17:47.000Z

Did you find a solution for this? When i update i am not getting any 2017 data?

Answer 2 · 2018-07-09T19:50:02.000Z

I have not yet found a solution

Answer 3 · 2018-07-09T20:24:45.000Z

I believe my fork of nflgame has all of the games played in the regular and postseason of 2017 included.
https://github.com/ochawkeye/nflgame

No schedule has yet been posted to nfl.com for the 2018 season
http://www.nfl.com/liveupdate/scorestrip/ss.xml

Answer 4 · 2018-07-09T22:00:35.000Z

ochawkeye...how do i implement your fork? I am use to just running the nfldb-update script and the most recent data is pulled. This looks like something different?

Answer 5 · 2018-07-09T22:13:55.000Z

If you know of a solution to keep nfldb-udpate working, that would be ideal. I cant seem to figure out why when i ran it last year i was getting 2017 data and now I only get 2016 data. I did have to reinstall everything on a new machine so wondering if i just did that piece wrong...but doesnt seem like it

Answer 6 · 2018-07-09T23:03:06.000Z

Wellllll....I'm guessing that you're still running the original install from PyPI that hasn't been updated in ages.

If you are, then you can go one of two routes...

You can uninstall and re-install using instead the (also un-maintained github repository gulp) via #296 instructions.
You can just try copying and pasting my schedule.json into your /nflgame folder and try running your same nfldb-update script. That might succeed in downloading your missing games.

Answer 7 · 2018-07-09T23:11:48.000Z

Doesnt look like the schedule.json solution worked. This is what i get when running nfldb if that might help? As you can tell i am not a developer :)

STARTING NFLDB UPDATE AT 2018-07-09 16:10:38.037000
Connecting to nfldb... done.
Setting timezone to UTC... done.
Locking write access to tables... done.
Updating season phase, year and week... done.
Adding schedule data for 16 games... done.
Bulk inserting data for 273 games...
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Traceback (most recent call last):
File "nfldb-update", line 39, in
nfldb.update.run(**vars(args))
File "C:\Python27\lib\site-packages\nfldb\update.py", line 535, in run
doit()
File "C:\Python27\lib\site-packages\nfldb\update.py", line 525, in doit
update_games(db, batch_size=batch_size)
File "C:\Python27\lib\site-packages\nfldb\update.py", line 397, in update_games
bulk_insert_game_data(cursor, scheduled, batch_size=batch_size)
File "C:\Python27\lib\site-packages\nfldb\update.py", line 195, in bulk_insert_game_data
do()
File "C:\Python27\lib\site-packages\nfldb\update.py", line 188, in do
nfldb.db._big_insert(cursor, table, bulk[table])
File "C:\Python27\lib\site-packages\nfldb\db.py", line 356, in _big_insert
% (table, insert_fields, values))
File "C:\Python27\lib\site-packages\psycopg2\extras.py", line 249, in execute
return super(RealDictCursor, self).execute(query, vars)
psycopg2.IntegrityError: insert or update on table "play" violates foreign key constraint "play_pos_team_fkey"
DETAIL: Key (pos_team)=(JAX) is not present in table "team".

Answer 8 · 2018-07-09T23:15:59.000Z

With the second solution i get the following response. I didn't think git was something that needed to be installed...at least this isnt described in any of the tutorials i have gone through

C:\Users\matth>pip install git+https://github.com/BurntSushi/nflgame.git
Collecting git+https://github.com/BurntSushi/nflgame.git
Cloning https://github.com/BurntSushi/nflgame.git to c:\users\matth\appdata\local\temp\pip-req-build-swe2sb
Error [Error 2] The system cannot find the file specified while executing command git clone -q https://github.com/BurntSushi/nflgame.git c:\users\matth\appdata\local\temp\pip-req-build-swe2sb
Cannot find command 'git' - do you have 'git' installed and in your PATH?

Answer 9 · 2018-07-09T23:20:12.000Z

The JAX not being present in the table is not related to the schedule not updating.

See #252 for related discussion on that.

For your second issue...Yes, if you aren't able to git clone then you won't be able to pip install from github directly.

Answer 10 · 2018-07-09T23:31:19.000Z

Interesting so it looks like it added a ton more games...but i am still not getting 2017?

STARTING NFLDB UPDATE AT 2018-07-09 16:29:28.216000
Connecting to nfldb... done.
Setting timezone to UTC... done.
Locking write access to tables... done.
Updating season phase, year and week... done.
Adding schedule data for 339 games... done.
Bulk inserting data for 596 games...
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Sending batch of data to database.
Traceback (most recent call last):
File "nfldb-update", line 39, in
nfldb.update.run(**vars(args))
File "C:\Python27\lib\site-packages\nfldb\update.py", line 535, in run
doit()
File "C:\Python27\lib\site-packages\nfldb\update.py", line 525, in doit
update_games(db, batch_size=batch_size)
File "C:\Python27\lib\site-packages\nfldb\update.py", line 397, in update_games
bulk_insert_game_data(cursor, scheduled, batch_size=batch_size)
File "C:\Python27\lib\site-packages\nfldb\update.py", line 195, in bulk_insert_game_data
do()
File "C:\Python27\lib\site-packages\nfldb\update.py", line 188, in do
nfldb.db._big_insert(cursor, table, bulk[table])
File "C:\Python27\lib\site-packages\nfldb\db.py", line 356, in _big_insert
% (table, insert_fields, values))
File "C:\Python27\lib\site-packages\psycopg2\extras.py", line 249, in execute
return super(RealDictCursor, self).execute(query, vars)
psycopg2.IntegrityError: insert or update on table "play" violates foreign key constraint "play_pos_team_fkey"
DETAIL: Key (pos_team)=(JAX) is not present in table "team".

Answer 11 · 2018-07-09T23:53:37.000Z

This was after installing git and then updating pip install git+https://github.com/BurntSushi/nflgame.git

I feel like i am close but something is still missing

Answer 12 · 2018-07-10T03:52:22.000Z

ochawkeye...would you be interested in working on a custom solution for me? Can work out payment arrangements of some sort. Let me know.

Thanks

Answer 13 · 2018-07-10T13:45:16.000Z

I'm not interested in taking payment, no, but I'm happy to assist in getting you going. Would be good to see what a semi-clean install looks like so I could take some notes to maybe get something documented to assist others in the future.

If you're comfortable with me temporarily accessing your system, run TeamViewer QuickStart and contact me at benjaminDOTjDOTschroederATgmailDOTcom.

I believe your schedule & games are now updating properly but you've still got an issue with the JAC/JAX changeover that happened a couple of years ago since your traceback states DETAIL: Key (pos_team)=(JAX) is not present in table "team".

Answer 14 · 2018-07-20T03:43:33.000Z

Funny, I had preseason data in there from when I ran it last year in late august. Today, when i first went back in and reran the nfldb-update, I got week 17 but that was it.

I then ran nfdb-update --update-schedules. Took some time but didnt do anything!

Anyone have any other ideas?

Answer 15 · 2018-07-20T06:09:24.000Z

So I ran python update_sched.py --year 2017 from the nflgame directory. This updated the schedule.json file with the appropriate games. Then i ran nfldb-update and the games updated:

Answer 16 · 2018-07-26T18:40:53.000Z

Any updates on getting the 2018 players roster?
When I run:
import nfldb
db = nfldb.connect()
q = nfldb.Query(db)
q.game(season_year=2017, season_type='Regular')
for pp in q.as_aggregate():
if str(pp.player.position) in ['QB', 'RB', 'WR', 'TE', 'K']:
print pp.player

I get all the players coming in. But when I run it with 2018, 'Regular' or 'Preseason'
Nothing returns...

-Front-End developer lost in the back-end world!

Answer 17 · 2018-07-31T05:44:05.000Z

There's not anything for you to get yet as far as a player roster. The query you performed above is aggregating statistics compiled over the course of the 2017 regular season. There haven't been any plays yet run in the 2018 regular or preseason.

I succeeded in updating my schedule today and can see the games with nfldb but there are no players that have aggregated any stats yet for the season like you're looking for.

import nfldb

db = nfldb.connect()
q = nfldb.Query(db)
q.game(season_year=2018, season_type='Regular', week=1)

for game in q.as_games():
    print game

print 'v'*25
for pp in q.as_aggregate():
    print pp.player
print '^'*25

Regular 2018 week 1 on 09/10 at 09:20PM, LA (0) at OAK (0)
Regular 2018 week 1 on 09/10 at 06:10PM, NYJ (0) at DET (0)
Regular 2018 week 1 on 09/09 at 07:20PM, CHI (0) at GB (0)
Regular 2018 week 1 on 09/09 at 03:25PM, SEA (0) at DEN (0)
Regular 2018 week 1 on 09/09 at 03:25PM, DAL (0) at CAR (0)
Regular 2018 week 1 on 09/09 at 03:25PM, WAS (0) at ARI (0)
Regular 2018 week 1 on 09/09 at 03:05PM, KC (0) at UNK (0)
Regular 2018 week 1 on 09/09 at 12:00PM, JAC (0) at NYG (0)
Regular 2018 week 1 on 09/09 at 12:00PM, TB (0) at NO (0)
Regular 2018 week 1 on 09/09 at 12:00PM, HOU (0) at NE (0)
Regular 2018 week 1 on 09/09 at 12:00PM, SF (0) at MIN (0)
Regular 2018 week 1 on 09/09 at 12:00PM, TEN (0) at MIA (0)
Regular 2018 week 1 on 09/09 at 12:00PM, CIN (0) at IND (0)
Regular 2018 week 1 on 09/09 at 12:00PM, PIT (0) at CLE (0)
Regular 2018 week 1 on 09/09 at 12:00PM, BUF (0) at BAL (0)
Regular 2018 week 1 on 09/06 at 07:20PM, ATL (0) at PHI (0)
vvvvvvvvvvvvvvvvvvvvvvvvv
^^^^^^^^^^^^^^^^^^^^^^^^^