Retrieving EPG data in the background
DiR3Wolf opened this issue · 9 comments
I have found that, while retrieving EPG on Kodi works, and using a service like service.psvue.epg makes it seamless, it also introduces a few quirks:
First, while the EPG.xml is being written, specially is you are retrieving all PS Vue channels (in my case 104 ch.), it seems to take about 2 minutes for it to complete on a beefy machine. It seems to tie up Kodi resources to where there are minor hiccups when navigating through Kodi when the process is running, and it makes Kodi unstable (delays/hangs/crash) if you try to exit while it is writing the epg.xml.
Second, it only retrieves a few hours worth of programming from Sony, making writing a new EPG.xml necessary within a few hours (default is currently 1 hour). So every hour we are potentially experience issues referenced above. This also makes it necessary for the PVR to refresh the guide in a shorter interval, preferably at the same interval of the EPG process. This is mostly not a problem if Kodi is running uninterrupted 24/7.
However, is one quits Kodi, and resumes Kodi after a few hours, then EPG data is missing, especially if overnight. To counter this, one have to start Kodi, wait for EPG.xml to be fully written, exit, and restart Kodi to get a refreshed guide.
So, my solution, is to have the EPG.xml to run as an option. service.psvue.epg would still create the playlist file on startup. As an alternative, with EPG.xml turned off, I can create an xml file to run through webgrab, so it can pull more hours/days of EPG data to match those of the playlist. webgrab will only run once a day, and EPG data will always be available on Kodi startup.
I agree, there is much room for improvement with the epg building. I'd like to improve load times and amount of show info by using a different call or even get the info from a different service than ps vue, but I haven't done much research on this yet. I'll create a setting to turn this off so you may supply your own epg.xml file.
I quickly glanced at the changes just now, but I haven't had time to go into Kodi and test it. Probably tomorrow. A few things I have thought about re: last_update variable. Since it runs in memory, I thought it would be great to write/get last_update from settings so it doesn't necessarily runs at Kodi startup (especially when testing stuff, I have to comment "check_files()" so I can do other stuff).
Anyway, I thought about using time.time() instead of datetime.now(). I will test it and if successful, will submit a pull request tomorrow, unless you beat me to it or don't want this at all.
Also, I don't know if there is something "better" as far as plug n play guide. Zap2it, for example, don't displays all PSVue channels (HBO and Cinemax are missing). I have used directv with webgrab++, but some channels are off (for example, ABC is on time, but Bravo is off by an hour), so you have to manually edit the config files, which I plan to make and share, but would be for people who decides to do background EPG grab.
Thanks for all these changes, it is turning out awesome!!!
So I add my post code to eracknaphobia's xmltv write definition. You can get way more data than just a few hours, but at the cost of no plot and genres. The POST json doesn't have that data but it does have the program ids which can be used to get the plot and genre. Small issue I had was trying to save the plot and genre info after using the program id to call a new json. I believe it's a json dictionary and not json list. Bigger problem is time. Calling the running the post_json to get the program, then call the get_json to get the information slows down the loading process a lot. If you two wanna look at what I got so far, I made a subrelease from my fork.
https://github.com/snoopyjoe/service.psvue.epg/releases/download/2018.4.7.2/mainservice.py
After a very quick test, I have found this code to be https://github.com/edit4ever/script.module.zap2epg to be super fast if only one day is pulled. 2 days slow things down exponentially, but not as bad considering the amount of EPG data being retrieved compared to Sony's website of only a few hours.
I'm planning on adapting this code for PSVue, but since I am not as technically gifted as @eracknaphobia or @snoopyjoe , it may take me a while, so if either of you want to jump on it, go for it. Just a suggestion is you indeed go with this, choose ATT Uverse provider instead of Sony PSVue. I have found Sony's listing to be missing channels (like local and premium channels.) I was using DirecTv with Webgrab before, but that one is missing Epix hits, and some channels are off relative to rest of the guide. For example, you adjust ABC along with most channels to match your local time, but a few channels like Bravo, E! are off by an hour. Uverse so far has been the best, but requires channel matching for PS Vue lineup.
Also, I'm thinking maybe it'd be a good idea to split the "improved" EPG service and PVR play module as a required module, like @eracknaphobia initially conceptualized it. That way, the cherrypy service doesn't have to be alive all the time, just called on play, correct? That may help with crashing/hanging issues, or maybe not, just a thought.
@d21spike and I are going to try and leverage this epg call and see if we can get better results. Upside is the call allows you to set start and end dates for epg info, downside no art work or genre.
BTW, the wip branch has dropped cherrypy in favor of BaseHTTPServer. It currently only works on the latest nightlies of kodi 18, but the channel changing seems much quicker than the initial 18 alpha (March 4).
I kind of have the same thing using post to the "/airings" url here. It's just messier than yours. Not having the synopsis is probably the biggest downside to this call. My only solution is to call another url using the program_id and get synopsis and genre, but that increases the download time significantly.
Would it be viable to have both BaseHTTPServer and Cherrypy in the code but only load one based on Kodi build? At least until Kodi v17 is no longer supported(which may be a while).
@snoopyjoe ok, cool. tried your initial link and got 404, but I figured you were using the same call by the sound of it.
BaseHTTPServer will replace cherrypy all together, I just need to add support for playback in 17 before it will work, by just doing the version check and play as listitem on 17 instead of directly handing off the stream to the pvr client that only works in 18.
Closing issue. Original comments were too broad in scope, some fixes already implemented and third party background service not direction the addon is taking.