----------------------------------------------------------------------------------- - epgd - - This daemon is used to download EPG data from the internet and manage it in a maria database. - - Written by: C++/SQL - Jörg Wendel (vdr at jwendel dot de) - SQL/Procedures - Christian Kaiser - Documentation - Ingo Prochaska - - Homepage: http://projects.vdr-developer.org/projects/vdr-epg-daemon - Source-code: http://projects.vdr-developer.org/git/vdr-epg-daemon.git - - This software is released under the GPL, version 2 (see COPYING). - Additionally, compiling, linking, and/or using the OpenSSL toolkit in - conjunction with this software is allowed. ----------------------------------------------------------------------------------- Contents: --------- Contents Description EPG Merge Get the source from git Requirements MariaDb Setup epgd Installation epgd Configuration Starting epgd and init-Scripts Upgrade from older versions MariaDb Hints Considerations Description: ------------ epgd is part of the double team epgd+epg2vdr to effectively retrieve, store and import epg data to vdr. It is designed to handle large amount of data and pictures in a distributed environment with one epg-server and many possible vdr-clients - therefore it relies on mariadb. Though it is possible to use epgd alone with mariadb it only makes sense to use it as backend to the vdr-plugin epg2vdr. That being said you need to install, setup and configure mariadb, epgd and epg2vdr in order to get a working environment. EPG Merge: ---------- The DVB and external events will be merged for the next 72 hours, DVB EPG is the main event provider which will be enhanced by the external EPG and series data from constabel.net. If events can be merged, just starttime, duration and vps-flag of the DVB event will be used from external provider. The epgd can mix events with up to 20 minutes difference of the start time depending on title and duration. If DVB and external EPG can't be merged (due to late program changes or sg.), the DVB event will be placed in your EPG to avoid wrong events. Due to the EPG-merge the x-components delivered by DVB are also available. Notice that they can only be handled for DVB and merged events, not for events which are only provided by the external source. The merge behavior is configured in the channlemap.conf, 0 disables the merge for the corresponding channel, 1 enables it and with merge 2 the provider ist hold in background and used as fall-back which is used in case the merge 1 provider delivers poor data. Get the source from git: ------------------------ get the source - probably done, when you reading this locally: git clone git://projects.vdr-developer.org/vdr-epg-daemon.git update the source: cd /to/your/source/epgd git pull throwing away local changes and update to latest source: cd /to/your/source/epgd git checkout -f master git reset --hard git pull Requirements: ------------- - libarchive-dev - libcurl 7.10+ - libxslt 1.1.24+ - libxml2 - libmariadb >= 10.x - libz - libssl-dev - uuid uuid-dev - libjansson4 libjansson-dev - libmicrohttpd libmicrohttpd-dev (epghttpd) - libimlib2 libimlib2-dev (epghttpd) - libjpeg libjpeg-dev (epghttpd) - python libpython libpython-dev python-dev - mailutils (deinstall heirloom-mailx) !! - libsystemd-daemon-dev (if you like to use the systemd status interface) -> debian previous to 'sid' maybe libsystemd-dev? Example for Ubuntu (21.10): - libarchive12, libarchive-dev - libz, libz-dev - libssl-dev - libcurl4-nss-dev (oder libcurl3-nss-dev) - libxslt1.1 libxslt1-dev libxml2 libxml2-dev - libmariadb-dev mariadb-common - libjpeg62-dev libjpeg62 - uuid uuid-dev - libjansson4 libjansson-dev - libmicrohttpd libmicrohttpd-dev (epghttpd) - libimlib2 libimlib2-dev (epghttpd) - libjpeg libjpeg-dev (epghttpd) - python libpython libpython-dev python-dev - mailutils (deinstall heirloom-mailx) !! - libsystemd-daemon-dev (if you like to use the systemd status interface) My-SQL Setup: ------------- You need a running mariadb database. Consult the manual of your linux-flavour how to install and configure mariadb. Remember to put your db-files in a large enough filesystem: with many channels, many prefetched days and many pictures you will easily need 3 or more Gb of disk space. When mariadb is up and running you need to prepare the database and access-rights for use with epgd: - login as root: #> mysql -u root -p CREATE DATABASE epg2vdr charset utf8; CREATE USER 'epg2vdr'@'%' IDENTIFIED BY 'epg'; GRANT ALL PRIVILEGES ON epg2vdr.* TO 'epg2vdr'@'%'; GRANT ALL PRIVILEGES ON epg2vdr.* TO 'epg2vdr'@'localhost' IDENTIFIED BY 'epg'; FLUSH PRIVILEGES; - make sure the database characterset (character_set_database) is set to latin1, otherwise the index size will be exeeded, you can adjust the client and session character sets as you like. - adjust the bind address in my.cnf: adjust the line bind-address = 127.0.0.1 in youre my.cnf (mostly found here: /etc/mysql/...) to the address where the mariadb server should listen on - Or comment out if it should listen on all interfaces. - switch of binary logging: comment line starting with 'log_bin' in youre my.cnf Login as epg2vdr: ----------------- #> mysql -u epg2vdr -pepg -Depg2vdr - or remote #> mysql -u epg2vdr -pepg -Depg2vdr -h <host> If you have problems with setting up and granting access rights to your epg2vdr read "MySQL Hints" near the end of this file. - show all users: use mysql SELECT * FROM user; Possible Create Index Problems: ------------------------------- If you get this error on epgd start while epgd is creating indexes and tables: "Index column size too large. The maximum column size is 767 bytes" Make sure the server charecter-set is set to latin1 (character-set-server = latin1) Remove all other collations from your mariadb config. We use UTF8 only on client side! epgd Installation: ------------------ epgd is source distributed only. So you have to build your binary and make some adjustments to your system. So we try to help with some stuff in the contrib directory, it's not a bad idea to learn something about your linux's init-system and init-scripts. - Unpack - Modify Make.config (BINDEST, PLGDEST and INIT_SYSTEM) to meet your local systems requirements - Call "make" - Call "make plugins" - Call "make install" - Call "make install-plugins" - Create directory /etc/epgd - cp configs/* /etc/epgd - merge(*) the config parts of the epgd plugins (./PLUGINS/*/configs) to /etc/epgd -> merge /etc/epgd/channelmap.conf and /etc/epgd/epgd.conf -> copy ./PLUGINS/*/configs/*.xsl to /etc/epgd - modify config (/etc/epgd/channelmap.conf and /etc/epgd/epgd.conf) like your needs - Create the database (see below) - Install epglv - the lib installed by make install and the functions will installed as described in epglv/README) - Start epgd (see below) If you don't know how to merge you can start of with #> cat ./PLUGINS/*/configs/channelmap.conf >>/etc/epgd/channelmap.conf #> cat ./PLUGINS/*/configs/epgd.conf >>/etc/epgd/epgd.conf This is only true if you use all plugins. If you use only one of them replace "*" with the plugin you use. Look carefully through your generated files! epgd Update: ------------ If a table alter is required the epgd alter and create tables automatically after start (if the new, modified epg.dat is installed in the config directory). Therefore always start the epgd before starting the epghttpd and the VDR after an update. Please be patient, the alter can take some time depending on the size of the table! Starting epgd and init-Scripts: ------------------------------- There are many ways to start epgd. For a first try or debugging sessions you may want to start it simply by typing at the root command-prompt (paths have to be adjusted according to your changes to BINDEST and PLGDEST in Make.config): #> export LANG="de_DE.UTF-8" #> ulimit -c unlimited #so you can torture the developers with back traces #> /usr/local/bin/epgd -n -p /usr/local/lib/epgd/plugins epgd is configured in /etc/epgd/epgd.conf. But you can overwrite the following options on the command line: -n don't daemonize -t log to stdout -c <config-dir> use config in <config-dir> -p <plugin-dir> load plugins from <plugin-dir> -l <log-level> set log level For production use you should start epgd after net- and mariadb services via your init-system. Some start-scripts have been put to ./contrib hopefully serving your needs (or at least giving you an idea of how to proceed). epgd Configuration: ------------------- DbHost = ip/name of mariadb server host (default localhost) DbPort = port of the mariadb server (default 3306) DbName = name of the database (default epg2vdr) DbUser = database user (default epg2vdr) DbPass = database password (default epg) DaysInAdvance = (default 8) Download (insert) EPG information for the next # days DaysToUpdate = (default 2) Update # already insert days UpdateTime = 0 Perform automatic update every # hours (default: 2 hours) XmlStoreToFs = Store XML files to the filesystem, for debugging (default 0) GetEpgImages = (default 0) Download images with the EPG information NetDevice = (default first device) network device of epgd (used eg. for WOL) HttpDevice (default all devices) epghttpd listen on this decive MaxImagesPerEvent How many pictures per event should be downloaded EpgImageSize = 0 = 174x130 1 = 329x245 2 = 525x400 CachePath = path to cache of the xml files if XmlStoreToFs or SeriesStoreToFs enabled (default /var/cache/epgd) SeriesEnabled get series from eplists.constabel.net (default 1) SeriesPort (default 2006) SeriesStoreToFs (default 0) SeriesUrl (default eplists.constabel.net) ScrapEpg (default 1) scrap EPG content 0/1 ScrapRecordings (default 1) scrap recordings 0/1 EpgView (default eventsview.sql) use this if you want to design your own view for detailed EPG views and live-plugin eventsview.sql can be overwritten on updates, so use a different name EpgViewWeb (default eventsviewplain.sql) use this if you want to design your own view for detailed EPG views in WebIF eventsviewplain.sql can be overwritten on updates, so use a different name CheckInitial = 1 (default 1) enable or disable the initial update of your external EPG provider after starting epgd UpdateThreshold = 500 (default 500) call merging of DVB and external EPG after <x> DVB changes and update clients UseProxy Enable HTTP Proxy (default 0) HttpProxy Name of the proxy UserName Username of your proxy Password Password of your proxy If you are using some kind of pattern driven logfile parsing software like logchecker you may want to use ./contrib/epgd.ignore HINTS / PITFALLS: ----------------- - the recordings hosted on a central storage (NAS, ...) and the vdr2epg toggle some recordings in the table round robin without any changes on it this can occor if: - the VDRs use different options for the filesystem (--vfat and/or --dirnames) - the VDRs use different cahracter encoding and language settings --vfat bzw. --dirnames MariaBb HINTS: --------------- - If you cannot figure out why you get Access denied, remove all entries from the user table that have Host values with wildcards contained (entries that match '%' or '_' characters). A very common error is to insert a new entry with Host='%' and User='some_user', thinking that this allows you to specify localhost to connect from the same machine. The reason that this does not work is that the default privileges include an entry with Host='localhost' and User=''. Because that entry has a Host value 'localhost' that is more specific than '%', it is used in preference to the new entry when connecting from localhost! The correct procedure is to insert a second entry with Host='localhost' and User='some_user', or to delete the entry with Host='localhost' and User=''. After deleting the entry, remember to issue a "FLUSH PRIVILEGES" statement to reload the grant tables. - *ATTENTION* if you have binary-logging (log_bin) enabled! disable it OR add log-bin-trust-function-creators to your configuration and restart the database Slow Database : --------------- On problems with slow database performance try this settings (adjust to the available): innodb_buffer_pool_size = 10G key_buffer_size = 10M max_allowed_packet = 1G thread_stack = 192K thread_cache_size = 8 query_cache_limit = 1M query_cache_size = 8M Considerations: --------------- By design epgd can handle multiple clients. If you are looking for a solution for your vdr-network plan your implementation carefully. Which clients should be able to connect? Which epg-solution is in use on each client? Has any of the clients patches or plugins running manipulating the epg? And now the most important hint: read the epg2vdr README before you continue. epgd can generate large amounts of data, as well db-data and logging-data - so don't be surprised. Manipulating data (read, write, move around) does not only need appropriate disk space, but also appropriate I/O-performance. If you run into problems and want to start from a clean base, stop epgd and drop all data (take a look at the scripts in ./scripts) and restart epgd. ============ HOWTOs ============ select starttimes from table timers: select from_unixtime(day), starttime, from_unixtime(_starttime) as 'event starttime', from_unixtime(day + ((starttime % 100) * 60) + ((starttime div 100) * 60 * 60)) as 'timer starttime' from timers;