timestamp out of range during database update

[ego1105]

Hi Frank,

since last few days I get an overflowError:

$ mtv_dl -v list channel=ZDF topic=Heute
2022-02-18T18:23:24  DEBUG    Opening Filmliste database
                              PosixPath('/smb/video/MediathekView/.Filmliste.0.18.1.sqlite').
                     DEBUG    Opening History database PosixPath('/smb/video/MediathekView/.History.sqlite').
                     DEBUG    Initializing Filmliste database in
                              PosixPath('/smb/video/MediathekView/.Filmliste.0.18.1.sqlite').
                     DEBUG    Opening database from 'https://liste.mediathekview.de/Filmliste-akt.xz'.
Downloading database ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:00
2022-02-18T18:23:29  DEBUG    Loading database items.
Reading database items ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.1% 0:00:48
2022-02-18T18:25:21  CRITICAL <class 'OverflowError'>: timestamp out of range for platform time_t
                                File "/home/pi/.local/bin/mtv_dl", line 10, in <module>
                                  sys.exit(main())
                                File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 1239, in main
                                  history=cw_dir / HISTORY_DATABASE_FILE)
                                File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 434, in __init__
                                  self.initialize_filmliste()
                                File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 387, in
                              initialize_filmliste
                                  """, self._get_shows())
                                File "/home/pi/.local/lib/python3.7/site-packages/mtv_dl.py", line 540, in _get_shows
                                  start = datetime.fromtimestamp(int(show['start']), tz=utc_zone).replace(tzinfo=None)

The code is running on Raspian Linux 10 32bit, mtv_dl installed via pip3.

Could you please have a look?
Thx
Maik

[fnep]

Hmm, interesting issue. I was not aware that fromtimestamp is platform dependent. I will try to apply the approach described here: https://stackoverflow.com/a/65494704

[fnep]

Would you please test if it works with the master branch (before I release the fix)? I have no system where it failed.

[ego1105]

Thanks for your effort.

Running the updated version from the repo

./mtv_dl.py -v --logfile out list channel=ZDF topic=Heute

fails a bit later at a timestamp method:

2022-02-19T18:20:38+0100 DEBUG    Opening Filmliste database PosixPath('/home/pi/devel/mtv_dl/.Filmliste.0.0.0.sqlite').
2022-02-19T18:20:38+0100 DEBUG    Opening History database PosixPath('/home/pi/devel/mtv_dl/.History.sqlite').
2022-02-19T18:20:38+0100 DEBUG    Initializing Filmliste database in PosixPath('/home/pi/devel/mtv_dl/.Filmliste.0.0.0.sqlite').
2022-02-19T18:20:38+0100 DEBUG    Opening database from 'https://liste.mediathekview.de/Filmliste-akt.xz'.
2022-02-19T18:20:43+0100 DEBUG    Loading database items.
2022-02-19T18:22:23+0100 CRITICAL <class 'OverflowError'>: timestamp out of range for platform time_t
  File "./mtv_dl.py", line 1295, in <module>
    main()
  File "./mtv_dl.py", line 1233, in main
    history=cw_dir / HISTORY_DATABASE_FILE)
  File "./mtv_dl.py", line 434, in __init__
    self.initialize_filmliste()
  File "./mtv_dl.py", line 387, in initialize_filmliste
    """, self._get_shows())
  File "./mtv_dl.py", line 542, in _get_shows
    'hash': self._show_hash(channel, topic, title, size, start),
  File "./mtv_dl.py", line 467, in _show_hash
    h.update(str(start.timestamp()).encode())

Adding some logging:

467        logger.debug('h.update %s', str(start))
468        logger.debug('h.update %s', str(start.timestamp()))
469        h.update(str(start.timestamp()).encode())

shows this:

2022-02-19T18:28:20+0100 DEBUG    h.update 2022-01-31 08:00:00
2022-02-19T18:28:20+0100 DEBUG    h.update 1643612400.0
2022-02-19T18:28:20+0100 DEBUG    h.update 2202-02-15 08:00:00
2022-02-19T18:28:20+0100 CRITICAL <class 'OverflowError'>: timestamp out of range for platform time_t
  File "./mtv_dl.py", line 1297, in <module>
    main()
  File "./mtv_dl.py", line 1235, in main
    history=cw_dir / HISTORY_DATABASE_FILE)
  File "./mtv_dl.py", line 434, in __init__
    self.initialize_filmliste()
  File "./mtv_dl.py", line 387, in initialize_filmliste
    """, self._get_shows())
  File "./mtv_dl.py", line 544, in _get_shows
    'hash': self._show_hash(channel, topic, title, size, start),
  File "./mtv_dl.py", line 468, in _show_hash
    logger.debug('h.update %s', str(start.timestamp()))

The year is 2202, which is probably a typo in the Filmliste. This is causing the overflow.

[fnep]

Well, they are just planning ahead a bit. ;) At least it clearly explains how they made it behind the 2038 border.

I changed the code to skip shows causing the issue, hoping that once we come closer to real shows airing after 2038 we don't care about 32 platforms anymore.

Please give it another try.

[ego1105]

Thanks, now it works.

[fnep]

Perfect, glad to hear, I just updated the pypi package.