Download Cap & Delete Old Media/Days to Keep conflict

Question

Download Cap & Delete Old Media/Days to Keep conflict

Closed this issue 17 days ago · 10 comments

It seems like the two items mentioned in title conflict (perhaps just my interpretation).

My understanding of Download Cap is to have tubesync NOT download videos which were posted by the uploader beyond the set date.
My understanding of Delete Old Media/Days to Keep is to delete video files which were downloaded more than “Days to keep” days ago.

However it seems when monitoring a playlist, even with Download Cap set to set to No Cap, if Days to Keep is set to 14, and Delete Old Media is ticked, items added to the playlist which were posted by the uploaded more than 14 days ago will not be downloaded.

Answer 1 · 2024-12-09T21:57:57.000Z

The situation is a bit complicated. Here is my understanding:

Download cap is to prevent downloading items published before that time (I use 7 days for news channels).
Delete old media (I use 10 days for news channels) is when the media item is removed from the TubeSync database.
After it's removed, a later index of the channel is likely to add it back but it won't download again because it is now older than the download cap.
Delete removed media is for removing items that are no longer in the source channel or playlist from the TubeSync database.
Delete files on disk should control if the downloaded media file is deleted from the disk when anything is removed from the TubeSync database.

I've personally seen my files on disk survive beyond 10 days, so I'm not sure how well that last option is currently working. However, it's better to fail to delete files than to delete too many files. 🙃

Answer 2 · 2024-12-10T00:12:55.000Z

Yes this is exactly as I have mentioned above so we are both expecting the same behaviour.

Issue I’m having is setting Download Cap is not working as you and I both expect it.
I have a playlist as a media source. Download Cap is set to No Cap. I expect this to mean that any video added to the playlist, regardless of date uploaded, should download. Unfortunately this does not happen.
I also have Delete Old Media enabled and Days to Keep set to 14 Days. Unfortunately what this does is stops video’s which were uploaded more than 14 days ago to NOT be downloaded. Whereas I would expect this to mean media which was downloaded by TubeSync more than 14 days ago to be deleted.

So as I mentioned, it seems to be some kind of conflict or wrong logic with these settings.

Answer 3 · 2024-12-10T03:05:24.000Z

I have a playlist as a media source. Download Cap is set to No Cap. I expect this to mean that any video added to the playlist, regardless of date uploaded, should download. Unfortunately this does not happen.

I have a similar playlist. I do see downloads happening from that playlist even for videos that were uploaded many years ago.

I also have Delete Old Media enabled and Days to Keep set to 14 Days. Unfortunately what this does is stops video’s which were uploaded more than 14 days ago to NOT be downloaded.

I don't use delete old media and days to keep for the playlist I have. I'm using delete removed media instead. This is an attempt to keep my TubeSync web interface matched to the playlist.

The actual media files aren't supposed to be deleted, and indeed they do remain, after I remove videos from the playlist. I configured Jellyfin to allow my user to delete media from this library, so I can delete or not after I watch the videos.

Whereas I would expect this to mean media which was downloaded by TubeSync more than 14 days ago to be deleted.

I agree that the logic should be to download the video, keep them for 14 days, then remove the media file (if you have that setting checked) and remove the metadata from the database and set it to skip downloads.

I'll have a look at the code soonish since I'm also seeing unexpected results.

Answer 4 · 2024-12-10T06:48:58.000Z

Historically the unexpected behavior around the download cap and days to keep was due two common-ish events:

The download cap uses the publish date from the metadata of the media item, this can cause unexpected things such as media is uploaded and left hidden or private for longer than your download cap, then made visible again. The media item is "new" but as the publish date is older than your download cap it gets skipped.
Media items can be queued to be deleted in the future and the task gets deleted, this gets reset with a reset-tasks but it's generally another annoying task scheduling problem

There may be others, but these are the most common two situations I've encountered myself.

Answer 5 · 2024-12-10T15:47:33.000Z

For my particular situation it's certainly not 1.

The download cap uses the publish date from the metadata of the media item, this can cause unexpected things such as media is uploaded and left hidden or private for longer than your download cap, then made visible again. The media item is "new" but as the publish date is older than your download cap it gets skipped.

I've use LTT as a basis for testing as they upload & publish regularly and they often have timely content so it would not have been uploaded and hidden for long periods. For those which were published after the "Days to Keep" they have the following rationale for why they were skipped: This media may be skipped due to error(s) or not matching a filter condition. If I change the Days to Keep to include just 1 day prior to the publish date then the media is downloaded no problem. Definitely seems to be just that setting interfering with the logic of Download Cap

Media items can be queued to be deleted in the future and the task gets deleted, this gets reset with a reset-tasks but it's generally another annoying task scheduling problem

Media deletion works perfectly for me.

Answer 6 · 2024-12-10T16:00:09.000Z

Seems to be this bit here in filtering.py starting at lin 130 where it is filtering based on "Days to Keep" instead of "Download Cap"

# If the source has a cut-off, check the upload date is within the allowed delta
def filter_source_cutoff(instance: Media):
    if instance.source.delete_old_media and instance.source.days_to_keep > 0:
        if not isinstance(instance.published, datetime):
            # Media has no known published date or incomplete metadata
            log.info(
                f"Media: {instance.source} / {instance} has no published date, skipping"
            )
            return True

        delta = timezone.now() - timedelta(days=instance.source.days_to_keep)
        if instance.published < delta:
            # Media was published after the cutoff date, skip it
            log.info(
                f"Media: {instance.source} / {instance} is older than "
                f"{instance.source.days_to_keep} days, skipping"
            )
            return True

I think the 3 references here to days_to_keep can be changed to download_cap which is also defined in models.py

The logic being media older than download_cap will not be downloaded.
Downloaded media older than days_to_keep will be deleted.

Answer 7 · 2024-12-11T10:37:38.000Z

I filed PR #576 to stop using that function for filtering.

I think it needs to be comparing days to keep against downloaded date to work as we both expected.

Answer 8 · 2024-12-11T11:15:35.000Z

Oh right you are, yes this was introduced in a previous PR and I didn't notice it was using the wrong logic on filtering. Good catch.

Answer 9 · 2024-12-11T12:01:30.000Z

Glad I could help. Sorry it took a bit to explain it, I’m not super technical and writing yaml for my containers and home assistant is about the most “technical” thing I do 😂

Answer 10 · 2024-12-11T12:25:45.000Z

Not a problem at all, finding and reporting issues is half the work anyway. Thanks to @tcely for the fix.