thijstakken/MusicService

Files getting lost/out of sync

Closed this issue · 2 comments

thijstakken commented

When the container has been started, and has successfully downloaded and converted the song to MP3.
It will then save that this song has been downloaded in the downloaded.txt file which serves as a database.

But, when the container gets stopped right in the middle of a succesfull download and the upload process, the file will be lost, since the local music folder is not part of any volume (persistant storage).

This will result in the following behavior:

  1. When the container is run again, it will note that the song has already been downloaded
  2. Therefore, it will not be downloaded again
  3. And therefore the song will never be uploaded to your cloud storage
thijstakken commented

Possible solutions,

  1. Only record in the database that a song has been downloaded, after it also has been uploaded to the storage account

  2. Make the music folder persistent as a volume, this will then serve as a cache place for MP3's, so that when the container is restarted it can continue where it left off. If a volume is the best option, there has to be a mechanism that deletes the local cache with downloaded MP3's after they have been uploaded to cloud storage. Otherwise the folder will pile up endlessly with old MP3's.

thijstakken commented

It has to be option 2.

It looks that because of YT-dls implementation, they record the video id in the archive (database) file, the moment that it has been successfully downloaded. Because of this design, it seems to be impossible to record the ID to a database later. It just so happens that is does this the moment a download is successful. Therefore option 1 is not possible.
https://github.com/ytdl-org/youtube-dl/blob/master/README.md#how-do-i-download-only-new-videos-from-a-playlist