Error: [Errno 2] No such file or directory. Files exist can touch and ls.
Opened this issue · 47 comments
I am getting the bellow. Very confused as I can see the files. Running on Unraid
`root@TheLibrary:/mnt/user/scripts/PlexCache# python3 plexcache.py
*** PlexCache ***
Debug mode is active, NO FILE WILL BE MOVED.
Fetching DuzAwe's onDeck media...
Editing file paths...
Fetching subtitles...
Fetching watched media...
Fetching DuzAwe's watched media...
Editing file paths...
Fetching subtitles...
Nothing to move to array
Error: [Errno 2] No such file or directory: "/mnt/user/TV/It's Always Sunny in Philadelphia (2005) [TVdb-75805]/Season 13/It's.Always.Sunny.in.Philadelphia.-.S13E03.-.The.Gang.Beats.Boggs-.Ladies.Reboot.WEBDL-1080p.mkv"
root@TheLibrary:/mnt/user/scripts/PlexCache# nano plexcache_settings.json
root@TheLibrary:/mnt/user/scripts/PlexCache# python3 plexcache.py
*** PlexCache ***
Fetching DuzAwe's onDeck media...
Editing file paths...
Fetching subtitles...
Loading watched media from cache...
Nothing to move to array
Error: [Errno 2] No such file or directory: "/mnt/user/TV/It's Always Sunny in Philadelphia (2005) [TVdb-75805]/Season 13/It's.Always.Sunny.in.Philadelphia.-.S13E03.-.The.Gang.Beats.Boggs-.Ladies.Reboot.WEBDL-1080p.mkv"
root@TheLibrary:/mnt/user/scripts/PlexCache# ^C
root@TheLibrary:/mnt/user/scripts/PlexCache# ^C
root@TheLibrary:/mnt/user/scripts/PlexCache# ls /mnt/user/TV/It's Always Sunny in Philadelphia (2005) [TVdb-75805]/Season 13/
^C
root@TheLibrary:/mnt/user/scripts/PlexCache# python3 plexcache.py
*** PlexCache ***
Fetching DuzAwe's onDeck media...
Editing file paths...
Fetching subtitles...
Loading watched media from cache...
Nothing to move to array
Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Trek - Strange New Worlds (2022) [TVdb-382389]/Season 02/Star.Trek-.Strange.New.Worlds.-.S02E07.-.Those.Old.Scientists.WEBDL-2160p.mkv'`
Would you mind sending the log file instead?
Of Course Sorry, Was late when I was messing with this.
plexcache_log_20230727_2231.log
plexcache_log_20230727_2235.log
plexcache_log_20230727_2236.log
plexcache_log_20230727_2241.log
plexcache_log_latest.log
Thank you for the logs file! I had a first look and it doesn't seem you have any misconfiguration, and the script does seem to be working with other files, correct?
I am not saying there is no issue with the script, but I can't seem to find the culprit at the moment that could explain why that specific file(s) doesn't exist for the script. The only things it comes up are permissions of those files, even though I thought the script would run as root in unraid?
Just to rule it out, would you mind checking the permissions of the file that are giving you issues? Also check the permissions of the files (possibly even of the same show) that are actually working fine. ls -l
I will be able to pay more attention to this issue at the beginning of the week, I'm currently in between night shifts and my free time (and brain power) is limited.
It does not appear to be move any files
Last login: Thu Jul 27 22:09:54 2023 from 192.168.0.46
Linux 6.1.38-Unraid.
root@TheLibrary:~# cd /mnt/user/scripts/PlexCache
root@TheLibrary:/mnt/user/scripts/PlexCache# ls -l
total 4325
-rw-rw-rw- 1 root root 3307 Jul 27 21:55 README.md
-rw-rw-rw- 1 root root 64589 Jul 27 22:29 plexcache.py
-rw-rw-rw- 1 root root 3642057 Jul 27 22:32 plexcache_log_20230727_2231.log
-rw-rw-rw- 1 root root 239363 Jul 27 22:36 plexcache_log_20230727_2235.log
-rw-rw-rw- 1 root root 108179 Jul 27 22:36 plexcache_log_20230727_2236.log
-rw-rw-rw- 1 root root 104966 Jul 27 22:41 plexcache_log_20230727_2241.log
lrwxrwxrwx 1 root root 59 Jul 27 22:41 plexcache_log_latest.log -> /mnt/user/scripts/PlexCache/plexcache_log_20230727_2241.log
-rw-rw-rw- 1 root root 778 Jul 27 22:41 plexcache_settings.json
-rw-rw-rw- 1 root root 18264 Jul 27 21:55 plexcache_setup.py
-rw-rw-rw- 1 root root 85346 Jul 27 22:36 plexcache_watched_cache.json
-rw-rw-rw- 1 root root 16 Jul 27 21:55 requirements.txt
root@TheLibrary:/mnt/user/scripts/PlexCache# ls
README.md plexcache_log_20230727_2236.log plexcache_setup.py
plexcache.py plexcache_log_20230727_2241.log plexcache_watched_cache.json
plexcache_log_20230727_2231.log plexcache_log_latest.log@ requirements.txt
plexcache_log_20230727_2235.log plexcache_settings.json
root@TheLibrary:/mnt/user/scripts/PlexCache# nano plexcache_settings.json
root@TheLibrary:/mnt/user/scripts/PlexCache# cd /mnt/ingress/
root@TheLibrary:/mnt/ingress# ls -l
total 16
drwxrwxrwx 4 nobody users 4 Jul 24 14:32 Back\ Up/
drwxrwxrwx 3 nobody users 3 Jul 28 05:44 Movies/
drwxrwxrwx 9 nobody users 9 Jul 26 11:17 TV/
drwxrwxrwx 5 nobody users 5 Jul 27 21:55 scripts/
root@TheLibrary:/mnt/ingress# cd TV/
root@TheLibrary:/mnt/ingress/TV# ls -l
total 7
drwxrwxrwx 3 nobody users 3 May 21 19:07 Foundation\ (2021)\ [tvdb-366972]/
drwxrwxrwx 3 nobody users 3 Jul 28 01:24 Good\ Omens\ (2019)\ [tvdb-359569]/
drwxrwxrwx 3 nobody users 3 May 22 07:12 Harley\ Quinn\ (2019)\ [tvdb-365677]/
drwxrwxrwx 3 nobody users 3 Jul 27 21:04 Looney\ Tunes\ Cartoons\ (2020)\ [tvdb-380849]/
drwxrwxrwx 3 nobody users 3 May 21 19:08 Love,\ Death\ &\ Robots\ (2019)\ [tvdb-357888]/
drwxrwxrwx 3 nobody users 3 Jun 18 00:59 What\ We\ Do\ in\ the\ Shadows\ (2019)\ [tvdb-358211]/
drwxrwxrwx 3 nobody users 3 Jul 3 03:54 Witcher,\ The\ (2019)\ [tvdb-362696]/
root@TheLibrary:/mnt/ingress/TV#
Also don't worry about it. I assume your NHS, don't be stressing about hobby bugs
Also don't worry about it. I assume your NHS, don't be stressing about hobby bugs
Guilty! I've updated the script ever so slightly:
- The script will not exit while in debug and encountering an issue during file 'movement', hopefully it will make it easier to understand where the issue is.
- Introduced a fixed list of subtitles extensions, as I noticed from your log it was moving the .nfo files as well as the media.
That is probably it for today.
Also...thank you so so much! ❤️
Afraid, It still thinks the file doesn't exist. It's hardly as straight forward as it not escaping the ( ) or [ ].
So the behaviour is the same, It trys the first file and doesn't progress. I git pulled before I ran.
So as far I understood the file /mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 05/Star.Wars-.The.Clone.Wars.-.S05E17.-.Sabotage.Bluray-1080p.mkv does exist in your system. Is it present in the user0 folder or in the cache?
My bad about the exiting, I forgot that logging.critical also halt the script, now it should not stop according to the logic. I've also quickly added another check when calculating the free space.
It still doesn't fix the issue, but I'm trying to understand what's going on in you case.
Time to go for real this time!
It would be in the user. /mnt/user/TV.
cache is a practically empty zpool can confirm the script moves nothing to it.
First file fails, doesn't try the rest. All the files exist on the array under user.
I don't think I'm gonna be able to do much today or tomorrow but I'm actively thinking on how to solve this issue.
When the debug mode is off, the script is designed to exit in case it encounters an error during the moving process. I've removed this logic in this version of the file plexache.
My idea is to let the script move at least some files to see if there is any underlying issue I'm/We're not seeing and could lead to a solution.
Perhaps might be better to set the script to copy only the minimum necessary files and have a backup of the data, just in case the script accidentally corrupts any files. It is unlikely but it's always better to be safe than sorry.
I find it odd that some other users, including myself (though I may be biased), use the script without any issues. I'm thinking openly, and the only difference I can think of is the zpool. However, I cannot confirm this since I don't know the exact setup of other users, so they might actually use one already. Regardless, whether using zpool or not, it should be completely transparent to the script and shouldn't matter at all, that's why I'm not focusing on this but I just wanted to share this intrusive thought.
Well-aware coding is 99% in the brain, not the computer. Ran the new file same result. To test that theory you have I've run the script with it pointing at a disk on the array rather than the cache. Still wouldn't work, however I removed the () and [] from two of the directory and they no longer trigger the error....but there does seem to be an issue with the zpool.
See attached when going to disk13 (xfs) it appears to work with an estimated move size. With the target being ingress (zpool) It tells me there are no files to move.
brackets.removed.debug.to.zpool.log
brackets.removed.debug.to.disk.log
Good morning!
Thank you for the testing! It made me think a bit. Apparently I have to properly handle zpools with the script, so I have to test if any of the sources and/or destination are zpools and act accordingly. This will take a few hours and testing, unfortunately, I will work blind on it because I have no zpool in my Unraid system. So I beg for your patience.
Regarding the parenthesis, well, now I've wrapped the folder/file name properly, so it should not give any more issues.
I will work on it between later tonight or tomorrow. Now that I have a potential solution in my mind, I'm quite excited to see how it turns out and improve the script further. I'll keep you updated on my progress!
So I've made some edits but I cannot test it so I'm not sure it will work:
The script now checks the file system of the source and destination and uses "zfs clone" and "zfs destroy" to move files around. It will also calculate the free space accordingly as zfs also handle that differently.
Please let me know if it works at all!
Forgot to say, this is the new version: zfs
Ran it, No logs were produced. Still getting the file error.
Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 06/Star.Wars-.The.Clone.Wars.-.S06E01.-.The.Unknown.Bluray-1080p.mkv' Execution time of the script: 15 seconds
Will "zfs clone" and "zfs destroy" work of the array is XFS?
Ran it, No logs were produced. Still getting the file error.
Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 06/Star.Wars-.The.Clone.Wars.-.S06E01.-.The.Unknown.Bluray-1080p.mkv' Execution time of the script: 15 seconds
I can't understand why the script hasn't written the log file, I did not touch that at all, and even in the event of an error during the zfs operation, it will still produce a log entry, also I can see it did end the reach as it printed the execution time, which is only invoked at the end of the script.
Just odd.
I've added a few extra logging info to locate exactly where the error happens.
Will "zfs clone" and "zfs destroy" work of the array is XFS?
The script will use the zfs clone and destroy commands only when the file system is ZFS, otherwise it will use 'normal' commands for moving files.
Before running the last updated version, please install psutil (pip install psutil) because I'm using that to check for the free space as it is more system agnostic and should work on ZFS too.
Syntax error and my pythons not good enough to figure it out :P
:/mnt/user/scripts/PlexCache# python3 plexcache_zfs.py
File "/mnt/user/scripts/PlexCache/plexcache_zfs.py", line 1144
if check_path_exists(user_path)
^
SyntaxError: invalid syntax
Sorry if I took so long to actually answer. My bathroom decided was a good time to leak some water.
Either way I have no idea what the hell happened in my code, it was spamming me with a very generic module error even though I did test it. I've now basically rewritten the zfs logic from scratch, tested the script and it works, but as usual, I cannot confirm the zfs logic actually works.
Thank you for your patience!
Still no logs
Fetching subtitles...
Nothing to move to array
Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 06/Star.Wars-.The.Clone.Wars.-.S06E02.-.Conspiracy.Bluray-1080p.mkv'
So no log at all? Like not even interrupted?
I get having the error, but it would still start logging at the beginning way before interacting with the new zfs logic, and even if it return an error, the script will still continue as I've put no exits on purpose.
No logs from either script or branch? This is new. Otherwise still chocking don't the file
verbose-output.txt
That is so odd. So I guess if you run plexcache_exclude_file.py the log will work? The only difference between the zfs one is exactly the zfs logic, otherwise is line by line identical.
If this is the case then it means the zfs logic does make the whole thing crash (but why delete the log file? That is just weird).
No log from that either, nor when I switch back to main....odd there where logs. All the permissions are nobody and the scripts are root.
So you are not getting any logs at all? Have you tried setting another folder for the logs?
It's weird that it just stopped working for you, but I can't see the issue lying in the script to be fully honest with you. Not that it matters, I will keep trying to help anyway.
script_folder = "/mnt/user/system/plexcache/" # Folder path for the PlexCache script storing the settings, watchlist & watched cache files
logs_folder = script_folder # Change this if you want your logs in a different folder
.
plexcache_log_latest.log
.......Looking in the wrong folder. Here are the logs.
It happens, at least is sorted!
Just to be sure that is the log for the latest zfs script? If you don't mind getting the latest version (just pushed), try it and send the log again?
Most recent run from the most recent push.
plexcache_log_latest.log
Thank you for testing.
I've changed the logic to check the filesystem as it was actually giving some issue if run locally in a test script. It doesn't appear it was the issue per se judging from the log but because it's a wrapped function it might not highlight the error as it should. Worth a shot...step by step will get there.
I'm not an expert, I'm sure someone would be able to look at the code and fix it in a couple of minutes, but unfortunately I'm still a noob XD
plexcache_log_latest.log
No worries, Your leagues ahead of me. I gave up the code life to be a sys admin. Still getting the error.
Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 06/Star.Wars-.The.Clone.Wars.-.S06E04.-.Orders.Bluray-1080p.mkv'
So the error seems to happen when checking for the total size of the files to move, I've swapped the function so instead of os.path.getsize() I'm using os.stat() instead that should be more system agnostic.
I've also added some extra debug in the function itself to see where exactly is getting stuck, I think we are getting closer.
plexcache_log_latest.log
We have a slightly different error.
rootfs rootfs ('/mnt/ingress/Movies/The Whale (2022)/The Whale (2022) [imdb-tt13833688][Remux-2160p][DV HDR10][DTS-HD MA 7.1][HEVC]-CiNEPHiLES.mkv', '/mnt/user0/Movies/The Whale (2022)') Editing file paths... Error: [Errno 2] No such file or directory: '/mnt/user/TV/Star Wars - The Clone Wars (2008) [TVdb-83268]/Season 06/Star.Wars-.The.Clone.Wars.-.S06E05.-.An.Old.Friend.Bluray-1080p.mkv' Execution time of the script: 1 minute, 4 seconds
So it appears to have issues when calculating the total size of the files...
I've temporarily removed it, I will fix it later, I need to know what else doesn't work.
still No such file or directory:
Was this actually moved?
INFO - Moved file from /mnt/user0/Movies/The Machine (2023)/The Machine (2023) [imdb-tt11040844][AMZN][WEBDL-2160p][HDR10Plus][EAC3 5.1][h265]-CMRG.mkv to /mnt/ingress/Movies/The Machine (2023)
I suppose your /user0/ which refers to the actual array, it's not zfs, right? Not that it matter for the end goal but I want to understand why is giving errors if it isn't zfs...
It would appear yes that is on Ingress, so it did move. So that's progress! \0/
The array is XFS only the pools are ZFS, Cache and Ingress.
That's good, but for what I can see it used the "normal" logic rather than the zfs one to move that file. I'm not 100% sure but judging by the fact I didn't put a logging message on that logic (all the other zfs ones have it)...
In the meanwhile I've put that logging message to be sure about it and see what I will be able to do.
Unfortunately work calls me back, I will have time again in the weekend and hopefully we'll manage a working script!
plexcache_log_latest.log
No worries, New logs for the future
Good evening!
I've modified the script slightly:
- I've added find_disk function to get the actual disk where the given file is located, useful with unraid as it use FUSE system;
- The above function is called by get_filesystem_type which has been also modified, now has better error handling, plus windows compatibility while I was at it;
- The function move_file has been slightly modified in a way that it will use the above two and theoretically use the right logic depending on the filesystem;
- get_total_size_of_files function that was giving issues earlier on, I've modified it so that has better error handling, it will probably give error again, but I've made also an "alt" version of the zfs script which swaps when to call the function to fetch the actual disk on unraid, that might actually permanently fix it. But I cannot test it properly because I've made a mess with my tvseries folder (nothing to do with the script).
- There is also a version with no get_total_size_of_files which also doesn't check free space at all for both versions.
Let me know whenever you can 🤞🏻
So there are still issues, definitely.
I will need to figure out why is still giving errors with the free space / total size calculation, the logic is sound, I will work something out, in meantime I want to actually move stuff...but the new logic seems to not be working properly judging from your log.
I've added a couple extra logging and changed the debugging function to show me the move command in a different function, this should allow me to also see the filesystems in the log when not moving files.
It's not gonna work, this is literally for debugging.
Thank you for your patience, once again
More logs
More logs
Hey there! I'm really sorry for the wait. I promise I haven't forgotten about the project or your issue in specific. Life's been a whirlwind lately with work, family, and soon, university stuff. I'll jump back in as soon as I can.
Thanks for understanding and sorry again 🙌
No worries