Automatic indexing not working if the file is added by an external Debian client
Closed this issue · 7 comments
Describe the bug
Hello!
When compressing a game with 7zip with an external client (another server) and storing it under the gamevault's /files directory, it avoids completely the 'watch' detection
There's neither a "file has been detected" or an "error" produced. It literally skips it. The game will only get detected on a manual re-index
From my personal testing, this is only happening on multiple Debian 12 LXCs (Proxmox), but not on Ubuntu LXCs, for example, my Gamevault current installation.
My syntaxis for 7zip command is:
7zz a /mnt/gamevault/game.7z game
To Reproduce
Steps to reproduce the behavior:
- Get your game in a folder separate from gamevault's directory
- 7zip it, with the output directory to your /files directory (the gamevault's location for games)
- After it's been completed, the logs do not report anything new. During the compression you could have a separate ssh client open seeing the logs with 'docker compose logs -f'
Alternative for reproducing:
You could create a zero-ed file with dd with a recognisable name, for example "Minecraft" and try the command, like this:
dd if=/dev/zero of=Minecraft bs=1M count=1024
then 7zip it
7zz a gamevaultStorage/Minecraft.7z Minecraft
Notice that it isn't detected in the logs.
Do note that all my files have the same permissions and my gamevault is running as root (else, the manual reindexing wouldn't work)
Expected behavior
the folder that is being 'watched' should automatically reindex incoming files
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
OS: Debian/Ubuntu 22.04.03 LTS Unprivileged LXC
External client is running Debian 12
Additional context
7zip application was downloaded with apt (7zip package), but it seems there's multiple versions depending on distro. Not sure why, it's either 21.03 or 22.03. Shouldn't matter much
Compression is handled by an external powerful client instead of my gamevault's LXC. Both gamevault and my external client store games in a CephFS (kinda like NFS/SMB) shared cluster-wide filesystem.
I was not able to reproduce this issue in the same container as my Gamevault installation (as it's running Ubuntu), but I was consistently able to reproduce it by storing directly to the shared storage from Debian-based containers. Permissions are correctly set and manual reindexing works as expected, it's just that it isn't detected automatically.
It seems like the root cause of this issue may be related to the watch functionality provided by Chokidar. Considering this, it might be more appropriate to address the problem directly on their GitHub repository.
I've already implemented listening to all events, and as of now, there doesn't appear to be any further action we can take on our end.
If it helps, I can bring back the traditional 5-minute timer in addition to scan files. Let me know if that would be a satisfactory solution for you.
If you decide to open up an issue on https://github.com/paulmillr/chokidar make sure to link it here please.
Would like the traditional interval_based indexing back as a non-default option. My other option was reading the code and setup an API call to reindex, but I think that could wait (as I'm a bit new to API stuff)
I think it would be more efficient to store the "last time folder was edited" variable from the /files directory and compare against it in X minutes, instead of the previous Cron job that used to reindex the entire library every few minutes. Or could be done with a simple 'while loop'.
I don't know Typescript, so I can't contribute or make a pull request, but I can drop the idea as an algorithm
while true{
oldTime = get.folder_mtime
sleep(minutesInterval*60)
newTime = get.folder_mtime
if oldTime != newTime
index()
}
Maybe it is a shortcoming of chokidar and we should maybe try to use @parcel/watcher instead. Would you like to try out a build with another watch-library and tell me if you can still reproduce this?
Please try out docker pull phalcode/gamevault-backend:parcel-file-watcher
Exact same issues with the image provided. May be the use of a remote distributed filesystem as Ceph the cause?
I tested NFS too (exporting a local ssd instead) to confirm if this filesystem was the problem, but it didn't work at all.
NFS testing server settings: /nfs *(rw,async,no_subtree_check,all_squash)
Well this is very unfortunate, and yes, the cause of this lies out of my hands i think.
I will reimplement an optional timer based reindex, like it used to be as it is the simplest for me to do rn.
implemented. Default will be 60 minutes.