Document useful way of editing files locally with automatic sync on Jean Zay : (itnotify + rsync), editor extensions, and sshfs
lesteve opened this issue · 2 comments
One usual way people work is to edit their file locally on their computer (they have all the tools they are used to, editor, etc ...) and synchronize on Jean Zay automatically (adding an explicit step like rsync or git push/pull is painful because you will forget it sometimes and when you do it will cost you a lot of time and frustration).
To synchronize a code folder (local -> Jean Zay)
inotify + rsync
The simplest and most generic thing we have found for automatic local -> Jean Zay sync is to use a script like this on your local computer (needs inotify and rsync):
while inotifywait -r -e modify,create,delete ~/folder-to-sync
do
rsync -avz ~/folder-to-sync jean-zay:~/folder-to-sync --delete
done
If you are on a OSX you need to use fswatch rather than inotifywait, according to this the following command should work (not tested, comment if this issue if it does not work for you!).
fswatch -o ~/path/to/watch | xargs -n1 -I{} \
rsync -avz ~/folder-to-sync jean-zay:~/folder-to-sync --delete
Editor extensions
Tip from @tomMoral if you are using Visual Studio Code (I also seem to remember there is an equivalent PyCharm functionality as well so this may be accessible in other editors as well):
You might consider using the vscode extension: https://marketplace.visualstudio.com/items?itemName=vscode-ext.sync-rsync
You can define a remote folder and sync your local project files to it.
There is also a on save option that allows you to sync your project on save in vscode.
To view an arbitrary Jean Zay folder on your local computer
An alternative is sshfs with the right options to prevent being automatically disconnected
mkdir -p ~/mnt/jean-zay
sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3 \
your-jean-zay-login@jean-zay:/your/jean/zay/folder ~/mnt/jean-zay
sshfs can be useful for example to look at TensorBoard logs on Jean Zay with TensorBoard running on your local computer.
For editing files, user feed-back seem to indicate that using sshfs to edit files is a bit laggy so not that useful in practice, if you have tweaks or suggestions to make sshfs a more viable option for editing files, please comment in this issue!
The command rsync -avz ~/folder-to-sync jean-zay:~/folder-to-sync --delete may be slow when the folder you want to update contains many small files. This command is nice but does not scale well as rsync check every files in the destination folder before uploading any new change. A workaround in my case where I upload code is to add the option --cvs-exclude to avoid copying .git that contains many small files and clean up the folder once in a while to keep it at a reasonnable size (below 1-2 Go).