hard-linking fails if source and tmp folder are on different partitions or devices
nvictus opened this issue · 13 comments
import_file() will raise an OSError: [Errno 18] Invalid cross-device link.
There should be an alternative if hard-linking is not possible (e.g. copying to tmp).
Is there support for this now by any chance?
Would be great to visualize tracks hosted on a different server without having to download them locally.
Are the tracks accessible by http?
Yes, they are hosted on a public web server and accessible by http.
I presume that this is related to #48? If so, I propose we continue the discussion there so that there aren't two threads to follow?
Hi! I'm having the same problem. I'm running HiGlass on an Ubuntu server, I have a hg-data folder but no hg-tmp, it runs on my mac but not on the server. I get the error OSError: [Errno 18] Invalid cross-device link: 'Growing.Rep2_hicpro.mcool' -> '/tmp/higlass-docker/Growing.Rep2_hicpro.mcool'
May I suggest creating a folder that you'll use for your data and letting higlass-manage know about it with --media-dir? Then you should be able to ingest datasets using --no-upload.
Thanks a lot for the quick reply!
I added the --media-dir as follows:
higlass-manage start -m /home/rishi/higlass_data
Then I tried to ingest it as follows:
higlass-manage ingest higlass_data/Growing.Rep2_hicpro.mcool --no-upload
I got the error:
state True
Inferred filetype: cooler
Inferred datatype: matrix
state True
name_text:
hg_name: default
command: python higlass-server/manage.py ingest_tileset --filename higlass_data/Growing.Rep2_hicpro.mcool --filetype cooler --datatype matrix --no-upload --uid MvVDCY9LTd-gOSc_NZdfIA
ERROR: CommandError: File does not exist under media root
I realised that since I've already specified --media-dir as higlass_data, I tried without it:
higlass-manage ingest Growing.Rep2_hicpro.mcool --no-upload
Then I got no error and it stopped at:
state True
Inferred filetype: cooler
Inferred datatype: matrix
state True
name_text:
hg_name: default
command: python higlass-server/manage.py ingest_tileset --filename Growing.Rep2_hicpro.mcool --filetype cooler --datatype matrix --no-upload --uid VTDNwwYLTLCOU6FBVUUo5g
When I tried to view the dataset, I got this error:
higlass-manage view Growing.Rep2_hicpro.mcool
state True
temp_dir: /tmp/higlass-docker
Inferred filetype: cooler
Inferred datatype: matrix
Traceback (most recent call last):
File "/home/rishi/.local/bin/higlass-manage", line 11, in
sys.exit(cli())
File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/rishi/.local/lib/python3.6/site-packages/higlass_manage/view.py", line 95, in view
checksum1 = md5(tileset_path)
File "/home/rishi/.local/lib/python3.6/site-packages/higlass_manage/common.py", line 15, in md5
with open(fname, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/rishi/hg-data/media/media/Growing.Rep2_hicpro.mcool'
Hi,
I tried changing both the --data-dir and --media-dir to my data folder, and it gave me this error:
state True
temp_dir: /tmp/higlass-docker
Inferred filetype: cooler
Inferred datatype: matrix
Traceback (most recent call last):
File "/home/rishi/.local/bin/higlass-manage", line 11, in
sys.exit(cli())
File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/rishi/.local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/rishi/.local/lib/python3.6/site-packages/higlass_manage/view.py", line 95, in view
checksum1 = md5(tileset_path)
File "/home/rishi/.local/lib/python3.6/site-packages/higlass_manage/common.py", line 15, in md5
with open(fname, "rb") as f:
PermissionError: [Errno 13] Permission denied: '/var/lib/docker/volumes/ebc90bbe11e5a8288e30325b613cf431db46dc0773b6b977764f6fbf820645a0/_data/media/media/Growing.Rep2_hicpro.mcool'
Which is weird because I'm already in the sudoers list. I tried sudo higlass-manage view Growing.Rep2_hicpro.mcool, but it gave me:
sudo: higlass-manage: command not found
Hey, sorry, this is poorly documented on our part.
There are a couple caveats:
- The media directory needs to be an absolute directory
- The file path needs to be relative to the media directory
This worked for me on OS X:
cp my_bigwig.bw data/
higlass-manage start --no-public-data --media-dir $(pwd)/data
higlass-manage ingest --no-upload my_bigwig.bw
I just created a PR to update the docs for the --no-upload option and to not require an absolute path for the media dir: #52
Hi!
I still get the following error:
FileNotFoundError: [Errno 2] No such file or directory:
'/home/rishi/hg-data/media/media/Growing.Rep2_hicpro.mcool'
I am not sure why the file path has media/media, because this is not a directory that I created :/
Okay, I don't know why this happened, but I just manually created media/media directories and now it works!
I have experienced the same OSError: [Errno 18] Invalid cross-device link: error when hg-tmp is on a different filesystem, and the commit referenced above is how I fixed it (works on Ubuntu 18.04). I'm not sure if the limitations of this approach are appropriate, or align with the desired behavior for higlass-manage ingest.
I suppose this is also indirectly related to #48