Little project that helps copying/backing up data from HTTP sources to any kind of destination supported by rclone.
The basic idea is to fake a directory listing similar to nginx' that rclone can used as a http remote. When a file is requested by rclone the response will be a 301 redirect to the real location of the file.
The directory listing will be created from the files specified in the input file, there are several options that can be used to customize the resulting directory index. This includes the ability to use different filenames and paths than what is present in the original URL. This is possible since rclone will use the name and path of the link rather than what it gets redirected to.
Dependencies:
- Python 3 (written in a 3.6 env, not tested with anything below)
- Flask
Command line options
$ python3 app.py -h
usage: app.py [-h] [-f FILE] [-i HOST] [-p PORT] [-k]
optional arguments:
-h, --help show this help message and exit
-f FILE, --file FILE Input file with JSON list of files to include in fake
index. (Default: files.json)
-i HOST, --host HOST Host to listen to, e.g. if rclone is run on a
different machine (default: 127.0.0.1)
-p PORT, --port PORT Port to listen on (default: 8000)
-k, --keep-path Keep path in filename if only url is supplied
(default: false)
-l LIMIT, --limit LIMIT
Maximum number of items to show in index (for testing)
-o OFFSET, --offset OFFSET
Offset for items to show in index (for testing)
Run flask app using python
$ python3 app.py
In order to add it as a remote to rclone you can now use rclone config
and create a new http remote,
alternatively you can edit your rclone.conf
directly.
Example rclone.conf
entry:
[http]
type = http
url = http://127.0.0.1:8000
Now you can list the files in the remote using rclone:
$ rclone ls http:
104857600 0.1GB.bin
104857600 100MB.bin
10485760 10Mio.dat
10485760 files/10Mio.dat
104857600 tester/test2/0.1GB.bin
104857600 test/0.1GB.bin
104857600 test/filename w_ slash.bin
104857600 test/test/0.1GB.bin
Note that rclone will make a HEAD request for any item in the directory, if you have a lot of files this might take some time depending on how fast the origin server responds.
The basic format is a JSON array where each item represents a file in the listing.
Each item can have the following properties:
url
(required)keep_path
filename
directory
When only url
is set and keep_path
or the cli parameter --keep-path
are not true
/used only the last part of the
URL (i.e. everything after the final /
) will be used as the filename and the file will be in the root.
Otherwise the path in the URL will be replicated in the fake index as well.
filename
overrides the filename that will be used in the listing and subsequently in the rclone target,
if it contains slashes those will be used for the file's path in the listing. It should not contain any path when used
together with the directory
property.
directory
specifies the path to the file that should be used. Note that if the filename already contains a path
(or any /
) it will be replaced with underscores and the result used as filename instead.
Example:
[
{
"url": "https://speed.hetzner.de/100MB.bin"
},
{
"url": "http://www.ovh.net/files/10Mio.dat"
},
{
"url": "http://www.ovh.net/files/10Mio.dat",
"keep_path": true
},
{
"url": "https://speed.hetzner.de/100MB.bin",
"filename": "0.1GB.bin"
},
{
"url": "https://speed.hetzner.de/100MB.bin",
"filename": "test/0.1GB.bin"
},
{
"url": "https://speed.hetzner.de/100MB.bin",
"filename": "test/test/0.1GB.bin"
},
{
"url": "https://speed.hetzner.de/100MB.bin",
"filename": "tester/test2/0.1GB.bin"
},
{
"url": "https://speed.hetzner.de/100MB.bin",
"filename": "filename w/ slash.bin",
"directory": "test"
}
]