Fix partially done downloads and choke groups in git version
chros73 opened this issue · 1 comments
Choke groups
I. Summary
This is a powerful feature that should be used if you deal with group of torrents (e.g. you use throttle
groups) like here.
It allows to set 3 major property per group (2 others (up, down) per property, except for tracker.mode
):
choke_group.tracker.mode
: decides on how aggressive a tracker should bechoke_group.up.heuristics
: set the heuristics used when deciding on which peers to choke and unchokechoke_group.up.max
: set the max total number of unchoked peers for all torrents in this choke group
The last property is the really interesting one! It restricts the number of unchoked peers in a group!
I just found it by accident (while I have been working on fixing partially done donwloads) that dealing with choke heuristics has been dramatically changed since libtorrent v0.13.0 but rtorrent (client) part left untouched since then (last good version is 0.8.9-0.12.9 in this regard).
- until 0.8.9 it worked as a per download setting
- since 0.9.0 it should work as a group setting
- choke groups feature is enabled and working (means it's actively used by resource_manager) in
libtorrent
since v0.13.0- only 1 group is created, called
default
- only 1 group is created, called
rtorrent
hasn't been modified (just a bit to don't get broken)- if you want to modify group for a download then it affects all the downloads
- only 1 group is used (
default
) for all the torrents, meansupload_leech
heuristics is used for seeding as well - setting
choke_group.up.max
doesn't work
It works fine by default until somebody wants to change the config - there's a default
choke group - (a hack was applied for integrating into the client).
II. Bugs
-
- all the old
choke_heuristics
code is unnecessary in command_download.cc
- all the old
-
- there are unnecessary code in resume(), confirm_finished() part of
core/download_list.cc
- there are unnecessary code in resume(), confirm_finished() part of
-
protocol.choke_heuristics.up.seed
hasn't been updated toupload_seed
in command_network.cc , but I think it's not needed anymore
-
- the new
d.group
property isn't saved and not initialized with the download in core/download_factory.cc , old code can be removed
- the new
-
d.group.name
command gives back index instead of the name of the group
-
- UI only displays the index of the group not the name in ui/download.cc
-
- all the old
choke_heuristics
code is unnecessary in libtorrent@torrent/download.h and libtorrent@torrent/download.cc
- all the old
III. Fixes for mentioned bugs
-
- removed old code from
command_download.cc
- removed old code from
-
- added new logic and removed old code from
core/download_list.cc@resume()
andcore/download_list.cc@confirm_finished()
- added new logic and removed old code from
-
- removed old code from
core/download_list.cc
- removed old code from
-
- removed old code from
core/download_factory.cc
- removed old code from
-
- fixed
d.group_name
incommand_download.cc
- fixed
-
- fixed and modified UI on Info screen in
ui/download.cc
- fixed and modified UI on Info screen in
-
- removed old code from
libtorrent@torrent/download.*
- removed old code from
-
- enabled real choke group code in
command_groups.cc
and got rid of the hack code (also incore/download.h
)
- enabled real choke group code in
IV. Additions
- renamed default choke group
default
todefault_leech
in libtorrent@manager.cc - added a new default choke group called
default_seed
inmain.cc
- added new helper UI command
convert.group
has been created incommand_ui.cc
(similarly toconvert.throttle
) - modify UI on Info screen to include all the statistics that belongs to a group
Choke group: default_seed [upload_seed, download_leech, aggressive] [Max --/--]
Choke group stat: [Size 149] [Unchoked 18/0] [Queued 0/0] [Total 18/0] [Rate 952.3/ 0.0 KB]
V. Other fixes
- fixed saving
total_skipped
between sessions incore/download_factory.cc
andcore/download_store.cc
(used on Info screen)
VI. Choke groups in action
I'll update the wiki page if it will be merged, but until then here's a teaser:
let's extend the Favoring group of torrents example to include public
(!!!) torrents as well! :)
- we add one more throttle group for public torrents, called
tardyup
, along withslowup
- let's create 4 more choke groups (
slowup_leech
,slowup_seed
,tardyup_leech
,tardyup_seed
) along with the built-in 2 ones (default_leech
,default_seed
)- name them after the corresponding throttle groups!
- assign the different groups to the proper download with the corresponding throttle group all the time!
- it's just 4 lines of code when the above naming convention is used! :)
- we can also modify the
choke_group.up.max
setting on the fly (!) with a help of an external script (like we did with thethrottle
speed)- external script mod isn't included, so this part work work in this way
# Define throttle groups
throttle.up = slowup,600
throttle.up = tardyup,300
# Setting up choke groups that restricts the number of unchoked peers in a group
# Modify default choke groups for specail group
choke_group.up.heuristics.set=default_leech,upload_leech_experimental
choke_group.tracker.mode.set=default_leech,aggressive
choke_group.tracker.mode.set=default_seed,aggressive
# Set up choke groups for slowup group
choke_group.insert=slowup_leech
choke_group.insert=slowup_seed
choke_group.up.heuristics.set=slowup_leech,upload_leech
choke_group.up.heuristics.set=slowup_seed,upload_seed
choke_group.down.max.set=slowup_leech,200
choke_group.up.max.set=slowup_leech,200
choke_group.up.max.set=slowup_seed,125
# Set up choke groups for tardy group
choke_group.insert=tardyup_leech
choke_group.insert=tardyup_seed
choke_group.up.heuristics.set=tardyup_leech,upload_leech
choke_group.up.heuristics.set=tardyup_seed,upload_seed
choke_group.down.max.set=tardyup_leech,150
choke_group.up.max.set=tardyup_leech,150
choke_group.up.max.set=tardyup_seed,75
# helper method: Sets choke group to one of the default ones if there's no throttle or throttle is special one (NULL) otherwise sets it to one of the throttle name ones
method.insert = d.modify_choke_group, simple, "branch=((and,((d.throttle_name)),((not,((equal,((d.throttle_name)),((cat,NULL)))))))),((d.group.set,(cat,(d.throttle_name),_,(d.connection_current)))),((d.group.set,(cat,default_,(d.connection_current))))"
# Modify choke group when a torrent is resumed (even after hashchecking or after rtorrent is restarted) or finished or partially restarted
method.set_key = event.download.resumed, resumed_choke_group, "d.modify_choke_group="
method.set_key = event.download.finished, finished_choke_group, "d.modify_choke_group="
method.set_key = event.download.partially_restarted, partially_restarted_choke_group, "d.modify_choke_group="
# helper method: gets one of the below info with the help of getLimits.sh script (Variables are inside the script, you have to edit those values there!)
method.insert = get_limit, simple|private, "execute.capture=\"$cat=$pyro.bin_dir=,getLimits,$cfg.postfix=,.sh\",$argument.0=,$argument.1=,$argument.2=,$argument.3="
# Adjust max setting on the fly
schedule2 = adjust_slots_slowup, 47, 100, "choke_group.up.max.set=slowup_seed,\"$get_limit=$cat=slslots,$convert.kb=$throttle.global_up.rate=,$convert.kb=$throttle.up.rate=slowup,$convert.kb=$throttle.up.rate=tardyup\""
schedule2 = adjust_slots_tardyup, 48, 100, "choke_group.up.max.set=tardyup_seed,\"$get_limit=$cat=taslots,$convert.kb=$throttle.global_up.rate=,$convert.kb=$throttle.up.rate=slowup,$convert.kb=$throttle.up.rate=tardyup\""
Partially done downloads
Let's try to fit partially done downloads into the ecosystem of rtorrent:
libtorrent
already has some code related to this state, e.g. in torrent/data/download_data.h , but it hasn't been really usedrtorrent
already has commands for partially done downloads:d.is_partially_done
,d.is_not_partially_done
- every finished download is also partially done!
I. Fixes for known bugs
- it will operate entirely based on selection of files (that's the only thing that makes sense from the user point of view)
- let's use partially done state of a download most of the time instead of finished (when it makes sense)
- release seeds when a download is partially done in libtorrent:
torrent/peer/connection_list.cc
,protocol/peer_connection_leech.cc
,protocol/peer_connection_base.cc
,protocol/handshake_manager.cc
- let's trigger
event.download.finished
inrtorrent
when a download is partially finished - define a new slot and a corresponding method in core/download_list.cc
slot_partially_restarted()
that will handle client part (also intocore/download_list.h
)- reset choke groups,
d.complete
,d.connection_current
,d.peers_min
,d.peers_max
- it will trigger a new
event.download.partially_restarted
event
- reset choke groups,
- don't send complete info to trackers only if a download is truely finished
- create a new
event.download.partially_restarted
event that is triggered when a partial download has been restarted- e.g. it can be used to assign custom choke groups to downloads
- add this new event to relevant view filters in main.cc
complete
,incomplete
,seeding
,leeching
- modify
d.timestamp.finished.set
method in main.cc to be able to be overridden - create a new porperty and setter method in
libtorrent@torrent/data/file_list.cc
forselected_size_bytes
- will be equal to
size_bytes
if a download is finished - will be equal to
completed_bytes
if a download is partyally done (and not the slected size of files, since they can be turnd off later!) - otherwise calculate the partial size based on the selected chunks of the selected files
- it's a property in
File_list
: to don't fire up this expensive calculation all the time - save it into session and reload it during restart (similarly to
chunks_wanted
):- required by stopped torrents (
update_priorities()
method isn't triggered for them upon start)
- required by stopped torrents (
- will be equal to
- release seeds when a download is partially done in libtorrent:
- it should not break remusing downloads
- it should not break fast resume data
- it should not break hash-checking of a download
II. New commands in rtorrent
d.is_done
: to be able to distinguish between finished and partially done downloadsd.selected_size_bytes
: gets the correct size of a torrent
III. UI changes in rtorrent
- don't change the % , displayed total size (we want to differentiate between partially done and finished downloads), but display
done
in the place of remaining time if a download is partially done13.6G / 22.8G Rate: 0.0K / 0.0K Uploaded: 762.4M [59%] done [T R: 0.05 low tardyup]
- on the new compact view: don't change anything (there's no place for it), if a donwload isn't 100% and there's no remaining time that means it's partially finished
- display additional Size info on Info screen: downloaded / selected / total size:
Size: 450MB / 712MB / 970MB
- modify Chunk screen also to take into account partial downloads
- remaining time should display the correct amount all the time