Priority.IGNORE Piece Priority is reset to Priority.NORMAL
Closed this issue · 17 comments
Setting pieces priorities to Priority.IGNORE has no real effect, the ignored pieces get downloaded.
// Adding the torrent to download, setting all files' priority to `IGNORE`.
Priority[] filesPriority = new Priority[Objects.requireNonNull(getTorrentInfo()).files().numFiles()];
Arrays.fill(filesPriority, Priority.IGNORE);
Objects.requireNonNull(getSessionManager()).download(
mTorrentInfo,
saveDir,
null,
filesPriority,
null
);
And later in ADD_TORRENT alert I set All Pieces' priorities to Priority.IGNORE, except a few first and a few last pieces.
Priority[] piecesPriority = getTorrentHandle().piecePriorities();
// Ignore all pieces.
for (int i = 0; i < piecesPriority.length; i++) {
getTorrentHandle().piecePriority(i, Priority.IGNORE);
}
// Set priority for first ten pieces
for (int i = 0; i < 10; i++) {
getTorrentHandle().piecePriority(i, Priority.SEVEN);
}
// Set priority for last ten pieces
for (int i = (pieceCount - 1); i >= (pieceCount - 10); i--) {
getTorrentHandle().piecePriority(i, Priority.SEVEN);
}
However in PIECE_FINISHED alert when I check piece priorities of all pieces:
StringBuilder msg = new StringBuilder("All Pieces Priority: ");
Priority[] priorities = getTorrentHandle().piecePriorities();
for (int i = 0; i < priorities.length; i++) {
msg.append(i).append("= ").append(priorities[i]).append(", ");
}
Logger.d(TAG, msg.toString());
I see that all the pieces have either Priority.NORMAL for the ones which were set as Priority.IGNORE or they have Priority.SEVEN which is what I set for the first and last ten pieces.
Not only the priorities are reset to Priority.NORMAL on their own the ignored pieces start downloading, which defeats the purpose.
I was using 1.2.5.0 version, so I updated to the latest 1.2.7.0 version.
You might be onto something, I believe things changed, and I know that Priority in jlibtorrent is an enum we created.
Let's look at the most recent documentation from libtorrent:
prioritize_pieces() get_piece_priorities() piece_priority()
download_priority_t piece_priority (piece_index_t index) const;
void prioritize_pieces (std::vector<std::pair<piece_index_t, download_priority_t>> const& pieces) const;
void piece_priority (piece_index_t index, download_priority_t priority) const;
std::vector<download_priority_t> get_piece_priorities () const;
void prioritize_pieces (std::vector<download_priority_t> const& pieces) const;These functions are used to set and get the priority of individual pieces. By default all pieces have priority 4. That means that the random rarest first algorithm is effectively active for all pieces. You may however change the priority of individual pieces. There are 8 priority levels. 0 means not to download the piece at all. Otherwise, lower priority values means less likely to be picked. Piece priority takes precedence over piece availability. Every piece with priority 7 will be attempted to be picked before a priority 6 piece and so on.
The default priority of pieces is 4.
Piece priorities can not be changed for torrents that have not downloaded the metadata yet. Magnet links won't have metadata immediately. see the metadata_received_alert.
piece_priority sets or gets the priority for an individual piece, specified by index.
prioritize_pieces takes a vector of integers, one integer per piece in the torrent. All the piece priorities will be updated with the priorities in the vector. The second overload of prioritize_pieces that takes a vector of pairs will update the priorities of only select pieces, and leave all other unaffected. Each pair is (piece, priority). That is, the first item is the piece index and the second item is the priority of that piece. Invalid entries, where the piece index or priority is out of range, are not allowed.
get_piece_priorities returns a vector with one element for each piece in the torrent. Each element is the current priority of that piece.
It's possible to cancel the effect of file priorities by setting the priorities for the affected pieces. Care has to be taken when mixing usage of file- and piece priorities.
Our old Priority enum should still work:
public enum Priority {
/**
* piece or file is not downloaded at all
*/
IGNORE(0),
/**
* normal priority. Download order is dependent on availability
*/
NORMAL(1),
/**
* higher than normal priority. Pieces are preferred over pieces with
* the same availability, but not over pieces with lower availability
*/
TWO(2),
/**
* pieces are as likely to be picked as partial pieces.
*/
THREE(3),
/**
* pieces are preferred over partial pieces, but not over pieces with
* lower availability
*/
FOUR(4),
/**
* *currently the same as 4*
*/
FIVE(5),
/**
* piece is as likely to be picked as any piece with availability 1
*/
SIX(6),
/**
* maximum priority, availability is disregarded, the piece is
* preferred over any other piece with lower priority
*/
SEVEN(7);Will see what torrentHandle.piecePriority is doing, perhaps it's using a deprecated libtorrent API.
Thank you @abbasshah17 for bringing this to my attention.
Also, would you be interested in helping maintain FrostWire, we can pay for issues solved, between $50 to $100, perhaps more if you take on a bigger challenge. We can pay in crypto.
torrent_handle.piecePriority(piece_index, Priority) goes all the way down to this C++/JNI wrapper in libtorrent_jni.cpp:
SWIGINTERN void libtorrent_torrent_handle_piece_priority2__SWIG_1(libtorrent::torrent_handle *self,piece_index_t index,int priority){
self->piece_priority(index, download_priority_t{std::uint8_t(priority)});
}
Looks like it's using the latest source code with the download_priority_t type.
does this affect your test in any way @abbasshah17 ?
Piece priorities can not be changed for torrents that have not downloaded the metadata yet. Magnet links won't have metadata immediately. see the metadata_received_alert.
When are you changing the pieces priorities?
Piece priorities can not be changed for torrents that have not downloaded the metadata yet. Magnet links won't have metadata immediately. see the metadata_received_alert.
I am using online .torrent files so if I understand correctly there is no need to wait for METADATA_RECEIVED alert, however I went ahead setup an alert for METADATA_RECEIVED, turns out I'm not getting one, which is the correct behavior according to the java-docs:
METADATA_RECEIVED: It is not generated on torrents that are started with metadata, but only those that needs to download it from peers (when utilizing the libtorrent extension).
does this affect your test in any way @abbasshah17 ?
So it turns out ultimately, no.
When are you changing the pieces priorities?
I think I mentioned that in my OP, I'm changing the priorities in ADD_TORRENT alert, I have however tried setting the priorities in various stages all with the same result.
I will add this though: When checking pieces' priorities immediately after setting them via TorrentHandle.piecePriorities() I get the correct priorities.
It's possible to cancel the effect of file priorities by setting the priorities for the affected pieces. Care has to be taken when mixing usage of file- and piece priorities.
I find this part interesting; from my original post:
Priority[] filesPriority = new Priority[Objects.requireNonNull(getTorrentInfo()).files().numFiles()];
Arrays.fill(filesPriority, Priority.IGNORE);
What happens is when I go look for priorities of individual Pieces of these files they are still set to Priority.NORMAL.
In this gist, I'm ignoring all files here for the sake of clarity. However what I am doing is I looking for the largest file in the torrent and setting that file's priority to Priority.SEVEN and all others to Priority.IGNORE. Which, if I'm understanding correctly means that priorities of all pieces related to the file should also change, i.e. given that there are three files in the torrent, the second one being the largest with Priority.SEVEN, the first and last file's pieces should have Priority.IGNORE while all the pieces related to second file's pieces should have Priority.SEVEN. Is that correct? If so this is another bug.
would you be interested in helping maintain FrostWire, we can pay for issues solved, between $50 to $100, perhaps more if you take on a bigger challenge. We can pay in crypto.
If you'd only offered that a couple weeks ago I'd have jumped on it right away but as it stands I've my hands full, with this application (Which I really wanna just be able to release since I've been working on it for the last 2 years), my fulltime day-work and to cap it all I just joined a startup.
Perhaps if I can be done with at least one of them I would reconsider. Anyways, thanks for the opportunity and your consideration but I really have no the time or the mental capacity 😆 to take on yet another project.
These functions are used to set and get the priority of individual pieces. By default all pieces have priority 4
Where are you getting this? From TorrentHandle.piecePriority() in jlibtorrent-1.2.7.0-sources I see:
// These functions are used to set and get the priority of individual
// pieces. By default all pieces have priority 1.
Which actually translates to NORMAL, the default Priority.
I realize that the exact definition of these priority levels are subject to change from the same method's docs in sources. However I'd like to know where are you getting the information from for us to remain in sync.
@abbasshah17 You can ask your questions in the telegram https://t.me/libtorrent_ru.
@gubatron is @master255 part of the frostwire team? He is not in the contributor's page. If he is, the telegram URL isn't working.
not that I know of, just a community member.
@abbasshah17 In this post, I have come to the conclusion that there is no need to prioritize the pieces at all.
Maybe you can give me a scenario where piece priorities are really needed?
And yes, I'm not a contributor. But I'm writing my fork library with blackjack and streaming :-)
the telegram URL isn't working.
If the link does not work, you must first install a telegram and then find the group: libtorrent_ru in it.
@master255 Yes, I was aware of this (for the lack of a better word) hacky way to get around the issue temporarily, however the issue is still there. First off, logically a piece whose priority is set to IGNORE should not be downloaded at all. Now, by setting deadline to 0 and increasing priority to SEVEN will only discourage download of other pieces and only until all the prioritized or wanted pieces are not downloaded. As soon as they are, the rest of the pieces with deadline whatever and priority whatever will start downloading unless the torrent is removed from SessionManager at which point it will effectively stop seeding of the torrent. Which depending on the app may or may not be desirable.
If you have any other arguments against this let me know.
First off, logically a piece whose priority is set to IGNORE should not be downloaded at all.
We cannot replicate this. We use that priority to perform handpicked file downloads off of multi-file torrents and it's working perfectly fine. FrostWire will only download the pieces that haven't been set to IGNORE.
hacky way to get around the issue temporarily
I think it'll be a permanent solution. Using priorities for streaming is too much of a CPU load.
First off, logically a piece whose priority is set to IGNORE should not be downloaded at all.
Why would you prioritize the pieces for that? There is a file priority for that.
Now, by setting deadline to 0 and increasing priority to SEVEN
Setting deadline fully satisfies the priority of the pieces. There is no need to prioritize the pieces.
We use that priority to perform handpicked file downloads off of multi-file torrents and it's working perfectly fine
Does that mean the only reason for the piece priority to exist in the API is to facilitate file priority?
FrostWire will only download the pieces that haven't been set to IGNORE.
But that is not the case, if you set file priorities to anything but IGNORE and all pieces' priorities to IGNORE, all pieces will still download. As I pointed out the pieces' priority is auto reset to NORMAL. I could write a simple test for you?
I think it'll be a permanent solution.
@master255 maybe.
Using priorities for streaming is too much of a CPU load.
I'm not saying using priorities is better than setting deadline, just that it is part of the API. So either that API should work as expected or should not be present at all. Or maybe docs should be updated to reflect this behavior whether intended or not.
There is no need to prioritize the pieces.
Ah yes, that is what I meant, sorry for the confusion.
Why would you prioritize the pieces for that? There is a file priority for that.
Thank you for that. I thought file priority setting would then work with the piece picker, but I'm reading the storage code and so far no use of it. If this is the case, then my assumption that FrostWire is working fine is not a valid to say the code works as expected. The more reason we need a test.