Tracking Issue for Stale Bot
dwijnand opened this issue · 22 comments
We are evaluating the use of the Stale bot, and this is the tracking issue to gather feedback on it.
I find this extremely annoying.
I really really really hate this stale bot.
I disagree with the very premise of this bot. An issue without recent activity often means that it hasn’t been prioritized, that doesn’t make it less of an issue.
What is the problem that this is trying to solve? PR #6020 that set this up gives no reasoning or discussion at all.
What is the problem that this is trying to solve?
There does not seem to be enough manpower to triage cargo issues, so this bot is basically pinging every issue to force the people that participated in them to triage them themselves.
This will probably result in issues being closed if those who participated in them got hit by a bus - just because nobody in the issue answers this bot does not imply that the problem the issue describes has been fixed :/
So I proposed using this bot. I realise this is a contentious issue (I believe Aaron mentioned it's been discussed in the past in the context of the Rust issue tracker).
My motivation is that in trying to help out resolving issues in the Cargo project I found myself often bumping into issues that just aren't ready for an implementation, often because its isn't clear what the issue is or what solution is desired, making it effectively un-actionable.
So I proposed using this bot to help close off some issues so that the issues that have the greatest interest can stay open, and those that are more niche don't clog up the backlog.
Now, I too disagreed with the premise of this bot, but have since changed my mind.
A closed issue doesn't mean the issue isn't valid, and a closed issue doesn't mean the issue was resolved. Simply, there isn't sufficient interest in the issue to keep it open at this time. However issues can also be re-opened and, indeed, the team has no problem with that.
Alex kindly accepted trialing this bot. I personally think that using this bot can be beneficial to a project. But every project and community is different, and this is just an evaluation. If you have any suggestions or recommendations for its usage, I/we'd be interested to hear them.
Issues should not be closed just because they're "niche" or low priority, especially when "niche" often is just whatever the people in charge don't care enough about. Issues should only be closed when they've been resolved. There are better ways to prioritize issues, whether via priority labels or meta issues or github projects or external lists somewhere.
There is a need to ensure old issues get triaged, and there should definitely be some sort of thing to automatically direct some sort of triage team at a rotating list of inactive issues to see what can be done to get them moving. Automatically closing issues is not triaging, it's just sweeping issues under the rug, and it definitely won't solve issues with a lack of manpower.
My motivation is that in trying to help out resolving issues in the Cargo project
If the scenario we want to improve is that of a contributor searching for something to work on, I think that a positive signal like the E-easy label is much better suited.
its isn't clear what the issue is or what solution is desired, making it effectively un-actionable.
Just because an issue isn’t immediately actionable doesn’t mean that there isn’t a real problem that should be resolved.
A closed issue doesn't mean the issue isn't valid, and a closed issue doesn't mean the issue was resolved.
I suppose this is where we fundamentally disagree. I view closing an issue as a statement that no further action will be pursued: either the issue is resolved, or we made a decision not to resolve it (even if that decision can be reversed later).
clog up the backlog
How do inactive issues “clog up” anything? What does that mean? Why is that a problem?
GitHub allows sorting issues by most recently updated. This way you can avoid looking at them, without closing them.
Perhaps automatically add a tag for inactive after x number of days of no activity on the issue (and automatically remove the tag on activity). It certainly shouldn't be closed though, closing an issue implies either it's resolved, or there's no intention for it to be resolved.
An alternative would be to just use P-low, P-medium, P-high labels to prioritize things. Everything that doesn't have a P-label, can be interpreted as P-lowest.
While I appreciate the need to keep the number of open issues down, I think this is a very bad idea. I work on another large open source project that uses a bot like this (Kubernetes). It results in lots of issues with the only activity being the annoying interaction being with this bot, e.g. kubernetes/kubernetes#59119 (comment) and kubernetes/kubernetes#58477 (comment). This is a terrible form of automation since it doesn't help resolve issues any faster or more efficiently. In fact, it makes it harder to figure out whether a particular issue has already been reported or solved.
I found myself often bumping into issues that just aren't ready for an implementation, often because its isn't clear what the issue is or what solution is desired, making it effectively un-actionable.
The number of issues without a clear actionable path is minimal. The vast majority of issues without activity are just not being triaged, prioritized, or otherwise being worked on. The solution is not to pretend those issues don't exist by closing them.
A closed issue doesn't mean the issue isn't valid, and a closed issue doesn't mean the issue was resolved.
In my mind this is exactly what a closed issue means. It's a signal to the reporter that the community doesn't care about their issue.
If a feature request ticket is closed, imo this often signals "the feature-idea was rejected by the team" or "the feature has been implemented already".
If a bug has been closed imo this means "the bug has been fixed" or "nobody was able to reproduce the bug in the current version of the program (somtimes there are wontfix or works-for-me tags).
If now old tickets are closed automatically and people search for features they want in cargo or bugs they struggle with, but all they find is tickets that are closed, the might not even report "hey, I got this old bug again, here is a way to repro" or "hey, I want this features as well, may I start working on it?" because the case was closes already...
I don't like having to comment on feature requests every N days to keep them from getting closed (maybe someone could setup an auto-comment bot...), of course I still want the feature, otherwise I would have closed it myself already.
If we have an issue with several "stalebot threatened to close this issue" comments and "bump"-comments after it, just to prevent the issue from being closed, I don't think this is going to help a lot...
A half-way compromise is possible: the stale bot adds the "stale" label and requests a summary, but never actually closes the issue. In some cases the original author will decide to close the issue.
It may also be worth adding support for commands, such as @stale close (with a 7-day delay) and @stale cancel.
I am 100% against the use of this bot. Closing issues does not make them go away, but it does guarantee that they won't be seen anymore. Issues getting old because there aren't enough person-hours to fix them doesn't make the issues invalid.
The only effect of this bot is that people who are contributing by reporting issues see an email, months later, that tells them nobody cares about their issue and it will never be fixed. It's impersonal and off-putting.
If we are suffering from low-quality issues, we could have a bot (or triage team) that posts a boilerplate comment asking for more details. If we don't have enough eyes on existing issues, maybe start asking for help in TWiR or something. But goosing the numbers by summarily closing issues is counterproductive.
Thanks all for taking the time to provide some feedback on this, it's definitely appreciated! We discussed this during the @rust-lang/cargo triage meeting yesterday, and I'm going to try to do my best to summarize the result of the discussion.
The main aspect that we concluded was that we view it as quite valuable to, in an automated fashion, poke issues to try to keep them moving forward. This is a great way for the team to be reminded about older issues (as well as contributors). This not only provides an opportunity for someone to realize that the issue has been fixed by something else but it also provides a great time to refresh the context to make sure the bug is up to date.
There was not consensus amongst the team about the usefulness of automatically closing issues if they had been stale for awhile. We all agreed, however, that many of the points brought up here were agree that we don't want to happen. For example we do not want to portray the message that if an issue is automatically closed that it will be forever closed. Furthermore we don't want to irritate ourselves or contributors by constantly fighting to keep a legitimate issue open.
With all that in mind, and in addition with the experience we've learned over the past weekend, we concluded that this is the steps we'd like to take at this time:
- We've updated the bot's configuration to slow down even more. Although 24 issues per day is as slow as the bot can go, we felt it was still too fast. We've increased the time it takes for an issue to be considered as "stale" (to three years) to only have a handful of issues be considered for being stale.
- We've also updated the amount of time given until an issue is automatically closed, now to 30 days. We're in no rush to close a stale issue, and we want to make sure that if an issue is closed that it truly is stale with very little interest!
- We're looking for feedback on wording of the message to clearly imply that the main purpose is poking the issue to help it move forward. We don't want the focus to be that this is closing and it's a race against the clock.
- We won't decrease the amount of time to the first poke by the bot until we can figure out how to execute at a slower rate than 24/day. This may require changes to the bot upstream, however.
- We'd love to get some feedback on labels that turn off the bot. For example right now all issues tagged as a feature request will not be poked by the bot. Are there other labels to add too?
Others from the @rust-lang/cargo team can feel free to chime in here too!
we don't want to irritate ourselves or contributors by constantly fighting to keep a legitimate issue open.
Maybe the lower frequency will make this less irritating, but it’ll keep happening.
@SimonSapin that's true, yes, but if truly nothing happens on an issue for literally years on end I think asking someone to say (like once a year) that the issue still exists is quite reasonable. If an issue has gone multiple years with only a bot and one person poking it then it's highly likely that it's entirely un-actionable its current form and needs to change somehow, and leaving it simply sit and collect dust won't actually do that.
- We've increased the time it takes for an issue to be considered as "stale" (to three years)
- We won't decrease the amount of time to the first poke by the bot
Sorry, I think I'm misinterpreting something but these seem contradictory... to clarify, how old will an issue get before the bot first pokes it?
Regarding tags that disable the bot, it seems to me that almost any tag should do it. That is, if a team member comes along to add tags and says "gee, this issue looks really unclear/unactionable/unreproducible", I'd imagine it should stay untagged or get a C-needs-details-plz tag right from the get-go. But yeah, I realize that's kind of idealistic and it's still quite possible for issues to become outdated.
I still think there should be a more human touch before closing issues, like maybe the bot can assign a random person to follow up like highfive does (I'd be on that ping list if you need volunteers).
@durka the exact amount of time is up in the air, currently it's set at three years. We won't make that shorter, though, until we can get the rate limits under control.
- We won't decrease the amount of time to the first poke by the bot until we can figure out how to execute at a slower rate than 24/day. This may require changes to the bot upstream, however.
I opened probot/stale#165 to track that. My initial attempt to fix that wasn't successful.
Given we can't get stale bot to operate as we'd like (poke up N older-then-X-period tickets a day) I'd say let's tear it down and close this ticket.
I think we should keep the tickets closed by the bot marked as stale.
@rust-lang/cargo WDYT?
Sounds good to me, thanks for investigating though!