Design for the Worst People
Opened this issue · 2 comments
Hi, sorry if this isn’t the preferred way to raise concerns, but I didn’t see a contact address. I want to very, very strongly encourage you to consider from the outset the ways people can use this in horrific ways, and either find ways to prevent that, or else abandon the idea entirely.
When I say horrific ways, here are just a few scenarios — there are many, many more that could be posited.
- A company site is targeted by a vengeful ex-employee.
- An ex-employee’s site is targeted by a vengeful company/boss.
- An abusive stalker finds out the focus of their obsession has an anonymous blog. They immediately start annotating it with attacks, threats, etc.
- A political commentator comes to the attention of the worst of 4chan/8chan/whatever, which immediately starts a brigade to annotate their sites/articles/etc. with doxxing information, verbal abuse, links to or images of child pornography, and more.
- What will very likely happen to the web sites of political figures, especially those who are especially divisive, or are seen to be by their opponents.
- See previous, but for political dissidents.
It’s not enough to say “site authors can just turn off the annotation endpoints”: they may use a platform in the style of Wix or Substack, one which might automatically enable annotations without notice, and without making it possible/easy to disable them.
Furthermore, a site may persist long past an author has access to it. Think of all the Blogspot and LiveJournal accounts that were available for years after the creator lost interest. Annotations might be enabled in that post-interest phase, and then become the target of a brigade, with nobody available to deal with them.
None of these should be problems if annotations are entirely private in nature; that is, if annotations are notes a user can make for themselves that only they will ever see. As soon as you open this to others being able to see annotations, whether that’s the site maintainer or other users of a site or both, the social attack surface becomes enormous, and there is risk of real human costs, up to and including lives.
Hi Eric, thanks for the response!
Hi, sorry if this isn’t the preferred way to raise concerns, but I didn’t see a contact address
Whoops - yes, issue was the intended way but I realized I didn't leave any directions. Amended the README, thanks for bringing it up.
consider from the outset the ways people can use this in horrific ways
We definitely are and this is our #1 concern we're thinking about (fitting that it's issue #1) and why we're approaching this so slowly and carefully. In addition to the more social oriented issues you list - there's also a big risk here in adding to attack vectors for fraud and coercion. IMHO, there isn't really a way to get to fully public annotations - where a given page has a canonical set of annotations seen by all users; nor should we be aiming for that.
I guess my personal perspective is that people can already write mean things in any number of ways; if I publish a page - a vengeful nemesis can publish their own page filled with hateful things about me, or shout abuse at me on Twitter. Please correct me if I'm characterizing, but I think your point is that, right now, the latter case is mostly people shouting into the void and annotation makes this qualitatively worse because now it will be co-located with the former so you no longer have to go looking for it? If so, I agree but I think it's possible to structure it so that we're not giving a megaphone (or at least, not a new megaphone) and prominence to bad actors without being entirely private.
I would suggest that there is a spectrum between "entirely private" and "entirely public" where we might be able to strike an acceptable balance. Some ideas we've been bouncing around that I think would alleviate this to some degree:
- Ephemeral but shareable annotations where the annotation lives entirely in the URL - you can share a note with a friend or put it on a public page you control (where you could already spew vitriol), but only people visiting that page from your reference to it see the annotations.
- Opt-in communities - maybe you see annotations regardless of the referrer, but only from a curated list of people. e.g. I might want to see annotations but only from my small social circle, or, in a work context, only from my colleagues.
I don't think these necessarily solve all these problems but (IMHO) they mitigate them significantly.
It’s not enough to say “site authors can just turn off the annotation endpoints”: they may use a platform in the style of Wix or Substack, one which might automatically enable annotations without notice, and without making it possible/easy to disable them.
Furthermore, a site may persist long past an author has access to it. Think of all the Blogspot and LiveJournal accounts that were available for years after the creator lost interest. Annotations might be enabled in that post-interest phase, and then become the target of a brigade, with nobody available to deal with them.
Thanks, there are all great points I hadn't considered - I would hope platforms like this would fairly quickly adapt but the point is well taken that not all of them may be responsive or well maintained.
bokand wrote:
MHO, there isn't really a way to get to fully public annotations - where a given page has a canonical set of annotations seen by all users; nor should we be aiming for that. (emphasis added)
While you may not aim to provide for "fully public annotations," you should avoid doing anything which is explicitly intended to prevent them. As we often find in software development, attempts to prevent "bad" things often have the unintended consequence of preventing "good" things.
As you suggest in your summary, a variety of annotation sources should be supported and users should have the ability to choose between them. Given this, we should recognize that some annotation source provider might choose to implement an annotation source that provides "fully public annotations." While there are all sorts of issues that would arise, those issues are not properly in the realm of an in-browser annotation display and editing interface -- in the same way that browsers should not be expected to, or even allowed to, impose content moderation for HTML pages unless the user has explicitly taken action to enable that content moderation.
One way to provide mitigation of some of the issues with "public" annotations, and others, is to ensure that annotations can themselves be annotated. (Some might be displayed as a comment on or reply to annotation, such as what Hypothes.is supports today.) Additionally, it should be possible to attach an annotation to the author of an annotation. Of course, we shouldn't expect that all annotation sources would support this, so, it would be necessary to allow the user to specify which annotation source should be used as a store and source for such "meta-annotations."
Allowing "meta-annotations" provides a means by which signals concerning the credibility of annotations and their authors can be made available. For instance, if I'm a member of a Society of Chemical Engineers, and I'm using that society's annotation server to comment on papers concerning chemistry, it should be possible for that server to make available the fact that I'm a society member, that I have a Phd, and possibly even what my specialty is. (Note: I am not a chemist.) Thus, readers of my annotations concerning chemistry would be able to form some sense of the credibility that they may associate with the content of my annotations.