Testing federation / realms?
kindrowboat opened this issue · 23 comments
Hello again! 👋
Would you be interested in testing realms by federating our servers? I realize this part of the spec is still pretty fluid, and the discoverability of realms will probably change, but nothing would stop our two servers from gossiping with each other. :)
Hi @motevets! 👋
I would love to start experimenting on this, absolutely! In fact @farktronix and I were just chatting last night that this really needs to be the next step even if it's still a bit ambiguous in the spec. If anything, I feel that experimenting with a proof-of-concept will help us work out the rough edges in the spec and inform us further.
Unfortunately, it being the work week again I will likely have little time until the weekend to really dig into this. Don't let that stop you from getting going though! I've invited you as a collaborator here and would love to have you on board properly as we make strides on this.
Thank you so much for bringing this up, I can't wait to get this going! 🌸 ❤️
Sounds good. I think for the first step, I'll have my server send boards it receives to your server. There isn't much (anything?) that distinguishes a PUT /:key from a user and from a peer (at least from the receiving server's perspective). There shouldn't be any changes needed to your server for this step. After that I can look into implementing something similar for your server (unless you beat me to it!)
For motevets/s83, I was planning on starting a config file (probably YAML) for the server with a something like
federation:
- https://0l0.lol
- ...I was also planning on responding to GET /federation.txt and serve a simple text file with federated members:
https://0l0.lol
...
I'm curious to know what you all think the benefits are of sharing boards between servers -- other than the pure fun of getting pings on your server which is, of course, not insubstantial!
In general, I like this feeling of "board flow", of a server "subscribing" to boards across a network/realm/whatever, but, for me it's nearly 100% about the (extremely theoretical) vision of being able to implement new "features" using that data: Spring '83 analytics, recommendations, a search engine, who knows.
I'm curious to know if there are other objectives/benefits, obvious to you, that I'm missing.
@robinsloan As a user of S83, federation has one significant objective benefit (in my mind): PUT to one server and (eventually) find your board on all (federated) servers, with no extra work on your part. Basically you chose a "home" server in the knowledge that it will forward along your content to the wider Spring '83 "world".
On the flip side (and with significantly less complexity): I have multi-board push already in the putnew tool and other clients could similarly implement it. This would also allow users direct control over where their board(s) end up. (@farktronix and I have also bandied about the idea of a "do-not-publish" header/meta tag that would serve a similar function in a federated model).
So is the complexity cost of federation worth this benefit alone? Not sure about that.
As a maintainer of an S83 server, honestly federation will probably cause more trouble than the benefit it brings my users. Just a gut feeling there (I worry as well about "thundering herds" created by mass PUTs as someone noted in their notes, for example).
I am interested in federation because:
- it keeps the system decentralized (no one person is a gatekeeper what gets out there)
- it keeps the system resilient (if the server I post to goes down, people who are following me can hopefully get it from somewhere else)
- it (might?) increase access to those who face political censorship or persecution (e.g. a large enough federation can provide resiliency and anonymity for political dissidents)
The flipped paradigm that @rpj has implemented its interesting. You can get most of the benefits of federation "for free," though maybe not anonymity, and it might exclude folks who are on a metered/limited connection.
I wonder if a concept of "board mirroring" -- encouraging publishers to send their boards to more than once place -- might accomplish many of these goals with a simpler architecture. Another way of saying it might be: clients federate, rather than servers.
My concern is that if the federation doesn't work 100% correctly, then it's not actually possible to request a board from a random server with any confidence that you'll receive what you want, and you are back to requiring a reliable/canonical source.
Also, I am rethinking my original notion that every server stores every board; that demand might not fit every server operator's vision for "what it is they're doing", or want to be doing.
This is just my opinion, of course!
Imagining a federation scheme that is "partial", i.e. cannot make the guarantee that every board will be available on every server, what benefits might still exist? I think there probably are some...
(I'll add that I'm interested in publishers specifying a "backup" location in their boards. This would be related to the approach to link rel=next tags discussed in the latest draft spec.)
I was thinking more today about federation from a moderation standpoint. Here's an exerpt from my journal. (Sorry the formatting's a little weird.)
Mastodon gets some group decentralized moderation benefit by
1. requiring members to be invited to a Mastodon instance
2. federates blocking servers that don't moderate or encourage bad behavior
While Spring '83 doesn't have any mechanism to block bad actors from posting
boards or even spewing out harmful (e.g. hate speech) boards to every known
spring server, federation provides a unique opportunity to allow good
distributed moderation practices.
Lets say there are two federated servers: Alice and Bob, and let's say there's a
malicious actor, who Dr. Evil who has been posting to Bob's server, but Bob is
on vacation and hasn't noticed yet. Alice notices Dr. Evil's boards are harmful
and blocks them from her server. The next time Dr. Evil posts to Bob's server
and the server tries to propagate it, Alice's server responds to the PUT with
403 Forbidden, because posts by that key are forbidden on Alice's server. Bob's
server notices the 403 and /may/ chose to also block Dr. Evil from his server.
If Bob does a poor job of moderating his server or encourages bad behavior,
Alice may decide to un-federate and block Bob's server from propagating to her
server. (Though this would require that /servers/ had their own asymmetric keys
and that they countersigned boards that they were propagating. Otherwise Alice's
server doesn't really know where propagated boards are coming from.)
Apologies for the lack of response here, work has been crazy this week.
@motevets the additional benefits you bring up for moderation & resiliency that federation would provide I hadn't fully considered and can definitely concur with. That certainly bolsters the case for adding the complexity necessary. And to that point, @farktronix and I had bandied about a scheme for S83 host public key signing that we think would work here well. So regardless what the eventual choice is, I'm confident that technically we can accomplish it.
Thus far though, it does seem like the board mirroring method is working pretty well, especially with bonafide support in the new client (🙌❤️🙇 @robinsloan). Maybe we just let that usage model soak for awhile and revisit if/when it starts to falter?
@motevets Oh yes indeed, please don't let this discussion discourage you! If anything we're still too early on to really know what path we'll take, so having all the options explored well will absolutely be valuable. And I'm a big believer in building proof-of-concepts to fully and comprehensively explore options, so I'm all for it. Thank you for working through it! ❤️
Hammer away! 💻 🔨 😆 More than happy to extract logs from the 0l0.lol server instance if you need to dig into anything as you experiment (though I figure you'll probably try it locally first, but don't be dissuaded from hitting the live 0l0.lol either: only one way to fully test things end-to-end!)
Heya,
Just a little update. I've started propagating posts sent to https://spring83.kindrobot.ca to https://bogbody.biz (cc @robinsloan) https://0l0.lol (cc @rpj) and https://spring83.mozz.us (cc @michael-lazar). If you'd rather me not propagate posts to you, let me know, I'll gladly stop.
Some thoughts I had on federation/realms/propagation so far are in my commit message here.
@motevets Stellar work, I'm seeing boards from you on 0l0! 🥂
From your commit message:
It might make sense to add the "Via" HTTP header when a server is propagating a board. That way, the receiving server knows not to try to propagate it back to the sending server.
Heartily agree here. I'd say toss it in there now, that way I can key on it when I start implementing federation and we'll avoid any loops off the bat. Obviously I'll make sure to include it here as well. (Or whomever gets to that, though you've inspired me so I want to try to get it working ASAP! ❤️ )
OH! I just realized that the Via header also gives us implicitly a behavior I was otherwise going to suggest we need: a "do not federate" field. This way publishers can decide when posting their board initially if the server should federate it for them. I'm thinking of stuff like bot boards, where the frequency of updates might cause issues with federation. Also it just isn't needed in that use case. With Via, simply the existence of the header will be enough to skip federation: this way bots/clients can just set it to something like Via: 1.1 do-not-federate (e.g. it needn't be a resolvable domain name) to request this functionality. 👍 💯
@rpj Just a thought: remember that metadata is only "durable" if it's included in the board HTML that gets signed. In the Via header scenario, a malicious server could snag a board and transmit it to other servers, simply omitting the header.
If, by contrast, a board included something like (just making it up) --
<meta name="spring:share" content="false">
-- then the preference would travel with the signed board everywhere, tamper-proof. Of course, servers could ignore it, but that requires a whole network of malicious servers, rather than just one.
(I think there are a lot of interesting possibilities for these kinds of "board preferences" generally.)
@robinsloan excellent catch! 💯 you're absolutely right: this only works safely if that metadata is signed. Of course including metadata in the boards has the drawback of costing bytes, so we could potentially instead sign the Via header (and/or any other metadata headers) with (here we are again) server key pairs...
I really do like the thought of fleshing out further the "board preferences" idea in whatever guise though, I feel like we'll absolutely want, maybe even need, this moving forward.
@rpj I think -- or, hope -- the board preferences thing is going to be an organic process, in which publishers devise ideas for new fields, thinking of clients as the initial "targets" probably, but/and servers soon after. I think it could be really fun & interesting. That's a little way down the road still.
And, costing bytes isn't a drawback; it's part of the game 😉😈
@robinsloan haha so true, and what a fun game it has been so far!
It might not be as far down the road as you might think either! I like the tag idea a lot, thinking I'll add support for it in the branch for this I'm working on today. Will be really nice to encode a tamper-proof directive to the server right in the signed body. 🎉🤯
The Via header can remain also in this scenario, if anything for tracking & logging purposes. (It already requires a good-faith effort on the sending server's side, so is just a nice-to-have now)
@motevets I have federation working well between my main and dev instances, would you mind if I added your host to the list to start testing against yours as well? As discussed above, it will not federate if the Via header exists or the board contains the <meta name="spring:share" content="false"> tag.
Awesome thank you @motevets, it's enabled! Please do let me know if you see any weirdness at all. Excited to see our "realm" come to life! 🙌
LOL dammit, I knew I'd forget something! I forgot to disable federation on my "Wine Country Weather" board before enabling federation. 🤦
But there are actually two silver linings because of that:
- It validated that it worked correctly, as it's on your host now, and I did not change which host(s) the bot posts to
- It will offer a great production validation on your host of the soon-to-be-released
DELETErequest 😆
I've now disabled sharing for board b03d3fca85920de9124b7e749061f03d23bf9bb403af634d5928abe3d83e0523 ("Wine Country Weather") by adding the meta tag. Unless something is broken, the last federated version of it posted anywhere other than 0l0.lol should be timestamped "Sun Jun 26 18:44:01 PDT 2022".
Great! I'll work on using the Via header and meta tag sometime this week.