This is the project that drives the FediverseAlmanac, a free and open source API for the Fediverse. It has two components- a web crawler that ingests data from all of the Fediverse instances it can find, and a Web API that exposes that data for use.
This API exposes a lot of fun information-
- General instance metadata.
- Software and Versions on the Fediverse, including instance lists.
- Network Data, again including instance lists based on either Network ID or Company.
- Lists of the most banned instances in the Fediverse.
- Which instances ban or are banned by other instances.
- Subdomain Clusters.
The Fediverse needs an API, and I like creating crawlers. The hope is that an easy to use API for the Fediverse will allow people to build tools on top of it- the idea is to reduce the barrier of entry by making it so people do not have to build and run their own web crawlers.
Have an instance and don't want us crawling it? Block this bot with your robots.txt! We use two separate user agents- fedimapper is used by the development version and anyone who clones this repo, while the FediverseAlmanac uses FediverseAlmanac for its user agent. To stop both add this to your robots.txt file.
User-agent: fedimapper
Disallow: /
User-agent: FediverseAlmanac
Disallow: /
Just want to stop us from reading your block list, but otherwise are okay with metadata being included? Update your robots.txt to forbid crawling of your domain blocks endpoint. You can enable this for all crawlers (although I can't promise others will honor it).
User-agent: *
Disallow: /api/v1/instance/domain_blocks
This project is developed and hosted by Robert Hafner, who currently pays for it out of pocket. If you find this project useful please consider sponsoring me using Github!
This project uses several upstream projects and data sources.
- mastodon.social and diasp.org are used to bootstrap the network.
- Alir3z4's Stop Words is used to remove noise when processing ban comments for keywords.
- cymru whois is used for network data.
If you use the data in from this site please provide attribution.