ipfs-search

Search engine for the Interplanetary Filesystem. Sniffs the DHT gossip and indexes file and directory hashes.

Metadata and contents are extracted using ipfs-tika, searching is done using ElasticSearch 5, queueing is done using RabbitMQ. The crawler is implemented in Go, the API and frontend are built using Node.js.

Docs

A preliminary start at providing a minimal amount of documentation can be found in the docs folder.

Contact

Please find us on our Freenode/Riot/Matrix channel #ipfssearch.

Snapshots

ipfs-search provides the daily snapshot for all of the indexed data using elasticsearch snapshots. To learn more about downloading and restoring snapshots, read docs

Related repo's

Contributors wanted

Building a search engine like this takes a considerable amount of resources (money and TLC). If you are able to help out with either of them, mail us at info@ipfs-search.com or find us at #ipfssearch on Freenode (or #ipfs-search:chat.weho.st on Matrix).

Roadmap

For discussing and suggesting features, look at the project planning.

Requirements

Go 1.12
Elasticsearch 5.x
RabbitMQ / AMQP server
NodeJS 9.x

Configuration

Configuration can be done using a YAML configuration file, see example_config.yml.

The following configuration options can be overridden by environment variables:

IPFS_TIKA_URL
IPFS_API_URL
ELASTICSEARCH_URL
AMQP_URL

or by using environment variables.

Building

$ go get ./...
$ make

Running

Docker

The most convenient way to run the crawler is through Docker. Simply run:

compose up

This will start the crawler and all its dependencies but will not (yet) launch the sniffer or search API. Hashes can be queued for crawling manually by running ipfs-search a <hash> from within the running container. For example:

compose exec ipfs-search ipfs-search add QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv

Local setup

Local installation is done using vagrant:

git clone https://github.com/ipfs-search/ipfs-search.git ipfs-search
cd ipfs-search
vagrant up

This starts up the API on port 9615, Elasticsearch on 9200 and RabbitMQ on 15672.

Vagrant setup does not currently start up the frontend.

Ansible deployment

Automated deployment can be done on any (virtual) Ubuntu 16.04 machine. The full production stack is automated and can be found here.

Contributors

This project exists thanks to all the people who contribute.

Backers

Thank you to all our backers! 🙏 [Become a backer]

xdengpao/ipfs-search