/zlib-searcher

search zlib/libgen index to get ipfs_cid. We don't save and provide files, we provide search. I hope everyone have a copy of the index locally, so that no need to rely on any centralized service.

Primary LanguageRustMIT LicenseMIT

zlib(libgen) searcher

GitHub stars GitHub forks Release GitHub issues GitHub license

Search zlib/libgen index to get ipfs_cid.

We don't save and provide files, we provide search.

I hope everyone have a copy of the index locally, so that no need to rely on any centralized service.

Usage

1. Download the pre-compiled binary from Release.

Or you can compile by yourself. Refer to Build from source for instructions.

2. Download the index file that has been created.

We will give the corresponding index download links for each version in the release page.

Or you can make your own via bin/index.rs.

Extract the index folder to the same level as the program, it should look like the following:

zlib_searcher_dir
├── index
│   ├── some index files...
│   └── meta.json
└── zlib-searcher

3. Run zlib-searcher, it will listen to 127.0.0.1:7070.

Access http://127.0.0.1:7070/ to use webui, or you can use the original api.

original search api

You can search by the following fields:

  • title
  • author
  • publisher
  • extension
  • language
  • isbn
  • zlib_id

Examples:

  • http://127.0.0.1:7070/search?limit=30&query=余华
  • http://127.0.0.1:7070/search?limit=30&query=title:机器学习 extension:azw3 publisher:清华
  • http://127.0.0.1:7070/search?limit=30&query=zlib_id:18557063
  • http://127.0.0.1:7070/search?limit=30&query=isbn:9787302423287

Build from source

1. Build zlib-searcher

You need the frontend code in the frontend branch, and organize it like this:

├── zlib-searcher // this repo
│   └── src
└── zlib-searcher-frontend
    ├── dist   // this is what we need.
    ├── public
    └── src

Then run

cd ../zlib-searcher-frontend
npm install && npm run build

cd ../zlib-searcher
cargo build --release

# move the compiled binary to the project root directory
mv target/release/zlib-searcher .

2. Build index

Download zlib_index_books.csv.zip and libgen_index_books.csv.zip and extract the csv files to the project root directory.

Then run cargo run --bin index --release. You may need to mkdir index or rm index/* first to make sure index is an existing empty folder.

The finally folder structure should look like this:

zlib_searcher_dir // in the example above, it is project root directory.
├── index
│   ├── some index files...
│   └── meta.json
└── zlib-searcher

Raw data

We downloaded libgen sql and zlib sql and exported the necessary data from them.

id, title, author, publisher, extension, filesize, language, year, pages, isbn, ipfs_cid

This raw data is used to generate our index, you can download the raw data from here:

License

zlib-searcher © zu1k, Released under the MIT License.

Blog zu1k.com · GitHub @zu1k · Twitter @zu1k_lv · Telegram Channel @peekfun