dystopian_index: A Ruby repository from alexyoung

DystopianIndex

This is a Rails plugin that uses Tokyo Dystopia to index models. It’s ideal for small projects that need a fast indexer because it’s easy to setup and understand, and doesn’t use too many resources or daemon processes.

Requirements

Gems:

faker
rufus-tokyo

Installation

Build and compile Tokyo Cabinet and Tokyo Dystopia from tokyocabinet.sourceforge.net — they’re clean and simple C projects and should build on your Mac or Linux machine.

Then install the rufus-tokyo gem.

Usage

To use the plugin with a model, define an index like this:

  dystopian_index do
    indexes :content, :name
    order_by :created_at
  end

Rake Tasks

rake dystopia:benchmarks                  # Runs benchmarks
rake dystopia:index                       # Indexes all models

Database files are stored in db/indexes. You can change the path where the database is stored by setting DystopianIndex.config.db_path.

Plugin Design

Each model has its own Dystopia index
Fields are concatenated into one big index record — this means you can’t currently search according to field
When an update is performed, the index is opened and closed on demand to help prevent data loss. I’ve indexed this scheme (see the benchmarks) and it’s not too slow
Data information is stored in the first few bytes of each index record to help sort values
Indexes are stored in db/indexes/ — the plugin will make this directory if required

TODO

Tests only run from within a Rails app, and by executing each test file with ruby
It might be better to create separate indexes for each field.
order_by needs desc
Sorting fetches every matching ID from dystopia and sorts according to date integers. There might be a better way to do this, but it’s still pretty fast

Acknowledgements

The Rake tasks need to load all models. This code was based on ThinkingSphinx’s code that does the same thing (which is a nice piece of work by the way). I hope to refactor this out later.

alexyoung/dystopian_index