xedni: A Lua repository from caryfitzhugh

A gem and backend solution for searching moderate amounts of data
with faceting.

* consistent (on writes as requested)
* simple for testing & production



Everything is a collection:
  :text=>  ['chicken', 'peas', 'pepper']
  :spicy=> 4

So your query comes like this:
filter:
  text:   ['chicken']
  spicy:  ['4',  '-']
score:
  :text => 1,
  :spicy => -1.1

Which means they want text matching occurrance to be a factor, and less spicy to be important

Implement:
 temp_redis_key_1 = [ all the recipe record ids which were in any of the filters]
 temp_redis_key_2 = [ all the recipe weights]
 result_key       = sort by key_2

Getting the weight for a redis key involves getting the key values,
function gets the values from the incoming filter
score = recipe.collection_weight(recipe.text, ['chicken']) * 1 +
        recipe.collection_weight(recipe.spicy, ['4','5','6'....'100']) * 4

recipe.text would initially return # of matched values
recipe.spicy wold return the spicy valu (since it's not an array, but an int) of 4

then sort by those values


Call Stack:

Model

HasXedni

XedniGem

(if in local mode )         ( If in remote mode)
XedniInterface              XedniServer
                            XedniInterface

Redis


Check out:
https://github.com/djanowski/redis-scripted/blob/master/lib/redis/scripted.rb


Xedni lets you do CRUD actions on the records:
Xedni does not supply GUIDs - you must do so. or some unique key (like table_name_#{id})

Query
Create  (or update works as well).
Read
Update
Delete

You can get all the information about a record through 'Read'.
You can update the records facets (or it's score) with update / create
You can remove a record from the system with Delete. (that's a hard remove... no going back!)

Create/Update have the same API: (Example)
:id    => "45_recipe" (just .to_i in rails :))
:keys  => ["chicken", "soup", "nuts", {:ingredients=>[5]}, {instructions=>[10]}]
:score => {:popularity => 0.8, :quality => 0.5}


Query allows you to make a query on the database.
:search => {
  :and/or =>
    ['chicken', {:instructions=>[10,11,12]}]
},
* Will this allow us to combine with 'merge' easily?

:facets => { :chicken, :instructions, :ingredients, :publisher, :heart_healthy }

:score  => ... (TBD, for now it is always default), at times
  {
    :score_key => custom_weighting
  },
:page / :per_page

 < ======== Returns
:records => [key, key, key, key, key, key],
:facets  =>
{
  :instructions => { '45' => 33 items, '44' => 31 items },
  :chicken      => 8000
}

Internal structure:

simple => {
  'chicken' => [ id, id, id, id, id ]
}
Redis:
xedni::instance::chicken => Set()

facets => {
  'ingredients' => {'1' -> [ ... ], '2' -> [ .... ] }
}
xedni:instance:ingredients:_keys => Set() of values (appended below)
xedni:instance:ingredients:1
xedni:instance:ingredients:2
xedni:instance:ingredients:3

records => {
  'id' => { verbatim data from the update... , :keys, :score, :id }
}
xedni:instance:records:#{id} => Hash()

Scores: ?
xedni:instance:scores:_keys => score types
::

Processing:
Query ->
  Create Redis call to find all record ids which match the OR and AND query

  In Redis -- uniq, etc those records.
  Calculate all the facet counts from these results.
  Sort the records given their score values and the passed in query weightings.
  Paginate
  Return.


HOW TO HANDLE USER-SCOPED RECORDS?
-- if I want to search a recipe box -- I guess I can just push in a set of record IDs that are in their box (maybe max of a few thousand?)
-- if I want to customize search for a user -- I can adjust weighting values... spicy factor + / -, etc


Use Cases:

Index all the records in the DB.
Every time we print a recipe -- >
MetaRecipe.update_xendi_rank('printed')

And in some mapping somewhere, we define what that means.
It shoots off a request to Xendi, to update the score for that record.


Load Balancer
     |
     V
[xedni-server, xedni-server, xedni-server]
     |
     V
[redis primary, redis secondary]
caryfitzhugh/xedni
