Simbase is a redis-like vector similarity database. You can add, get, delete vectors to/from it, and then retrieve the most similar vectors within one vector set or between two vector sets.
Current version is v0.1.0.
Simbase use a concept model as below:
+ - - - +
+----------->| Basis |<------------------+
| belongs + _ _ _ + belongs |
| |
| |
+ - - - - - + source + - - - - - - - -+
| VectorSet |<------------------------| Recommendation |
+ - - - - - + + - - - - - - - -+
^ target |
|________________________________________|
- Vector set: a set of vectors
- Basis: the basis for vectors, vectors in one vector set have same basis
- Recommendation: a one-direction binary relationship between two vector sets which have the same basis
A real example follow the model below:
+ - - - - - + + - - - - - - - -+
+--->| Articles |<----------------| User Profiles |
| + - - - - - + + - - - - - - - -+
| |
+----------+
This graph shows
- recommend article by article (recommend from article to article)
- recommend article by user profile (recommend from user profile to article)
To build the project, you need install leiningen first, and then
cd SIMBASE_HOME
lein uberjar
After the uberjar is created, you can start the system
cd SIMBASE_HOME
bin/start
You can use redis-cli directly for administration tasks.
Or you can use redis client bindings in different language directly in a programming way.
Python example
import redis
dest = redis.Redis(host='localhost', port=7654)
dest.execute_command('bmk', 'ba', 'a', 'b', 'c')
dest.execute_command('vmk', 'ba', 'va')
dest.execute_command('rmk', 'va', 'va', 'cosinesq')Node.js example
var redis = require("redis"), client = redis.createClient(7654, 'localhost');
client.send_command('bmk', ['ba', 'a', 'b', 'c'])
client.send_command('vmk', ['ba', 'va'])
client.send_command('rmk', ['va', 'va', 'cosinesq'])For example, we need to recommend articles to users, we may follow below steps:
Setup
> bmk b2048 t1 t2 t3 ... t2047 t2048
> vmk b2048 article
> vmk b2048 userprofile
> rmk userprofile article cosinesq
Fill data
> vadd article 1 0.11 0.112 0.1123...
> vadd article 2 0.21 0.212 0.2123...
...
> vadd userprofile 1 0.11 0.112 0.1123...
> vadd userprofile 2 0.21 0.212 0.2123...
...
Query
> rrec userprofile 2 article
All commands are explained in next section.
Then you can use redis-cli to connect to simbase directly
Basis related
-
blist
blist
List all basis in system
-
bmk basisname components...
bmk b512 universe time space human animal plant...
Create a basis
-
brev basisname components...
brev b512 plant animal human space time universe...
Revise a basis
Vector set related
-
vlist basisname
vlist b512
List all vector set with one basis
-
vmk basisname vecsetname
vmk b512 article
Create a vector set
-
vget vecsetname vecid
vget article 12345678
Get the vector for the article with id 12345678
-
vadd vecsetname vecid components...
vadd article 12345678 0.1 0.12 0.123 0.1234 0.12345 0.123456...
add the value for the article vector with id 12345678
-
vset vecsetname vecid components...
vset article 12345678 0.1 0.12 0.123 0.1234 0.12345 0.123456...
set the value for the article vector with id 12345678
-
vacc vecsetname vecid components...
vacc article 12345678 0.1 0.12 0.123 0.1234 0.12345 0.123456...
accumulate the value for the article vector with id 12345678
-
vrem vecsetname vecid
vrem article 12345678
remove the vector with id 12345678 from article vector set
Recommendation related
-
rlist vecsetname
rlist article
List all recommendation targets with the inputed vecset as source
-
rmk vecsetname1 vecsetname2 funcscore
rmk userprofile article cosinesq
Create a recommendation to article by userprofile and it use cosinesq as score function. Currently score functions you can choice are: 'cosinesq' and 'jensenshannon'
-
rrec vecsetname1 vecid vecsetname2
rrec userprofile 87654321 article
Recommend articles for user 87654321
Although Simbase can store arbitrary vectors, but score functions may apply some constraints on vectors.
For example, if you adopt "jensenshannon" as your score function, you should assure your vector is a probability distribution, i.e. the sum of all components equals to one.
The write operation is handled in a single thread per basis, and comparison between any two vectors is needed, so the write operation is scaled at O(n).
We had a non-final performance test for the dense vectors on an i7-cpu Macbook, it can easily handle 100k 1k-dimensional vectors with each write operation in under 0.14 sec; and if the linear scale ratio can hold, it means Simbase can handle 700k dense vectors with each write operation in under 1 sec.
Since the data is all in memory, the read operation is pretty fast.
We are still in the process of tuning the performance of the sparse vectors.
Simbase is dual licensed under the Apache License 2.0 and Eclipse Public License 1.0. Simbase is free for commercial use and distribution under the terms of either license.
Special thanks for Feng Sheng, we borrowed lots of code from his great project http-kit ( https://github.com/http-kit/http-kit/ ).
Also thanks for Kunwei Zhang from Tsinghua Univ. for his smart idea.
- Mingli Yuan ( https://github.com/mountain )
- Wanjian Wu ( https://github.com/jseagull )
- Yang Zhang ( https://github.com/zmouren )
- Jianjiang Zhu ( https://github.com/zjjott )
- Jiacai Liu ( https://github.com/jiacai2050 )