OomStore

Lightweight and Fast Feature Store Powered by Go

Quickstart · Architecture · Benchmark

Overview

oomstore allows you to:

Define features with YAML.
Store features in databases of choice.
Retrieve features for both online serving and offline training, fast.

Please see our docs for more details.

Features

🍼 Simple. Being serverless and CLI-friendly, users can be productive in hours, not months.
🔌 Composable. We support your preferred databases of choice.
⚡ Fast. Benchmark shows oomstore performs QPS > 50k and latency < 0.3 ms with Redis.
🌊 Streaming. We support streaming features to ensure your predictions are up-to-date.

Architecture

See Architecture for more details.

Quickstart

Install oomcli following the guide.
oomcli init to initialize oomstore. Make sure there is a ~/.config/oomstore/config.yaml as below.

online-store:
  sqlite:
    db-file: /tmp/oomstore.db

offline-store:
  sqlite:
    db-file: /tmp/oomstore.db

metadata-store:
  sqlite:
    db-file: /tmp/oomstore.db

oomcli apply -f metadata.yaml to register metadata. See metadata.yaml below.

kind: Entity
name: user
description: 'user ID'
groups:
- name: account
  category: batch
  description: 'user account'
  features:
  - name: state
    value-type: string
  - name: credit_score
    value-type: int64
  - name: account_age_days
    value-type: int64
  - name: 2fa_installed
    value-type: bool
- name: txn_stats
  category: batch
  description: 'user txn stats'
  features:
  - name: count_7d
    value-type: int64
  - name: count_30d
    value-type: int64
- name: recent_txn_stats
  category: stream
  snapshot-interval: 24h
  description: 'user recent txn stats'
  features:
  - name: count_10min
    value-type: int64

Import CSV data to Offline Store, then sync from Offline to Online Store

oomcli import \
  --group account \
  --input-file account.csv \
  --description 'sample account data'
oomcli import \
  --group txn_stats \
  --input-file txn_stats.csv \
  --description 'sample txn stats data'
oomcli sync --group-name account --revision-id 2
oomcli sync --group-name txn_stats --revision-id 4

Push stream data to both Online and Offline Store.

oomcli push --group recent_txn_stats --entity-key 1006 --feature count_10min=1

Fetch features by key.

oomcli get online \
  --entity-key 1006 \
  --feature account.state,account.credit_score,account.account_age_days,account.2fa_installed,txn_stats.count_7d,txn_stats.count_30d,recent_txn_stats.count_10min

+------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+
| user | account.state | account.credit_score | account.account_age_days | account.2fa_installed | txn_stats.count_7d | txn_stats.count_30d | recent_txn_stats.count_10min |
+------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+
| 1006 | Louisiana     |                  710 |                       32 | false                 |                  8 |                  22 |                            1 |
+------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+

Generate training datasets via Point-in-Time Join.

oomcli join \
	--feature account.state,account.credit_score,account.account_age_days,account.2fa_installed,txn_stats.count_7d,txn_stats.count_30d,recent_txn_stats.count_10min \
	--input-file label.csv

+------------+---------------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+
| entity_key |  unix_milli   | account.state | account.credit_score | account.account_age_days | account.2fa_installed | txn_stats.count_7d | txn_stats.count_30d | recent_txn_stats.count_10min |
+------------+---------------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+
|       1002 | 1950236233000 | Hawaii        |                  625 |                      861 | true                  |                 11 |                  36 |                              |
|       1003 | 1950411318000 | Arkansas      |                  730 |                      958 | false                 |                  0 |                  16 |                              |
|       1004 | 1950653614000 | Louisiana     |                  610 |                     1570 | false                 |                 12 |                  26 |                              |
|       1005 | 1950166137000 | South Dakota  |                  635 |                     1953 | false                 |                  7 |                  30 |                              |
|       1006 | 1950403162000 | Louisiana     |                  710 |                       32 | false                 |                  8 |                  22 |                            1 |
|       1007 | 1950160030000 | New Mexico    |                  645 |                       37 | true                  |                  5 |                  40 |                              |
|       1008 | 1950274859000 | Nevada        |                  735 |                     1627 | false                 |                 12 |                  51 |                              |
|       1009 | 1949958846000 | Kentucky      |                  650 |                       88 | true                  |                 11 |                  23 |                              |
|       1010 | 1949920686000 | Delaware      |                  680 |                     1687 | false                 |                  2 |                  39 |                              |
+------------+---------------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+

See Quickstart for more complete details.

Supported Databases

Online Store

Amazon DynamoDB
Redis
TiKV
Cassandra
PostgreSQL
MySQL
TiDB
SQLite

Offline Store

Snowflake
Amazon Redshift
Google BigQuery
PostgreSQL
MySQL
TiDB
SQLite

Metadata Store

PostgreSQL
MySQL
TiDB
SQLite

Community

Feel free to join the community for questions and feature requests!

Credits

oomstore is highly inspired by Feast.

oom-ai/oomstore