/oomstore

Lightweight and Fast Feature Store Powered by Go (and Rust).

Primary LanguageGoApache License 2.0Apache-2.0

OomStore

Lightweight and Fast Feature Store Powered by Go

CICD Version Version Platform

Quickstart · Architecture · Benchmark

Overview

oomstore allows you to:

  • Define features with YAML.
  • Store features in databases of choice.
  • Retrieve features for both online serving and offline training, fast.

Please see our docs for more details.

Features

  • 🍼 Simple. Being serverless and CLI-friendly, users can be productive in hours, not months.
  • 🔌 Composable. We support your preferred databases of choice.
  • ⚡ Fast. Benchmark shows oomstore performs QPS > 50k and latency < 0.3 ms with Redis.
  • 🌊 Streaming. We support streaming features to ensure your predictions are up-to-date.

Architecture

Architecture

See Architecture for more details.

Quickstart

  1. Install oomcli following the guide.

  2. oomcli init to initialize oomstore. Make sure there is a ~/.config/oomstore/config.yaml as below.

online-store:
  sqlite:
    db-file: /tmp/oomstore.db

offline-store:
  sqlite:
    db-file: /tmp/oomstore.db

metadata-store:
  sqlite:
    db-file: /tmp/oomstore.db
  1. oomcli apply -f metadata.yaml to register metadata. See metadata.yaml below.
kind: Entity
name: user
description: 'user ID'
groups:
- name: account
  category: batch
  description: 'user account'
  features:
  - name: state
    value-type: string
  - name: credit_score
    value-type: int64
  - name: account_age_days
    value-type: int64
  - name: 2fa_installed
    value-type: bool
- name: txn_stats
  category: batch
  description: 'user txn stats'
  features:
  - name: count_7d
    value-type: int64
  - name: count_30d
    value-type: int64
- name: recent_txn_stats
  category: stream
  snapshot-interval: 24h
  description: 'user recent txn stats'
  features:
  - name: count_10min
    value-type: int64
  1. Import CSV data to Offline Store, then sync from Offline to Online Store
oomcli import \
  --group account \
  --input-file account.csv \
  --description 'sample account data'
oomcli import \
  --group txn_stats \
  --input-file txn_stats.csv \
  --description 'sample txn stats data'
oomcli sync --group-name account --revision-id 2
oomcli sync --group-name txn_stats --revision-id 4
  1. Push stream data to both Online and Offline Store.
oomcli push --group recent_txn_stats --entity-key 1006 --feature count_10min=1
  1. Fetch features by key.
oomcli get online \
  --entity-key 1006 \
  --feature account.state,account.credit_score,account.account_age_days,account.2fa_installed,txn_stats.count_7d,txn_stats.count_30d,recent_txn_stats.count_10min
+------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+
| user | account.state | account.credit_score | account.account_age_days | account.2fa_installed | txn_stats.count_7d | txn_stats.count_30d | recent_txn_stats.count_10min |
+------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+
| 1006 | Louisiana     |                  710 |                       32 | false                 |                  8 |                  22 |                            1 |
+------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+
  1. Generate training datasets via Point-in-Time Join.
oomcli join \
	--feature account.state,account.credit_score,account.account_age_days,account.2fa_installed,txn_stats.count_7d,txn_stats.count_30d,recent_txn_stats.count_10min \
	--input-file label.csv
+------------+---------------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+
| entity_key |  unix_milli   | account.state | account.credit_score | account.account_age_days | account.2fa_installed | txn_stats.count_7d | txn_stats.count_30d | recent_txn_stats.count_10min |
+------------+---------------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+
|       1002 | 1950236233000 | Hawaii        |                  625 |                      861 | true                  |                 11 |                  36 |                              |
|       1003 | 1950411318000 | Arkansas      |                  730 |                      958 | false                 |                  0 |                  16 |                              |
|       1004 | 1950653614000 | Louisiana     |                  610 |                     1570 | false                 |                 12 |                  26 |                              |
|       1005 | 1950166137000 | South Dakota  |                  635 |                     1953 | false                 |                  7 |                  30 |                              |
|       1006 | 1950403162000 | Louisiana     |                  710 |                       32 | false                 |                  8 |                  22 |                            1 |
|       1007 | 1950160030000 | New Mexico    |                  645 |                       37 | true                  |                  5 |                  40 |                              |
|       1008 | 1950274859000 | Nevada        |                  735 |                     1627 | false                 |                 12 |                  51 |                              |
|       1009 | 1949958846000 | Kentucky      |                  650 |                       88 | true                  |                 11 |                  23 |                              |
|       1010 | 1949920686000 | Delaware      |                  680 |                     1687 | false                 |                  2 |                  39 |                              |
+------------+---------------+---------------+----------------------+--------------------------+-----------------------+--------------------+---------------------+------------------------------+

See Quickstart for more complete details.

Supported Databases

Online Store

  • Amazon DynamoDB
  • Redis
  • TiKV
  • Cassandra
  • PostgreSQL
  • MySQL
  • TiDB
  • SQLite

Offline Store

  • Snowflake
  • Amazon Redshift
  • Google BigQuery
  • PostgreSQL
  • MySQL
  • TiDB
  • SQLite

Metadata Store

  • PostgreSQL
  • MySQL
  • TiDB
  • SQLite

Community

Feel free to join the community for questions and feature requests!

Credits

oomstore is highly inspired by Feast.