sparkey-python: A Python repository from Spotify

sparkey-python is a ctypes-based binding for the Sparkey library.

Dependencies

Python
libsparkey

Optional

epydoc (to generate the API documentation)

Building

# Python 2
PYTHONPATH=. nosetests

# Python 3
PYTHONPATH=. python -m "nose"

python setup.py build

API documentation can be generated with epydoc:

epydoc --no-private sparkey

License

Apache License, Version 2.0

Usage

To help get started, take a look at the API documentation or an example usage: smoke_test.py

Build & Install

Build and install was based off of this article

pip3 install -U pip pep517 twine
rm -rf build dist
python3 -m pep517.build .

Performance

This data is the direct output from running

PYTHONPATH=. python test/bench.py

on the same machine ((Intel(R) Xeon(R) CPU L5630 @ 2.13GHz)) as the performance benchmark for the sparkey c implementation, so the numbers should be somewhat comparable. The python version is 2.6.6.

Testing bulk insert of 1000 elements and 1000.000 random lookups
  Candidate: Sparkey None
    creation time (wall):      0.01
    throughput (puts/wallsec): 100000.00
    file size:                 28384
    lookup time (wall):           20.26
    throughput (lookups/wallsec): 49358.34
Testing bulk insert of 1000.000 elements and 1000.000 random lookups
  Candidate: Sparkey None
    creation time (wall):      5.23
    throughput (puts/wallsec): 191204.59
    file size:                 34177984
    lookup time (wall):           20.50
    throughput (lookups/wallsec): 48780.49
Testing bulk insert of 10.000.000 elements and 1000.000 random lookups
  Candidate: Sparkey None
    creation time (wall):      53.49
    throughput (puts/wallsec): 186950.83
    file size:                 413777988
    lookup time (wall):           20.68
    throughput (lookups/wallsec): 48355.90
Testing bulk insert of 100.000.000 elements and 1000.000 random lookups
  Candidate: Sparkey None
    creation time (wall):      544.57
    throughput (puts/wallsec): 183631.12
    file size:                 4337777988
    lookup time (wall):           22.75
    throughput (lookups/wallsec): 43956.04
Testing bulk insert of 1000 elements and 1000.000 random lookups
  Candidate: Sparkey Snappy
    creation time (wall):      0.67
    throughput (puts/wallsec): 1492.54
    file size:                 19085
    lookup time (wall):           23.71
    throughput (lookups/wallsec): 42176.30
Testing bulk insert of 1000.000 elements and 1000.000 random lookups
  Candidate: Sparkey Snappy
    creation time (wall):      5.28
    throughput (puts/wallsec): 189393.94
    file size:                 19168683
    lookup time (wall):           23.26
    throughput (lookups/wallsec): 42992.26
Testing bulk insert of 10.000.000 elements and 1000.000 random lookups
  Candidate: Sparkey Snappy
    creation time (wall):      54.41
    throughput (puts/wallsec): 183789.74
    file size:                 311872187
    lookup time (wall):           23.34
    throughput (lookups/wallsec): 42844.90
Testing bulk insert of 100.000.000 elements and 1000.000 random lookups
  Candidate: Sparkey Snappy

    creation time (wall):      554.73
    throughput (puts/wallsec): 180267.88
    file size:                 3162865465
    lookup time (wall):           25.12
    throughput (lookups/wallsec): 39808.92

When running with pypy 2.1.0+dfsg-3 we get these results: