Sneller is a high-performance SQL engine built to analyze petabyte-scale un-structured logs and other event data.
Here are a couple major differentiators between Sneller and other SQL solutions:
- Sneller is designed to use cloud object storage as its only backing store.
- Sneller's SQL VM is implemented in AVX-512 assembly. Medium-sized compute clusters provide throughput in excess of terabytes per second.
- Sneller is completely schemaless. No more ETL-ing your data! Heterogeneous JSON data can be ingested directly.
- Sneller uses a hybrid approach between columnar and row-oriented data layouts to provide lightweight ingest, low storage footprint, and super fast scanning speeds.
Sneller Cloud gives you access to a hosted version of the Sneller SQL engine that runs directly on data stored entirely in your S3 buckets. Our cloud platform offers excellent performance and is priced at an extremely competitive $150 per petabyte of data scanned.
You can run queries for free against Sneller Cloud from your browser through our playground. We've created a public table containing about 1 billion rows from the GitHub archive data set. Additionally, you can create new ephemeral tables by uploading your own JSON data (but please don't upload anything sensitive!)
The Sneller playground is also usable directly with a local http client like curl:
If you have go installed on a machine with AVX512, you can build tables
from JSON files and run the query engine locally:
$ grep -q avx512 /proc/cpuinfo && echo "yes, I have AVX512"
yes, I have AVX512
$ # install the sdb tool (make sure $GOBIN is in your $PATH)
$ go install github.com/SnellerInc/sneller/cmd/sdb@latest
$ # pack a JSON object into a table that can be queried;
$ # here we're using some github archive JSON:
$ wget https://data.gharchive.org/2015-01-01-15.json.gz
$ sdb pack -o github.zion 2015-01-01-15.json.gz
$ # run a query, using JSON as the output format:
$ sdb query -v -fmt=json "select count(*), type from read_file('github.zion') group by type"
{"type": "CreateEvent", "count": 1471}
{"type": "PushEvent", "count": 5815}
{"type": "WatchEvent", "count": 1230}
{"type": "ReleaseEvent", "count": 60}
{"type": "PullRequestEvent", "count": 474}
{"type": "IssuesEvent", "count": 545}
{"type": "ForkEvent", "count": 355}
{"type": "GollumEvent", "count": 61}
{"type": "IssueCommentEvent", "count": 844}
{"type": "DeleteEvent", "count": 260}
{"type": "PullRequestReviewCommentEvent", "count": 136}
{"type": "CommitCommentEvent", "count": 73}
{"type": "MemberEvent", "count": 25}
{"type": "PublicEvent", "count": 2}
18874368 bytes (18.000 MiB) scanned in 1.475857ms 12.5GiB/sSee our SQL reference for more information on the Sneller SQL dialect.
If you don't have access to a physical machine with AVX512 support, we recommend renting a VM from one of the major cloud providers with one of these instance families:
- AWS: c6i, m6i, r6i
- GCP: N2, M2, C2, C3
- Azure: Dv4, Ev4
Our cloud platform simplifies the Sneller SQL user experience by giving you instant access to thousands of CPU cores to run your queries. Sneller Cloud also provides automatic synchronization between your source data and your SQL tables, so you don't have any batch processes to manage in order to keep your tables up-to-date. Our cloud solution has a simple usage-based pricing model that depends entirely on the amount of data your queries scan. (Since Sneller Cloud doesn't store any of your data, there are no additional storage charges.)
Sneller is generally able to provide end-to-end scanning performance in excess of 1GB/s/core on high-core-count machines. The core SQL engine is typically able to saturate the memory bandwidth of the machine; generally about half of the query execution time is spent decompressing the source data, and the other half is spent in the SQL engine itself. Scanning performance scales linearly with the number of CPU cores available, so for example a 1000-CPU cluster would generally provide scanning performance in excess of 1TB/s.
The zion compression format that the SQL engine consumes is "bucketized" so that
queries that don't touch all of the fields in the source data consume fewer cycles
during decompression. Concretely, the top-level fields in each record are hashed
into one of 16 buckets, and each of these buckets is compressed separately.
The query planner determines which fields are referenced by each query, and at
execution time only the buckets that contain fields necessary to compute the final
query result are actually decompressed. (Strictly columnar formats like Parquet
stripe data into one bucket per column, with the restriction that the columns
and their types are known in advance. Since Sneller operates on un-structured
data, our solution needed to be completely agnostic to the structure of the data itself.)
Most of Sneller is released under the AGPL-3.0 license. See the LICENSE file for more information.
The ion, iguana and date packages are released under the Apache 2.0 license.
See LICENSE.apache for more information.