/awesome-go-storage

A curated list of awesome Go storage projects and libraries

MIT LicenseMIT

Awesome Go Storage Awesome

A curated list of awesome Go storage projects and libraries. Inspired by awesome-go.

Contributing

Please take a quick gander at the contribution guidelines first. Thanks to all contributors; you rock!

If you see a package or project here that is no longer maintained or is not a good fit, please submit a pull request to improve this file. Thank you!

Contents

Storage Server

Storage Servers implemented in Go.

  • minio - Minio is an open source object storage server compatible with Amazon S3 APIs.
  • rclone - "rsync for cloud storage" - Google Drive, Amazon Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Cloudfile…
  • perkeep - Perkeep is your personal storage system for life: a way of storing, syncing, sharing, modelling and backing up content.
  • s3git - Git for Cloud Storage. Distributed Version Control for Data.
  • storj - Decentralized cloud object storage that is affordable, easy to use, private, and secure.
  • rook - Open, Cloud Native, and Universal Distributed Storage.
  • longhorn Longhorn is an open source persistent block storage server delivered via containers.

Key-Value Store

Key-Value Store implemented in Go.

  • BadgerDB - BadgerDB is an embeddable, persistent, simple and fast key-value (KV) database written in pure Go. It's meant to be a performant alternative to non-Go-based key-value stores like RocksDB.
  • biscuit - Biscuit is a multi-region HA key-value store for your AWS infrastructure secrets.
  • consul - Distributed consistent replicated key-value store for service discovery and configuration.
  • diskv - A disk-backed key-value store.
  • etcd - Distributed reliable key-value store for the most critical data of a distributed system.
  • go-cache - An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

File System

File Systems implemented in Go.

  • git-lfs - Git extension for versioning large files.
  • seaweedfs - SeaweedFS is a simple and highly scalable distributed file system for small files.
  • fsnotify - Cross-platform file system notifications for Go.
  • goofys - A high-performance, POSIX-ish Amazon S3 file system written in Go.
  • go-systemd - Go bindings to systemd socket activation, journal, D-Bus, and unit files.
  • gcsfuse - A user-space file system for interacting with Google Cloud Storage.
  • svfs - A virtual file system over Openstack Swift built upon fuse.
  • afero - A FileSystem Abstraction System for Go

Database

Databases implemented in Go.

  • BigCache - Efficient key/value cache for gigabytes of data.
  • bolt - A low-level key/value database for Go. This original version by Ben Johnson has been marked as unmaintained and forked by CoreOS bbolt.
  • buntdb - A fast, embeddable, in-memory key/value database for Go with custom indexing and spatial support.
  • cache2go - An in-memory key:value cache which supports automatic invalidation based on timeouts.
  • cockroach - A Scalable, Geo-Replicated, Transactional Datastore
  • couchcache - A RESTful caching micro-service backed by Couchbase server.
  • CovenantSQL - A SQL Database with Blockchain features.
  • dgraph - Scalable, Distributed, Low Latency, High Throughput Graph Database.
  • diskv - A home-grown disk-backed key-value store.
  • eliasdb - Dependency-free, transactional graph database with REST API, phrase search and SQL-like query language.
  • emitter - Scalable, low-latency, distributed & secure pub/sub database with time-series message storage, suitable for IoT, gaming, apps and real-time web.
  • forestdb - Go bindings for ForestDB.
  • GCache - Cache library with support for expirable Cache, LFU, LRU and ARC.
  • geocache - An in-memory cache that is suitable for geolocation based applications.
  • go-cache - An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.
  • goleveldb - An implementation of the LevelDB key/value database in the Go.
  • groupcache - Groupcache is a caching and cache-filling library, intended as a replacement for memcached in many cases.
  • influxdb - Scalable datastore for metrics, events, and real-time analytics
  • ledisdb - Ledisdb is a high performance NoSQL like Redis based on LevelDB.
  • levigo - Levigo is a Go wrapper for LevelDB.
  • moss - Moss is a simple LSM key-value storage engine written in 100% Go.
  • noms - The versioned, forkable, syncable database.
  • piladb - Lightweight RESTful database engine based on stack data structures.
  • pREST - Serve a RESTful API from any PostgreSQL database.
  • prometheus - Monitoring system and time series database.
  • rqlite - The lightweight, distributed, relational database built on SQLite.
  • scribble - A tiny flat file JSON store.
  • tidb - TiDB is a distributed SQL database. Inspired by the design of Google F1.
  • tiedot - Your NoSQL database powered by Golang.
  • Tile38 - A geolocation DB with spatial index and realtime geofencing.

Database schema migration.

  • darwin - Database schema evolution library for Go
  • goose - Database migration tool. You can manage your database's evolution by creating incremental SQL or Go scripts.
  • gormigrate - Database schema migration helper for Gorm ORM.
  • migrate - Database migration handling in Golang support MySQL, PostgreSQL, Cassandra, and SQLite.
  • pravasan - Simple Migration tool - currently for MySQL but planning to support soon for Postgres, SQLite, MongoDB, etc.,
  • soda - Database migration, creation, ORM, etc... for MySQL, PostgreSQL, and SQLite.
  • sql-migrate - Database migration tool. Allows embedding migrations into the application using go-bindata.

Database tools.

  • go-mysql - A go toolset to handle MySQL protocol and replication.
  • go-mysql-elasticsearch - Sync your MySQL data into Elasticsearch automatically.
  • kingshard - kingshard is a high performance proxy for MySQL powered by Golang.
  • myreplication - MySql binary log replication listener. Support statement and row based replication.
  • orchestrator - MySQL replication topology manager & visualizer
  • pgweb - A web-based PostgreSQL database browser
  • vitess - vitess provides servers and tools which facilitate scaling of MySQL databases for large scale web services.
  • usql - universal command-line interface for SQL databases.
  • pgweb - Cross-platform client for PostgreSQL databases.

SQL query builder, libraries for building and using SQL.

  • dat - Go Postgres Data Access Toolkit
  • Dotsql - Go library that helps you keep sql files in one place and use it with ease.
  • goqu - An idiomatic SQL builder and query library.
  • igor - Abstraction layer for PostgreSQL that supports advanced functionality and uses gorm-like syntax.
  • ozzo-dbx - Powerful data retrieval methods as well as DB-agnostic query building capabilities.
  • scaneo - Generate Go code to convert database rows into arbitrary structs.
  • sqrl - SQL query builder, fork of Squirrel with improved performance.
  • Squirrel - Go library that helps you build SQL queries.
  • xo - Generate idiomatic Go code for databases based on existing schema definitions or custom queries supporting PostgreSQL, MySQL, SQLite, Oracle, and Microsoft SQL Server.

Database Drivers

Libraries for connecting and operating databases.

  • Relational Databases

    • bgc - Datastore Connectivity for BigQuery for go.
    • firebirdsql - Firebird RDBMS SQL driver for Go
    • go-adodb - Microsoft ActiveX Object DataBase driver for go that using database/sql.
    • go-bqstreamer - BigQuery fast and concurrent stream insert.
    • go-mssqldb - Microsoft MSSQL driver in go language.
    • go-oci8 - Oracle driver for go that using database/sql.
    • go-sql-driver/mysql - MySQL driver for Go.
    • go-sqlite3 - SQLite3 driver for go that using database/sql.
    • gofreetds Microsoft MSSQL driver. Go wrapper over FreeTDS.
    • pgx - PostgreSQL driver supporting features beyond those exposed by database/sql.
    • pq - Pure Go Postgres driver for database/sql.

  • NoSQL Databases

    • aerospike-client-go - Aerospike client in Go language.
    • arangolite - Lightweight golang driver for ArangoDB.
    • asc - Datastore Connectivity for Aerospike for go.
    • cayley - A graph database with support for multiple backends.
    • dsc - Datastore connectivity for SQL, NoSQL, structured files.
    • dynago - Dynago is a principle of least surprise client for DynamoDB
    • go-couchbase - Couchbase client in Go
    • go-couchdb - Yet another CouchDB HTTP API wrapper for Go
    • gocb - Official Couchbase Go SDK
    • gocql - A Go language driver for Apache Cassandra.
    • gomemcache - memcache client library for the Go programming language.
    • gorethink - Go language driver for RethinkDB
    • goriak - Go language driver for Riak KV
    • mgo - MongoDB driver for the Go language that implements a rich and well tested selection of features under a very simple API following standard Go idioms.
    • neo4j - Neo4j Rest API Bindings for Golang
    • Neo4j-GO - Neo4j REST Client in golang.
    • neoism - Neo4j client for Golang
    • redigo - Redigo is a Go client for the Redis database.
    • redis - Redis client for Golang
    • redis - A simple, powerful Redis client for Go.
    • redis - Redis-protocol compatible TCP servers/services.

  • Search and Analytic Databases

    • bleve - A modern text indexing library for go.
    • elastic - Elasticsearch client for Go.
    • elastigo - A Elasticsearch client library.
    • goes - A library to interact with Elasticsearch.
    • skizze - A probabilistic data-structures service and storage.