FDB

FoundationDB client for Elixir

Status

API is in alpha state and backward incompatible changes may be introduced in subsequent versions.

Implementation Details

As there is no documented stable wire protocol the only practical option is to use the C API. NIF has some major downsides

pre-emptive scheduling

The FoundationDB C API uses event loop architecture. Nearly all the API functions are non blocking — blocking API functions are not used by FDB. The event loop runs on a seperate thread and the communication is done via callback functions. The callback function when invoked will send a message to Process. This architecture makes sure the NIF functions return immediatly and gives the control back to VM

memory protection

This mostly comes down to careful coding. Currenly I am running the tests under valgrind locally. With some effort it could be integrated in travis. FDB also runs the bindings tester (used to test other language bindings) in travis CI.

concurrency

The FoundationDB C API functions are thread safe except for the network intialization part. NIF implementation tries to avoid concurrency problems by not mutating the values once created.

Program testing can be used to show the presence of bugs, but never to show their absence!

Edsger W. Dijkstra

It's still possible that there are bugs in C API or the NIF implementation, which could lead to VM crash.

API Design

It's recommended to read the Developer Guide and Data Modeling to get a good understanding of FoundationDB. Most of the ideas apply across all the language bindings.

Async

Most of the operations in FDB are async in nature. FDB provides two kinds of api

a sync api that will block the calling process till the operation is done. In case of failure an exception will be raised.
an async api that will return t:FDB.Future.t/0 immediatly. The caller can later use FDB.Future.await/1 to resolve the value, which will block till the operation is done or will raise an exception in case of failure.

The async api ends with _q, for example FDB.Transaction.get/2 is the sync version and FDB.Transaction.get_q/2 is the async version of the same function.

Error Handling

FoundationDB uses optimistic concurrency. When a transaction is committed, it could get cancelled if there are other conflicting transactions. The common idiom is to retry the cancelled transaction till it succeeds. FDB.Database.transact/2 function automatically rescues and retries if the error is retriable. For this reason, the api is designed to raise exception instead of returning {:error, error}

Installation

FDB depends on FoundationDB client binary to be installed. The version of the client binary should be >= FDB library version — patch and build part in the version can be ignored. For example, if you want to use

{:fdb, "5.1.7-0"}

then you must have client binary >= 5.1. Only patch versions are guaranteed to be protocol compatible.

Additional Steps on Windows

To compile the library in Windows you must have the Visual C++ Tools installed or VS 2017, if you don't use it probably you'll get a message telling you that nmake isn't installed.

With Visual C++ Tools: search for the file vcvarsall.bat, the Tools version 2017 are commonly located at C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Auxiliary\Build and run the command vcvarsall amd64.
With Visual Studio 2017 intalled: type Developer Command in the search box and you will get the cmd program as a result.

Then move to your project directory and run mix compile

Getting Started

Before doing anything with the library, the API version has to be set and the network thread has to be started. FDB.start/1 is a helper function which does all of these.

:ok = FDB.start(710)

This must be called only once. Calling it second time will result in exception. Once started, t:FDB.Database.t/0 instance have to be created.

db = FDB.Database.create(cluster_file_path)

It's recommended to use a single db instance everywhere unless multiple db with different set of options are required. There are no performance implications with using a single db instance as none of the method calls are serialized either via locks or GenServer et al.

Any kind of interaction with Database requires the usage of t:FDB.Transaction.t/0. There are two ways of using transaction

FDB.Database.transact(db, fn transaction ->
  value = FDB.Transaction.get(transaction, key)
  :ok = FDB.Transaction.set(transaction, key, value <> "hello")
end)

transaction = FDB.Transaction.create(db)
value = FDB.Transaction.get(transaction, key)
:ok = FDB.Transaction.set(transaction, key, value <> "hello")
:ok = Transaction.commit(transaction)

The first version is the preferred one. The transaction is automatically committed after the callback returns. In case any exception is raised inside the callback or in the commit function call, the transaction will be retried if the error is retriable. Various options like max_retry_delay, timeout, retry_limit etc can be configured using FDB.Transaction.set_option/3

Coder

Most of the language bindings implement the tuple layer. It specifies how native types like integer, unicode string, bytes etc should be encoded. The main advantage of the encoding over others is that it preserves the natural ordering of the values, so the range function would work as expected.

alias FDB.{Transaction, Database, KeySelectorRange}
alias FDB.Coder.{Integer, Tuple, NestedTuple, ByteString, Subspace}

coder =
  Transaction.Coder.new(
    Subspace.new(
      {"ts", ByteString.new()},
      Tuple.new({
        # date
        NestedTuple.new({
          # year, month, date
          NestedTuple.new({Integer.new(), Integer.new(), Integer.new()}),
          # hour, minute, second
          NestedTuple.new({Integer.new(), Integer.new(), Integer.new()})
        }),
        # website
        ByteString.new(),
        # page
        ByteString.new(),
        # browser
        ByteString.new()
      })
    ),
    Integer.new()
  )
db = Database.create(%{coder: coder})

Database.transact(db, fn t ->
  m = Transaction.get(t, {{{2018, 03, 01}, {1, 0, 0}}, "www.github.com", "/fdb", "mozilla"})
  c = Transaction.get(t, {{{2018, 03, 01}, {1, 0, 0}}, "www.github.com", "/fdb", "chrome"})
end)

range = KeySelectorRange.starts_with({{{2018, 03, 01}}})
result =
  Database.get_range_stream(db, range)
  |> Enum.to_list()

A t:FDB.Transaction.Coder.t/0 specifies how the key and value should be encoded. The coder could be set at database or transaction level. The transaction automatically inherits the coder from database if not set explicitly. Under the hood all the functions use the coder transparently to encode and decode the values. Refer FDB.Database.set_defaults/2 if you want to use multiple coders.

See the documentation for more information.

Benchmark

A simple, unreliable and non-scientific benchmark can be found here

ananthakumaran/fdb

FDB