Google Bigtable is a NoSQL database which offers a fully managed, scaling solution for big data applications. It's the same database that powers many core Google services, including Search, Analytics, Maps, and Gmail.
Initially Bigtable was released with only two native libraries: Java and Go. The Java implementation is an augmentation of the Apache HBase API to work with Bigtable. The Go implementation provides a wrapper around the low-level Bigtable API to take care of some of the eccentricities therein.
This library provides the first native client for connecting your .NET applications with Bigtable. It is cross-platform in architecture, and has been tested on 32-bit and 64-bit Windows environments and well as on Ubuntu 15.04 x64. In theory, it should work on Mac OSX. See the Linux installation notes for more information.
You can read more about Google Bigtable's design in their white paper.
- Cross-platform / Cross-architecture
- 32 or 64 bit
- Windows or Linux (kernel 3.19+)
- .NET 4.5+ support
- Microsoft .NET
- Mono 4.2.1
- Built on Google's gRPC and protobuf frameworks
- HTTP/2.0
- Proto 3
- Purely asynchronous operation
- Task Parallels Library
- C# 5 async/await
- Rx/Reactive methods for result streams
- IObservable / IObserver
- Thread-pool marshaling
- Three layers of abstraction from which to choose
- Directly consume gRPC
- Models for each Bigtable resource
- Work directly with POCOs
The Bigtable API is a combination of Google's protobuf serialization framework and gRPC, which provides the transport of protobuf serialized messages to well-defined RPC end-points.
This library uses the Bigtable protobuf library and pairs that with the grpc implementation for C#. This foundation manages asynchronous authentication, serialization, and transportation of the protobuf encoding messages to the Bigtable API servers.
This library is designed with three tiers of functionality from which to choose:
-
Native: Directly consume the protobuf/grpc layer.
-
Models: An abstraction of the Bigtable resources which simplifies most tasks but does not interfere with using lower level functionality (i.e. providing the RowFilter directly).
-
Mapper: An additional layer of abstraction which provides both CRUD and advanced operations on POCOs. There are Field and Key abstraction classes as well as attributes which provide similar functionality to Json.NET, such as changing the storage field name vs the POCO property name, however you are not bound to using them. Advanced operations such compare-and-swap and atomically increment are provided. In addition, you can inject custom serialization for your fields and/or keys, as well as use provided implementations. One example is hashing your keys transparently (as described in the schema design document).
More information about the design of this library can be found in the documentation.
Show me the code! There are four example projects included:
Low-Level: Demonstrates creating your own client using the the Native library.
var service = new BigtableTableService.BigtableTableServiceClient( _channel );
var request = new ListTablesRequest { Name = /*resource*/ };
var response = await service.ListTablesAsync( request );
return response.Tables.Select( DeconstructTableResource );Modeled: Demonstrates using the BigDataClient and BigAdminClient using the Models library.
var row = await dataClient.GetRowAsync( "myTable", "someKey" );
var rows = await dataClient.GetRowsAsync( "myTable", "startKey", "endKey", rowLimit: 20 );Mapped: Demonstrates using POCOs with the Bigtable client using the Mapper library.
var start = new MyPoco { IntKey = 1 };
var end = new MyPoco { IntKey = 2 };
var poco = await _bigtable.ScanAsync<MyPoco>( start, end, rowLimit: 10 );
poco.StringValue = "Pig";
await _bigtable.UpdateAsync( poco );
await _bigtable.UpdateWhenTrueAsync<MyPoco>(
row => row.StringValue == "Pig",
x => "Chicken"
);You are not constrained to creating POCOs for read signatures, though, therefore you can specify keys directly as string or byte[]:
var poco = await _bigtable.ScanAsync<MyPoco>( "1", "2", rowLimit: 10 );Web: Demonstrates using a web-based interface to interact with your Bigtable instance. The web project uses Nancy (because MVC 5 is not portable yet) on top of the Models layer.
http://localhost:8913/big/data/sample?table=myTable&maxrows=10
This solution can be loaded in Microsoft Visual Studio 2013 or 2015. It is .NET 4.5, C# 5.
After cloning this repository, you can either follow the directions in the Installation Guide to get the gRPC submodule packages setup or work around this step by running downloading the submodule packages and extracting in the root of the repository directory ("Extract Here").
Before proceeding, remember that Bigtable is not a free product, and this sample will add data to your cluster! While it will not add much data, by using these examples you accept all liability for the charges you incur while using Bigtable! At the time this was written, Google was offering a free 30-day trial to new subscribers.
In addition, the bootstrapper in the project will write to the cluster specified in the next paragraph. It may be a good idea to setup a cluster specifically for running the examples.
Edit config\template.json and specify your project, zone, cluster. You must set the GOOGLE_APPLICATION_CREDENTIALS environment variable for the examples to work. You should download the credentials file from the developer console.
Save the file as config\examples.config.json.
The very first time you run this project, you will need to populate the database with the example data tables and data. You can do that by running the Examples.Bootstrap project.
From there you can run any of the other examples.
This library is functional, but under development. See each individual project for more detailed information.
Once it has reached a stable state, we will provide the core libraries as NuGet packages.
This product is not affiliated with nor endorsed by Google, Microsoft, or the .NET Foundation in any way.
Please read the LICENSE and DISCLAIMER.