FSharp.AWS.DynamoDB
FSharp.AWS.DynamoDB an F# wrapper over the standard Amazon.DynamoDB library which allows you to represent table items using F# records and perform updates, queries and scans using F# quotation expressions.
The API draws heavily on the corresponding FSharp.Azure.Storage wrapper for Azure table storage.
Introduction
Table items can be represented using F# records:
open FSharp.AWS.DynamoDB
type WorkItemInfo =
{
[<HashKey>]
ProcessId : int64
[<RangeKey>]
WorkItemId : int64
Name : string
UUID : Guid
Dependencies : Set<string>
Started : DateTimeOffset option
}
We can now perfom table operations on DynamoDB like so
open Amazon.DynamoDBv2
let client : IAmazonDynamoDB = ``your DynamoDB client instance``
let table = TableContext.Create<WorkItemInfo>(client, tableName = "workItems", createIfNotExists = true)
let workItem = { ProcessId = 0L ; WorkItemId = 1L ; Name = "Test" ; UUID = guid() ; Dependencies = set ["mscorlib"] ; Started = None }
let key : TableKey = table.PutItem(workItem)
let workItem' = table.GetItem(key)
Queries and scans can be performed using quoted predicates
let qResults = table.Query(keyCondition = <@ fun r -> r.ProcessId = 0 @>,
filterCondition = <@ fun r -> r.Name = "test" @>)
let sResults = table.Scan <@ fun r -> r.Started.Value >= DateTimeOffset.Now - TimeSpan.FromMinutes 1. @>
Values can be updated using quoted update expressions
let updated = table.UpdateItem(<@ fun r -> { r with Started = Some DateTimeOffset.Now } @>,
preCondition = <@ fun r -> r.DateTimeOffset = None @>)
Or they can be updated using the UpdateOp
DSL
which is closer to the underlying DynamoDB API
let updated = table.UpdateItem <@ fun r -> SET r.Name "newName" &&& ADD r.Dependencies ["MBrace.Core.dll"] @>
Supported Field Types
FSharp.AWS.DynamoDB supports the following field types:
- Numerical types, enumerations and strings.
- Array, Nullable, Guid, DateTimeOffset and TimeSpan.
- F# lists
- F# sets with elements of type number, string or
byte[]
. - F# maps with key of type string.
- F# records and unions (recursive types not supported).
Supported methods in Query Expressions
Query expressions support the following F# methods in their predicates:
Array.length
,List.length
,Set.count
andMap.Count
.String.StartsWith
andString.Contains
.Set.contains
andMap.containsKey
.Array.isEmpty
andList.isEmpty
.Option.isSome
,Option.isNone
,Option.Value
andOption.get
.fst
andsnd
for tuple records.
Supported methods in Update Expressions
Update expressions support the following F# value constructors:
(+)
and(-)
in numerical and set types.Array.append
andList.append
(or@
).- List consing (
::
). defaultArg
on optional fields.Set.add
andSet.remove
.Map.add
andMap.remove
.Option.Value
andOption.get
.fst
andsnd
for tuple records.
Example: Creating an atomic counter
type private CounterEntry = { [<HashKey>]Id : Guid ; Value : int64 }
type Counter private (table : TableContext<CounterEntry>, key : TableKey) =
member __.Value = table.GetItem(key).Value
member __.Incr() =
let updated = table.UpdateItem(key, <@ fun e -> { e with Value = e.Value + 1L } @>)
updated.Value
static member Create(client : IAmazonDynamoDB, table : string) =
let table = TableContext.Create<CounterEntry>(client, table, createIfNotExists = true)
let entry = { Id = Guid.NewGuid() ; Value = 0L }
let key = table.PutItem entry
new Counter(table, key)
Projection Expressions
Projection expressions can be used to fetch a subset of table attributes, which can be useful when performing large queries:
table.QueryProjected(<@ fun r -> r.HashKey = "Foo" @>, <@ fun r -> r.HashKey, r.Values.Nested.[0] @>)
which returns a tuple of specified attributes. Tuples can be of any arity and must contain non-conflicting document paths.
Secondary Indices
Global Secondary Indices can be defined using the GlobalSecondaryHashKey
and GlobalSecondaryRangeKey
attributes:
type Record =
{
[<HashKey>] HashKey : string
...
[<GlobalSecondaryHashKey(indexName = "Index")>]GSIH : string
[<GlobalSecondaryRangeKey(indexName = "Index")>]GSIR : string
}
Queries can now be performed on the GSIH
and GSIR
fields as if they were regular hashkey and rangekey attributes.
Global secondary indices are created using the same provisioned throughput as the primary keys.
Local Secondary Indices can be defined using the LocalSecondaryIndex
attribute:
type Record =
{
[<HashKey>] HashKey : string
[<RangeKey>] RangeKey : Guid
...
[<LocalSecondaryIndex>] LSI : double
}
Queries can now be performed using LSI as a secondary RangeKey.
NB: Due to API restrictions, secondary indices in the scope of FSharp.AWS.DynamoDB always project all table attributes which can incur additional costs from Amazon.
Pagination
Pagination is supported on both scans & queries:
let firstPage = table.ScanPaginated(limit = 100)
printfn "First 100 results = %A" firstPage.Records
match firstPage.LastEvaluatedKey with
| Some key ->
let nextPage = table.ScanPaginated(limit = 100, exclusiveStartKey = key)
Note that the exclusiveStartKey
on paginated queries must include both the table key fields and the index fields (if querying an LSI or GSI).
This is accomplished via the IndexKey
type - if constructing manually (eg deserialising a start key from an API call):
let startKey = IndexKey.Combined(gsiHashValue, gsiRangeValue, TableKey.Hash(primaryKey))
let page = table.QueryPaginated(<@ fun t -> t.GsiHash = gsiHashValue @>, limit = 100, exclusiveStartKey = startKey)
Notes on value representation
Due to restrictions of DynamoDB, it may sometimes be the case that objects are not persisted faithfully. For example, consider the following record definition:
type Record =
{
[<HashKey>]
HashKey : Guid
Optional : int option option
Lists : int list list
}
let item = { HashKey = Guid.NewGuid() ; Optional = Some None ; Lists = [[1;2];[];[3;4]] }
let key = table.PutItem item
Subsequently recovering the given key will result in the following value:
> table.GetItem key
val it : Record = {HashKey = 8d4f0678-6def-4bc9-a0ff-577a53c1337c;
Optional = None;
Lists = [[1;2]; [3;4]];}
Precomputing DynamoDB Expressions
It is possible to precompute a DynamoDB expression as follows:
let precomputedConditional = table.Template.PrecomputeConditionalExpr <@ fun w -> w.Name <> "test" && w.Dependencies.Contains "mscorlib" @>
This precomputed conditional can now be used in place of the original expression in the FSharp.AWS.DynamoDB API:
let results = table.Scan precomputedConditional
FSharp.AWS.DynamoDB also supports precomputation of parametric expressions:
let startedBefore = table.Template.PrecomputeConditionalExpr <@ fun time w -> w.StartTime.Value <= time @>
table.Scan(startedBefore (DateTimeOffset.Now - TimeSpan.FromDays 1.))
Observability
A hook is provided so metrics can be published via your preferred Observability provider. For example, using Prometheus.NET:
let dbCounter = Metrics.CreateCounter ("aws_dynamodb_requests_total", "Count of all DynamoDB requests", "table", "operation")
let processMetrics (m : RequestMetrics) =
dbCounter.WithLabels(m.TableName, m.Operation.ToString ()).Inc () |> ignore
let table = TableContext.Create<WorkItemInfo>(client, tableName = "workItems", metricsCollector = processMetrics)
If metricsCollector
is supplied, the requests will include ReturnConsumedCapacity = ReturnConsumedCapacity.INDEX
and the RequestMetrics
parameter will contain a list of ConsumedCapacity
objects returned from the DynamoDB operations.
Building & Running Tests
To build using the dotnet SDK:
dotnet tool restore
dotnet build
Tests are run using dynamodb-local on port 8000. Using the docker image is recommended:
docker run -p 8000:8000 amazon/dynamodb-local
then
dotnet run -p tests/FSharp.AWS.DynamoDB.Tests/FSharp.AWS.DynamoDB.Tests.fsproj
Maintainer(s)
The default maintainer account for projects under "fsprojects" is @fsprojectsgit - F# Community Project Incubation Space (repo management)