/dynein

DynamoDB CLI written in Rust.

Primary LanguageRustApache License 2.0Apache-2.0

dynein - DynamoDB CLI

dynein /daɪ.nɪn/ is a command line interface for Amazon DynamoDB written in Rust. dynein is designed to make it simple to interact with DynamoDB tables/items from terminal.

Why use dynein?

Less Typing

  • Auto completion for table/keyDefinitions enables using DynamoDB with minimum arguments. e.g. to get an item: dy get abc
  • Switching table context by RDBMS-ish "use".
  • Prefer standard JSON ( {"id": 123} ) over DynamoDB JSON ( {"id": {"N": "123"}} ).

Quick Start

  • Bootstrap command enables you to launch sample table with sample data sets.
  • Supports DynamoDB Local and you can test DyanmoDB at no charge.

For day-to-day tasks

  • Import/Export by single command: export DynamoDB items to CSV/JSON files and conversely, import them into tables.
  • Taking on-demand backup and restore data from them.

Installation

Method 1. Download binaries

You can download binaries of a specific version from the releases page. For example, below instructions are example comamnds to download the latest version in each platform.

macOS

$ curl -O -L https://github.com/awslabs/dynein/releases/latest/download/dynein-macos.tar.gz
$ tar xzvf dynein-macos.tar.gz
$ mv dy /usr/local/bin/
$ dy --help

Currently, the above binary is automatically built on intel mac as the GitHub Action doesn't support Apple M1 (ARM) environment yet.

Linux (x86-64)

$ curl -O -L https://github.com/awslabs/dynein/releases/latest/download/dynein-linux.tar.gz
$ tar xzvf dynein-linux.tar.gz
$ sudo mv dy /usr/local/bin/
$ dy --help

Method 2. Homebrew (macOS)

Currently not available. Please use the Method 1 above.

Method 3. Building from source

dynein is written in Rust, so you can build and install dynein with Cargo. To build dynein from source code you need to install Rust as a prerequisite.

$ git clone [[this_git_repository_url]]
$ cd dynein
$ cargo install --locked --path .
$ ./target/release/dy --help

You can move the binary file named "dy" to anywhere under your $PATH.

How to Use

Prerequisites - AWS Credentials

First of all, please make sure you've already configured AWS Credentials in your environment. dynein depends on rusoto and rusoto can utilize standard AWS credential toolchains - for example ~/.aws/credentials file, IAM EC2 Instance Profile, or environment variables such as AWS_DEFAULT_REGION / AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_PROFILE.

One convenient way to check if your AWS credential configuration is ok to use dynein is to install and try to execute AWS CLI in your environment (e.g. $ aws dynamodb list-tables). Once you've configured AWS CLI, you should be ready to use dynein.

Commands overview

After you installed dynein you should have a binary named dy in your $PATH. The first command you can try is dy ls, which lists tables you have:

$ dy ls --all-regions
DynamoDB tables in region: us-west-2
  EventData
  EventUsers
* Forum
  Thread
DynamoDB tables in region: us-west-1
  No table in this region.
DynamoDB tables in region: us-east-2
  UserBooks
  Users
...

Here --all-regions option enables you to iterate over all AWS regions and list all tables for you.

Next you can try dy scan with region and table options. dy scan command executes Scan API internally to retrieve all items in the table.

$ dy scan --region us-west-2 --table Forum
Name             attributes
Amazon S3        {"Category":"Amazon Web Services"}
Amazon DynamoDB  {"Views":1000,"Threads":2,"Messages":4,"Category":...

Here Name is a primary key of this Forum table and attributes column contains rest attributes of each item.

You don't want to pass --region and --table everytime? Let's mark the table as "currently using" with the command dy use.

$ dy use Forum --region us-west-2

Now you can interact with the table without specifying a target.

$ dy scan
Name             attributes
Amazon S3        {"Category":"Amazon Web Services"}
Amazon DynamoDB  {"Threads":2,"Views":1000,"Messages":4,"Category":...

To find more features, dy help will show you complete list of available commands.

$ dy --help
dynein x.x.x
dynein is a command line tool to interact with DynamoDB tables/data using concise interface.
dynein looks for config files under $HOME/.dynein/ directory.

USAGE:
    dy [OPTIONS] <SUBCOMMAND>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -r, --region <region>    The region to use (e.g. --region us-east-1). When using DynamodB Local, use `--region
                             local`. You can use --region option in both top-level and subcommand-level
    -t, --table <table>      Target table of the operation. You can use --table option in both top-level and subcommand-
                             level. You can store table schema locally by executing `$ dy use`, after that
                             you need not to specify --table on every command

SUBCOMMANDS:
    admin        <sub> Admin operations such as creating/updating table or GSI
    backup       Take backup of a DynamoDB table using on-demand backup
    bootstrap    Create sample tables and load test data for bootstrapping
    bwrite       Put or Delete multiple items at one time, up to 25 requests. [API: BatchWriteItem]
    config       <sub> Manage configuration files (config.yml and cache.yml) from command line
    del          Delete an existing item. [API: DeleteItem]
    desc         Show detailed information of a table. [API: DescribeTable]
    export       Export items from a DynamoDB table and save them as CSV/JSON file
    get          Retrieve an item by specifying primary key(s). [API: GetItem]
    help         Prints this message or the help of the given subcommand(s)
    import       Import items into a DynamoDB table from CSV/JSON file
    list         List tables in the region. [API: ListTables]
    put          Create a new item, or replace an existing item. [API: PutItem]
    query        Retrieve items that match conditions. Partition key is required. [API: Query]
    restore      Restore a DynamoDB table from backup data
    scan         Retrieve items in a table without any condition. [API: Scan]
    upd          Update an existing item. [API: UpdateItem]
    use          Switch target table context. After you use the command you don't need to specify table every time,
                 but you may overwrite the target table with --table (-t) option

dynein consists of multiple layers of subcommands. For example, dy admin and dy config require you to give additional action to run.

$ dy admin --help
dy-admin x.x.x
<sub> Admin operations such as creating/updating table or GSI

USAGE:
    dy admin [OPTIONS] <SUBCOMMAND>

FLAGS: ...

OPTIONS: ...

SUBCOMMANDS:
    create    Create new DynamoDB table or GSI. [API: CreateTable, UpdateTable]
    delete    Delete a DynamoDB table or GSI. [API: DeleteTable]
    desc      Show detailed information of a table. [API: DescribeTable]
    help      Prints this message or the help of the given subcommand(s)
    list      List tables in the region. [API: ListTables]
    update    Update a DynamoDB table. [API: UpdateTable etc]

By executing following command, you can create a DynamoDB table.

$ dy admin create table mytable --keys pk,S

Bootstrapping sample DynamoDB tables

The easiest way to get familiar with dynein and DynamoDB would be executing dy bootstrap. The bootstrap subcommand creates sample tables and automatically load sample data defined here. After that, you'll see some sample commands to demonstrate basic usage of dynein.

$ dy bootstrap

Bootstrapping - dynein will creates 4 sample tables defined here:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AppendixSampleTables.html

'ProductCatalog' - simple primary key table
    Id (N)

'Forum' - simple primary key table
    Name (S)

'Thread' - composite primary key table
    ForumName (S)
    Subject (S)

'Reply' - composite primary key table, with GSI named 'PostedBy-Message-Index'
    Id (S)
    ReplyDateTime (S)

...(snip logs)...

Now all tables have sample data. Try following commands to play with dynein. Enjoy!
  $ dy --region us-west-2 ls
  $ dy --region us-west-2 desc --table Thread
  $ dy --region us-west-2 scan --table Thread
  $ dy --region us-west-2 use --table Thread
  $ dy scan

After you 'use' a table like above, dynein assume you're using the same region & table, which info is stored at ~/.dynein/config.yml and ~/.dynein/cache.yml
Let's move on with the 'us-west-2' region you've just 'use'd...
  $ dy scan --table Forum
  $ dy scan -t ProductCatalog
  $ dy get -t ProductCatalog 101
  $ dy query -t Reply "Amazon DynamoDB#DynamoDB Thread 2"
  $ dy query -t Reply "Amazon DynamoDB#DynamoDB Thread 2"  --sort-key "begins_with 2015-10"

If you're interested in other available sample tables with data, check dy bootstrap --list and pass desired target to --sample option.

Working with DynamoDB tables

Using dynein, you can create a table:

$ dy admin create table app_users --keys app_id,S user_id,S
---
name: app_users
region: us-east-1
status: CREATING
schema:
  pk: app_id (S)
  sk: user_id (S)
mode: OnDemand
capacity: ~
gsi: ~
lsi: ~
stream: ~
count: 0
size_bytes: 0
created_at: "2020-03-03T13:34:43+00:00"

After the table get ready (i.e. status: CREATING changed to ACTIVE), you can write-to and read-from the table.

$ dy use app_users
$ dy desc
---
name: app_users
region: us-east-1
status: ACTIVE
schema:
  pk: app_id (S)
  sk: user_id (S)
mode: OnDemand
capacity: ~
gsi: ~
lsi: ~
stream: ~
count: 0
size_bytes: 0
created_at: "2020-03-03T13:34:43+00:00"

$ dy put myapp 1234 --item '{"rank": 99}'
Successfully put an item to the table 'app_users'.

$ dy scan
app_id  user_id  attributes
myapp   1234     {"rank":99}

Similarly you can update tables with dynein.

$ dy admin update table app_users --mode provisioned --wcu 10 --rcu 25

Infrastracture as Code - enpowered by CloudFormation

NOTE: currently this feature is under development

Infrastracture as Code is a concept that you define code to provision "infrastructures", such as DynamoDB tables, with "declarative" way (On the other hand you can say dy admin create table and dy admin update table commands are "imperative" way).

To manage DynamoDB tables with "declarative" way, dynein provides dy admin plan and dy admin apply commands. Internally dynein executes AWS CloudFormation APIs to provision DynamoDB resources for you.

$ ls
mytable.cfn.yml

$ cat mytable.cfn.yml
Resources:
  MyDDB:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
      - AttributeName: pk
        AttributeType: S
      KeySchema:
      - AttributeName: pk
        KeyType: HASH
      BillingMode: PAY_PER_REQUEST

(currently not available) $ dy admin plan
(currently not available) $ dy admin apply

CloudFormation manages DynamoDB tables through the resource type named AWS::DynamoDB::Table - visit the link for more information.

dy use and dy config to switch/manage context

Basically it's pretty straight forward to specify table with which you want to interact with: --table or -t option. Let's say you want to scan data in the customers table.

$ dy scan --table customers
... display items in the "customers" table ...

However, dynein assume that tipically you're interested in only one table at some point. It means that passing table name for every single command execution is a kinf of waste of your time.

By using dy use for a table, you can call commands such as scan, get, query, and put without specifying table name.

$ dy use customers
$ dy scan
... display items in the "customers" table ...

In detail, when you execute dy use command, dynein saves your table usage information in ~/.dynein/config.yml and caches table schema in ~/.dynein/cache.yml. You can dump them with dy config dump command.

$ ls ~/.dynein/
cache.yml   config.yml

$ dy config dump
---
tables:
  ap-northeast-1/customers:
    region: ap-northeast-1
    name: customers
    pk:
      name: user_id
      kind: S
    sk: ~
    indexes: ~
---
using_region: ap-northeast-1
using_table: customers

To clear current table configuration, simply execute dy config clear.

$ dy config clear
$ dy config dump
---
tables: ~
---
using_region: ~
using_table: ~

Working with DynamoDB items

As an example let's assume you have official "Movie" sample data. To prepare the table with data loaded, simply you can execute dy bootstrap --sample movie.

$ dy bootstrap --sample movie
... wait some time while dynein loading data ...
$ dy use Movie

After executing dy use <your_table> command, dynein recognize keyscheme and data type of the table. It means that some of the arguments you need to pass to access data (items) is automatically inferred when possible.

Before diving deep into each command, let me describe DynamoDB's "reserved words". One of the traps that beginners can easily fall into is that you cannot use certain reserved words in DynamoDB APIs. DynamoDB reserved words contains common words that you may want to use in your application. For example "name", "year", "url", "token", "error", "date", "group" -- all of them are reserved so you cannot use them in expressions directly.

Normally, to use reserved words in expressions, you need to use placeholders instead of actual values. For more information, see Expression Attribute Names and Expression Attribute Values.

To make it easy to interact with DynamoDB items, dynein automatically replace reserved words to placeholders internally - thus you don't need to care about it.

Read

dy scan

The simplest command would be dy scan, which list items in a table.

$ dy scan --limit 10
year  title                  attributes
1933  King Kong              {"info":{"actors":["Bruce Cabot","Fay Wray","Rober...
1944  Arsenic and Old Lace   {"info":{"actors":["Cary Grant","Priscilla Lane","...
1944  Double Indemnity       {"info":{"actors":["Barbara Stanwyck","Edward G. R...
1944  I'll Be Seeing You     {"info":{"actors":["Ginger Rogers","Joseph Cotten"...
1944  Lifeboat               {"info":{"actors":["John Hodiak","Tallulah Bankhea...
1958  Cat on a Hot Tin Roof  {"info":{"actors":["Burl Ives","Elizabeth Taylor",...
1958  Monster on the Campus  {"info":{"actors":["Arthur Franz","Joanna Moore","...
1958  No Time for Sergeants  {"info":{"actors":["Andy Griffith","Myron McCormic...
1958  Teacher's Pet          {"info":{"actors":["Clark Gable","Doris Day","Gig ...
1958  Touch of Evil          {"info":{"actors":["Charlton Heston","Janet Leigh"...

dy get

You may notice that non-key attributes are trimmed in above dy scan results. To get full details of a single item, you use dy get command with primary key. As the "Movie" table is defined with composite primary key, you have to pass "partition key (= year) and "sort key (= title)" to identify an item.

$ dy desc
---
name: Movie
region: us-west-2
status: ACTIVE
schema:
  pk: year (N)     <<<<=== "year" and
  sk: title (S)    <<<<=== "title" are the information to identify an item.
...

$ dy get 1958 "Touch of Evil"
{
  "info": {
    "actors": [
      "Charlton Heston",
      "Janet Leigh",
      "Orson Welles"
    ],
    "directors": [
      "Orson Welles"
    ],
    "genres": [
      "Crime",
      "Film-Noir",
      "Thriller"
    ],
    "image_url": "http://ia.media-imdb.com/images/M/MV5BNjMwODI0ODg1Nl5BMl5BanBnXkFtZTcwMzgzNjk3OA@@._V1_SX400_.jpg",
    "plot": "A stark, perverse story of murder, kidnapping, and police corruption in a Mexican border town.",
    "rank": 3843,
    "rating": 8.2,
    "release_date": "1958-04-23T00:00:00Z",
    "running_time_secs": 5700
  },
  "title": "Touch of Evil",
  "year": 1958
}

Note that if your table has a simple primary key, the only argument you need to pass is a partition key (e.g. dy get yourpk), as the only information DynamoDB requires to identify an item is only a partition key.

dy query

Next command you can try to retrieve items would be: dy query. By passing a partition key, dy query returns items that have the specified partition key.

$ dy query 1960
year  title                  attributes
1960  A bout de souffle      {"info":{"actors":["Daniel Boulanger","Jean Seberg...
1960  La dolce vita          {"info":{"actors":["Anita Ekberg","Anouk Aimee","M...
1960  Ocean's Eleven         {"info":{"actors":["Dean Martin","Frank Sinatra","...
1960  Plein soleil           {"info":{"actors":["Alain Delon","Marie Laforet","...
1960  Spartacus              {"info":{"actors":["Jean Simmons","Kirk Douglas","...
1960  The Apartment          {"info":{"actors":["Fred MacMurray","Jack Lemmon",...
1960  The Magnificent Seven  {"info":{"actors":["Charles Bronson","Steve McQuee...
1960  The Time Machine       {"info":{"actors":["Alan Young","Rod Taylor","Yvet...

Also you can add more conditions on sort key. For example, following command would return items that has sort keys begins with "The".

$ dy query 1960 --sort-key "begins_with The"
year  title                  attributes
1960  The Apartment          {"info":{"actors":["Fred MacMurray","Jack Lemmon",...
1960  The Magnificent Seven  {"info":{"actors":["Charles Bronson","Steve McQuee...
1960  The Time Machine       {"info":{"actors":["Alan Young","Rod Taylor","Yvet...

Other examples for the --sort-key option of dy query are: --sort-key "= 42", --sort-key "> 42", or --sort-key "between 10 and 42". For more details please visit a public document "Working with Queries" and DynamoDB Query API reference.

Write

dynein provides subcommands to write to DynamoDB tables as well.

dy put

dy put internally calls PutItem API and save an item to a target table. To save an item, you need to pass at least primary key that identifies an item among the table.

$ dy admin create table write_test --keys id,N
$ dy use write_test

$ dy put 123
Successfully put an item to the table 'write_test'.
$ dy scan
id  attributes
123

Additionally you can include item body (non-key attributes) by passing --item or -i option. The --item option takes JSON style syntax.

$ dy put 456 --item '{"a": 9, "b": "str"}'
Successfully put an item to the table 'write_test'.

$ dy scan
id  attributes
123
456  {"a":9,"b":"str"}

As dynein's --item option automatically transform standard JSON into DynamoDB style JSON syntax, writing items into a table would be simpler than AWS CLI. See following comparison:

$ dy put 789 --item '{"a": 9, "b": "str"}'

// The above dynein command is equivalent to AWS CLI's following command:
$ aws dynamodb put-item --table-name write_test --item '{"id": {"N": "456"}, "a": {"N": "9"}, "b": {"S": "str"}}'

Finally, in addition to the string ("S") and nubmer ("N"), dynein also supports other data types such as boolean ("BOOL"), null ("NULL"), string set ("SS"), number set ("NS"), list ("L"), and nested object ("M").

$ dy put 999 --item '{"myfield": "is", "nested": {"can": true, "go": false, "deep": [1,2,{"this_is_set": ["x","y","z"]}]}}'
Successfully put an item to the table 'write_test'.
$ dy get 999
{
  "nested": {
    "can": true,
    "deep": [
      1,
      2,
      {
        "this_is_set": [
          "x",
          "y",
          "z"
        ]
      }
    ],
    "go": false
  },
  "myfield": "is",
  "id": 999
}

dy upd

dy upd command internally executes UpdateItem API and you use "update expression" to update an item. Recommended way to update items is use SET and REMOVE in update expression.

with dynein, you can use --set or --remove option. Here's an exmaple:

$ dy get 42
{
  "flag": true,
  "id": 42
}

$ dy upd 42 --set "flag = false"
Successfully updated an item in the table 'write_test'.

$ dy get 42
{
  "flag": false,
  "id": 42
}

Next let me show an example to use --remove. Note that --remove in dy upd command never remove "item" itself, instead --remove just removes an "attribute".

$ dy upd 42 --remove "flag"
Successfully updated an item in the table 'write_test'.

$ dy get 42
{
  "id": 42
}

dynein has a special command named --atomic-counter. It increases specified number atribute by 1.

$ dy get 52
{
  "age": 28,
  "name": "John",
  "id": 52
}

$ dy upd 52 --atomic-counter age
Successfully updated an item in the table 'write_test'.

$ dy get 52
{
  "age": 29,
  "id": 52,
  "name": "John"
}

dy del

To delete an item, you use dy del command with primary key to identify an item.

$ dy get 42
{ "id": 42 }
$ dy del 42
Successfully deleted an item from the table 'write_test'.
$ dy get 42
No item found.

Working with Indexes

DynamoDB provides flexible way to query data efficiently by utilizing Secondary Index features. There're two types of secondary indexes: GSI (Global Secondary Index) and LSI (Local Secondary Index), but you can create LSI only when creating a table.

With dynein, you can add GSI to existing table.

$ dy admin create index top_rank_users_index --keys rank,N --table app_users
---
name: app_users
region: us-west-2
status: UPDATING
schema:
  pk: app_id (S)
  sk: user_id (S)
mode: OnDemand
capacity: ~
gsi:
  - name: top_rank_users_index
    schema:
      pk: rank (N)
      sk: ~
    capacity: ~
lsi: ~
stream: ~
count: 0
size_bytes: 0
created_at: "2020-06-02T14:22:56+00:00"

$ dy use app_users
$ dy scan --index top_rank_users_index

Import/Export for DynamoDB items

dy export

You can export DynamoDB items into JSON or CSV file. As the default format is json, you can simply call following command to export:

$ dy export --table Reply --format json --output-file out.json
$ cat out.json
[
  {
    "PostedBy": "User A",
    "ReplyDateTime": "2015-09-15T19:58:22.947Z",
    "Id": "Amazon DynamoDB#DynamoDB Thread 1",
    "Message": "DynamoDB Thread 1 Reply 1 text"
  },
  {
    "Id": "Amazon DynamoDB#DynamoDB Thread 1",
...

No --format option means --format json. If you want to dump data in oneline, try --format json-compact. Or, if you want to export in JSONL (JSON Lines), i.e. "one JSON item per one line" style, --format jsonl is also available.

$ dy export --table Reply --format jsonl --output-file out.jsonl
$ cat out.jsonl
{"PostedBy":"User A","ReplyDateTime":"2015-09-15T19:58:22.947Z","Message":"DynamoDB Thread 1 Reply 1 text","Id":"Amazon DynamoDB#DynamoDB Thread 1"}
{"PostedBy":"User B","Message":"DynamoDB Thread 1 Reply 2 text","ReplyDateTime":"2015-09-22T19:58:22.947Z","Id":"Amazon DynamoDB#DynamoDB Thread 1"}
...

When export data to CSV, primary key(s) are exported by default. You can explicitly pass additional attributes to export.

$ dy export --table Reply --output-file out.csv --format csv --attributes PostedBy,Message
$ cat out.csv
Id,ReplyDateTime,PostedBy,Message
"Amazon DynamoDB#DynamoDB Thread 1","2015-09-15T19:58:22.947Z","User A","DynamoDB Thread 1 Reply 1 text"
...

dy import

To import data into a table, you use with specified --format option. Here default format is JSON like dy export.

$ dy import --table target_movie --format json --input-file movie.json

Using DynamoDB Local with --region local option

DynamoDB provides free tier that consists of 25 GB of storage and 25 WCU/RCU which is enough to handle up to 200M requests per month. However, if you're already using DynamoDB in your account and worrying about additional costs by getting started with dynein, you can use DynamoDB Local.

Yes, dynein supports DynamoDB Local. The only difference you need to add would be --region local option for every command. To get start with DynamoDB Local Docker version with dynein is quite simple as follows.

Simply you can run docker container and expose 8000 port by following command.

$ docker run -p 8000:8000 -d amazon/dynamodb-local

Optionally, if you prefer Kubernetes, you can use manifest file in this repository.

$ kubectl apply -f k8s-deploy-dynamodb-local.yml
$ kubectl port-forward deployment/dynamodb 8000:8000

Now you can interact with DynamoDB Local with --region local option.

$ dy --region local admin create table localdb --keys pk
$ dy --region local use -t localdb
$ dy put firstItem
$ dy put secondItem
$ dy scan

Misc

Asides

dynein is named after a motor protein.

Troubleshooting

If you encounter troubles, the first option worth trying is removing files in ~/.dynein/ or the directory itself. Doing this just clears "cached" info stored locally for dynein and won't affect your data stored in DynamoDB tables.

$ rm -rf ~/.dynein/

To see verbose output for troubleshooting purpose, you can change log level by RUST_LOG environment variable. For example:

$ RUST_LOG=debug RUST_BACKTRACE=1 dy scan --table your_table

Ideas for future works

  • dy admin plan & dy admin apply commands to manage tables through CloudFormation.
  • Linux's top -like experience to monitor table status. e.g. dy top tables
    • inspired by kubectl top nodes
    • implementation: (CloudWatch metrics such as Consumed WCU/RCU, SuccessfulRequestLatency, ReplicationLatency for GT etc)
  • Shell (bash/zsh) completion
  • Retrieving control plane APIs, integrated with CloudTrail
  • dy logs command to retrieving data plane API logs via DynamoDB Streams (write APIs only)
    • tail -f -ish usability. e.g. dy logs -f mytable
  • truncate command to delete all data in a table
  • Support Transaction APIs (TransactGetItems, TransactWriteItems)
  • simple load testing. e.g. dy load --tps 100
  • import/export tool supports LTSV, TSV
  • PITR configuration enable/disable (UpdateContinuousBackups) and exporting/restoring tables (ExportTableToPointInTime, RestoreTableToPointInTime)