OpenST Storage

OpenST Storage contains DB storage libraries and respective services. It also contains data sharding libraries and services. While OpenST Storage is available as-is for anyone to use, we caution that this is early stage software and under heavy ongoing development and improvement. Please report bugs and suggested improvements.

Constructor parameters:

There is 1 parameter required while creating the storage implementer.

First parameter is mandatory and it specifies the configuration strategy to be used. An example of the configStrategy is:

configStrategy = {
  OST_DEBUG_ENABLED:'1',
  OS_DYNAMODB_API_VERSION: '2012-08-10',
  OS_DYNAMODB_ACCESS_KEY_ID: 'x',
  OS_DYNAMODB_SECRET_ACCESS_KEY: 'x',
  OS_DYNAMODB_REGION: 'localhost',
  OS_DYNAMODB_ENDPOINT: 'http://localhost:8000',
  OS_DYNAMODB_SSL_ENABLED: '0',
  OS_DYNAMODB_LOGGING_ENABLED: '1',
  AUTO_SCALE_DYNAMO: '0',

  OS_AUTOSCALING_API_VERSION: '2016-02-06' ,
  OS_AUTOSCALING_ACCESS_KEY_ID: 'x' ,
  OS_AUTOSCALING_SECRET_ACCESS_KEY: 'x' ,
  OS_AUTOSCALING_REGION: 'localhost',
  OS_AUTOSCALING_ENDPOINT: 'http://localhost:8000',
  OS_AUTOSCALING_SSL_ENABLED: '0',
  OS_AUTOSCALING_LOGGING_ENABLED: '1',

  OS_DYNAMODB_TABLE_NAME_PREFIX: 'd_pk_',
  OST_CACHING_ENGINE: 'memcached',
  OST_CACHE_CONSISTENT_BEHAVIOR: 1,
  OST_REDIS_HOST: '127.0.0.1',
  OST_REDIS_PORT: 6379,  
  OST_REDIS_PASS: 'st123',
  OST_REDIS_TLS_ENABLED: 0,
  OST_DEFAULT_TTL: '36000',
  OST_CACHE_ENDPOINT: '127.0.0.1:6379',
  OST_MEMCACHE_SERVERS: '127.0.0.1:11211'
};

DynamoDB Apis

For all DynamoDB methods parameters description please refer AWS DynamoDB Docs

Note: Response of all the apis is in ResponseHelper object wrapped in Promise.

DynamoDB constructor

params dynamodbConnectionParams

  const OpenSTStorage = require('@openstfoundation/openst-storage');
  let storage = OpenSTStorage.getInstance( configStrategy );
  let ddbServiceObj = storage.dynamoDBService;

Create table

params createTableParams

    //Create DynamoDB Table
    ddbServiceObj.createTable(createTableParams);

Create table migration

Table migration needs autoScaleObject
params createTableMigrationParams params.createTableConfig
params.updateContinuousBackupConfig
params.autoScalingConfig.registerScalableTargetWrite
params.autoScalingConfig.registerScalableTargetRead
params.autoScalingConfig.putScalingPolicyWrite
params.autoScalingConfig.putScalingPolicyRead
params.autoScalingConfig.globalSecondaryIndex[<GSI_INDEX_NAME>].autoScalingConfig - Auto Scaling config of Global Secondary Indexes same as of auto scale table config

 // Create Table Migration
 // 1. Creates table
 // 2. Enables read/write auto scaling
 // 3. Returns describe table response 
 ddbServiceObj.createTableMigration(createTableMigrationParams);

Update table

params updateTableParams

// Update DynamoDB Table
ddbServiceObj.updateTable(updateTableParams);

Describe table

params describeTableParams

// Describe DynamoDB Table 
ddbServiceObj.describeTable(describeTableParams);

List tables

params listTableParams

// List DynamoDB Tables
ddbServiceObj.listTables(listTableParams);

Update Continuous Backups

params updateContinuousParams

// Point in time recovery for DynamoDB Table
ddbServiceObj.updateContinuousBackups(updateContinuousParams);

Delete table

params deleteTableParams

// Delete DynamoDB table
ddbServiceObj.deleteTable(deleteTableParams);

Batch Get Item

params batchGetItemParams

// Batch Get Item
ddbServiceObj.batchGetItem(batchGetItemParams);

Batch Write Item

params batchWriteItemParams
params unprocessedItemsRetryCount Retry count for unprocessed Items

// Batch Write Item
ddbServiceObj.batchWriteItem(batchWriteItemParams, unprocessedRetryCount);

Query

params queryParams

// Query Items
ddbServiceObj.query(queryParams);

Scan

params scanParams

// Scan Items
ddbServiceObj.scan(scanParams);

Put Item

params putItemParams

// Put Items
ddbServiceObj.putItem(putItemParams);

Update Item

params updateItemParams

// Update an Item
ddbServiceObj.updateItem(updateItemParams);

Delete Item

params deleteItemParams

// Delete Items
ddbServiceObj.deleteItem(deleteItemParams);

Table Exists Using WaitFor

params tableExistsParams

// Check if table exists and is in ACTIVE state using wait for method
ddbServiceObj.tableExistsUsingWaitFor(tableExistsParams);

Table Does not Exists Using WaitFor

params tableNotExistsParams

// Check if table doesn't exists using wait for method
ddbServiceObj.tableNotExistsUsingWaitFor(tableNotExistsParams);

Check Table Exists

params tableExistParams

// Check if table exist in ACTIVE state using describe table method
// If table is being created, then response will be false
ddbServiceObj.checkTableExist(tableExistParams);

Shard Management Apis

Shard Management AvailableShards Table Schema

{
        TableName: "available_shards",
          AttributeDefinitions: [
        {
          AttributeName: "shardName",
          AttributeType: "S"
        },
        {
          AttributeName: "entityType",
          AttributeType: "S"
        },
        {
          AttributeName: "allocationType",
          AttributeType: "N"
        }
      ],
        KeySchema: [
        {
          AttributeName: "shardName",
          KeyType: "HASH"
        }
      ],
        GlobalSecondaryIndexes: [{
        IndexName: "available_shard_entity_type_allocation_type_index",
        KeySchema: [
          {
            AttributeName: "entityType",
            KeyType: 'HASH'
          },
          {
            AttributeName: "allocationType",
            KeyType: 'RANGE'
          }
        ],
        Projection: {
          ProjectionType: 'KEYS_ONLY'
        },
        ProvisionedThroughput: {
          ReadCapacityUnits: 1,
          WriteCapacityUnits: 1
        }
      }],
        ProvisionedThroughput: {
        ReadCapacityUnits: 1,
          WriteCapacityUnits: 1
      }
    }

Shard Management Managed shard table schema

{
      TableName: "managed_shards",
      AttributeDefinitions: [
        {
          AttributeName: "identifier",
          AttributeType: "S"
        },
        {
          AttributeName: "entityType",
          AttributeType: "S"
        }
      ],
      KeySchema: [
        {
          AttributeName: "identifer",
          KeyType: "HASH"
        },
        {
          AttributeName: "entityType",
          KeyType: "RANGE"
        }
      ],
      ProvisionedThroughput: {
        ReadCapacityUnits: 1,
        WriteCapacityUnits: 1
      }
    }

DynamoDB And AutoScaling constructor

DynamoDB params dynamodbConnectionParams
AutoScaling params autoScalingConnectionParams

  const OpenSTStorage = require('@openstfoundation/openst-storage');
  let storage = OpenSTStorage.getInstance( configStrategy );
  let ddbServiceObj = storage.dynamoDBService;
  let shardManagementObj = ddbServiceObj.shardManagement();
  let autoScalingObj = storage.autoScalingService;

Run shard migration

Shard migration params ddbServiceObj and autoScalingObj

// Run Shard Migration
// Created available_shards and managed_shards table
shardManagementObj.runShardMigration(ddbServiceObj, autoScalingObj);

Add shard

addShardParams as JSON params
params.shard_name(String) - Shard name to be added
params.entity_type(String) - Entity type to be assigned to shard

// Add Shard
// Creates item in available_shards table
shardManagementObj.addShard(addShardParams);

Configure shard

configureShardParams as JSON params
params.shard_name(String) - Shard name to be added
params.allocation_type(Enum) - Allocation type of Shard :- if
enabled: Provided shard is available for multiple assignment,
disabled: Provided shard is dedicated shard for single identifier

// Configure Shard
// Configure Enable/Disable allocation type
shardManagementObj.configureShard(configureShardParams);

Assign shard

assignShardParams as JSON params
params.identifier(String) - Identifier to be assigned to shard params.shard_name(String) - Shard name to be assigned
params.entity_type(String) - Entity type of the shard
params.force_assignment(Boolean) - (Optional default: false) Pass true if shard is dedicated and assignment needs to be done.
Note: It should be used in case dedicated shard is assigned first time.

// Assign Shard to an identifier
// Creates entry in managed_shards table
shardManagementObj.assignShard(assignShardParams);

Get Shards By Type

getShardsByTypeParams as JSON params
params.entity_type(String) - Entity type to be assigned to shard
params.shard_type(Enum) - Shard type :- if
   all: give all available shards,
   enabled: Shard is available for multiple assignment,
   disabled: Shard is dedicated for single Id

// Get Shards By Different Types
// Type Values : all/enabled/disabled
shardManagementObj.getShardsByType(getShardsByTypeParams);

Has shard

hasShardParams as JSON params
params.shard_names(Array{String}) - List of shard names to be queried for existence.

// Does this shard exist in available_shards table
shardManagementObj.hasShard(hasShardParams);

Get Managed Shard

managedShardParams as JSON params
params.entity_type(String) - Entity type of the shard to be queried
params.identifiers(Array) - List of Identifiers to be queried

// Get Managed shards
shardManagementObj.getManagedShard(managedShardParams);

Steps for Adding and Configuring New Shard

Call runShardMigration if shard migrations are not done already. ApiRef
This will create available_shards and managed_shards table in DynamoDB.
Create Shard Table. ApiRef
Call addShard api. This will add a item in available_shards table. Shard will be added in disabled state. ApiRef
Call configureShard with allocation_type='enabled' if it's a shared shard. For dedicated shard keep allocation_type='disabled'. ApiRef
Call assignShard to assign shard to a client/identifier. This creates an item in managed_shards table. ApiRef
Call getManagedShard to get shardName for a client/identifier. ApiRef

Auto Scaling Apis

For Parameters description please refer AWS DynamoDB Docs

AutoScaling constructor

params autoScalingConnectionParams

  const OpenSTStorage = require('@openstfoundation/openst-storage');
  let storage = OpenSTStorage.getInstance( configStrategy );
  let autoScalingObj = storage.autoScalingService;

Register Scalable Target

params registerScalableTargetParams

// Registers or updates a scalable target. Scalable target is a resource that Application Auto Scaling can scale out or scale in. After you have registered a scalable target, you can use this operation to update the minimum and maximum values for its scalable dimension.
autoScalingObj.registerScalableTarget(registerScalableTargetParams);

Put Scaling Policy

params putScalingPolicyParams

// Creates or updates a policy for an Application Auto Scaling scalable target
autoScalingObj.putScalingPolicy(putScalingPolicyParams);

Deregister Scalable Target

params deregisterScalableTargetParams

// Deregistering a scalable target deletes the scaling policies that are associated with it.
autoScalingObj.deregisterScalableTarget(deregisterScalableTargetParams);

Delete Scaling Policy

params deleteScalingPolicyParams

// Deletes the specified Application Auto Scaling scaling policy
autoScalingObj.deleteScalingPolicy(deleteScalingPolicyParams);

Describe Scalable Targets

params describeScalableTargetsParams

// Gets information about the scalable targets in the specified namespace. 
autoScalingObj.describeScalableTargets(describeScalableTargetsParams);

Describe Scaling Policies

params describeScalingPoliciesParams

// Describes the scaling policies for the specified service namespace.
autoScalingObj.describeScalingPolicies(describeScalingPoliciesParams);

abhayks1/openst-storage