hotpads/datarouter

Datarouter

Overview

Datarouter is a Java database abstraction layer that gives developers a simple interface to a variety of databases. It sits on top of other database client libraries and encourages access through a standard set of methods allowing an application to mix and match databases while staying independent of any particular database product. The application is programmed against Datarouter’s interfaces while JDBC, Memcached, HBase, etc are left as swappable dependencies. Developers can also choose to code directly against a specific database in performance critical situations or when using unique features of a particular product, still leaving the vast majority of the code in a portable state.

The first goal of Datarouter is to encourage type-safety and early error detection in all layers of your application so that it remains easy to refactor as it grows. Waiting for the results of an integration or even unit test suite is too slow for the frequent refactoring that's needed to quickly evolve an application beyond the original design goals. We want to catch errors at compile time, particularly when the Eclipse incremental compiler runs, and give your IDE the ability to automatically change large amounts of code.

The second goal is to encourage safe interactions with your database, making it easy to fetch data in a way that won't trigger expensive queries and making it harder to accidentally cause large tablescans or joins that can affect the overall application health.

The third goal is the portability mentioned above. Knowing your application is portable from the get-go means you can jump in and start writing code without spending a lot of time debating which database you'll use or which cloud provider you'll be tied to. A program can start out with a traditional relational database like MySQL, and as certain tables grow they can be offloaded to more scalable systems after trivial code changes. Or you might run on multiple systems at the same time, such as a multi-cloud application that uses Aurora, SQS, and Elasticache Memcached in Amazon's cloud while also using Cloud SQL, Cloud Pub/Sub and Memcached in Google's cloud.

For more information about the storage design, see the datarouter-storage README.

Datarouter includes a web framework (datarouter-web) that aims to be lightweight and to rely on Java (not XML, JSON, or plain text) for configuring things like web request handler mappings. Additional modules like datarouter-exception, datarouter-job, datarouter-trace, etc include many building blocks of a web app like log management, authentication, cron triggering, exception recording, counters, etc. Combining the storage and web frameworks and the added utils allows for building portable web apps that are easy to move between different database engines. For example, you could record exception stack traces to MySQL on your laptop, to HBase in your datacenter, DynamoDB when running on AWS, or Bigtable when running on Google Cloud.

Lightweight utils

These libraries can be used individually without importing the whole framework.

• scanner
  • Iterator library with features not present in Stream
• path-node
  • Data structure for defining a tree of strings for URLs and Filesystems
• enum
  • Enum-related utils, particularly MappedEnum
• types
  • A few data types pre-configured for Json and Databeans
• bytes
  • Low level utils for working with bytes
  • Codec interfaces for encoding and decoding objects
• binary-dto
  • A pure-java binary serialization format with sortable encoding
• util
  • Collection of miscellaneous utils

Core framework

Data

• model
  • Core storage classes like PrimaryKey, Databean, and Node
• storage
  • Generic code shared by storage client implementations
• virtual-node
  • Caching, Redundant, and Replication nodes

Web

• http-client
  • Wrapper around Apache Http Client
  • Includes many foundational datarouter web related classes
• web
• auth
  • User and Role system for web applications
• websocket
  • Makes websockets easier to use

Job

• conveyor
  • Run a number of threads in loop, usually consuming a queue
• job
  • Scheduled tasks via cron triggers
• joblet
  • Parallelized tasks stored in a queue+database for enhanced monitoring, restarting, etc
• joblet-mysql
  • Low-volume Joblet locking mechanism implemented in MySql

Additional Features

Server configuration features

• auto-config
  • Run a set of tasks to pre-configure a new service, commonly in dev and staging
• cluster-setting
  • Java-defined configurations per environment/serverType/etc that can be overridden at runtime
• logger-config
  • Dynamic configuration of log levels
• rate-limiter
  • Cache or memory-backed rate limiter with time windows
• secret
  • Interfaces for secret storage
• secret-web
  • UI for editing secrets
• service-config
  • Collects and publishes high-level information about running services

Server monitoring features

• instrumentation
  • Interfaces that datarouter uses to emit internal data
  • The interfaces need to be implemented separately
• changelog
  • Log of administrative actions in a service
• exception
  • Collects and publishes exception details defined by datarouter-instrumentation
• load-test
  • Simulate database traffic
• metric
  • Collects and publishes counts and metrics defined by datarouter-instrumentation
• task-tracker
  • Collect stats from a running task
• trace
  • Collects and publishes tracing data defined by datarouter-instrumentation
• webapp-instance
  • Collects stats about running servers in a service

Server data features

• copy-table
  • Copies databeans from one table to another, usually for migrating from one database to another
• data-export
  • Exports databeans from a table to BlobStorage
• nodewatch
  • Collects table statistics in background joblets and displays them
  • Stores every Nth key in each table enabling parallel processing later
• snapshot
  • File format for storing many objects in BlobStorage with random read access
• snapshot-manager
  • UI for tracking many snapshots and groups of snapshots

Data storage clients

Tables

• mysql
  • Map, Sorted, Indexed, Blob, Tally storage
• gcp-spanner
  • Map, Sorted, Indexed, Blob, Tally storage
• gcp-bigtable
  • Map, Sorted storage

Caches

• memcached
  • Map, Blob, Tally storage
• redis
  • Map, Blob, Tally storage

Messages

• aws-sqs
  • Queue, GroupQueue, BlobQueue storage
• gcp-pubsub
  • Queue, GroupQueue, BlobQueue storage

Blobs

• s3
  • Blob storage
• filesystem
  • Blob storage

Memory

• memory
  • Implement many storage interfaces in memory for high performance
  • A spec for how other clients should behave
  • Map, Sorted, Blob, Queue, GroupQueue, BlobQueue, Tally storage

Wrappers around other libraries

• checkstyle
  • Checkstyle configuration used by datarouter
• email
• gson
  • Gson TypeAdapters used elsewhere in datarouter
• inject
  • Generic injector interface with default Guice implementation
• logging
  • Additions to Log4j2
• mockito
• opencensus
• testng

AWS utils

• aws-alb
  • Application Load Balancer configuration and monitoring
• aws-rds
  • Relational Database Service configuration and monitoring
• aws-secrets-manager
  • Client implmenentation for writing to and reading from AWS Secrets Manager

Parent modules

• metadata-parent
• parent
• full-build

Example webapp

• example
  • Tiny web application with examples of Databeans, Handlers, Settings, Jobs, etc