/database-lab-engine

DBLab enables 🖖 database branching and ⚡️ thin cloning for any Postgres database and empowers DB testing in CI/CD. This optimizes database-related costs while improving time-to-market and software quality. Follow to stay updated.

Primary LanguageTypeScriptApache License 2.0Apache-2.0


Deutsch | Português (BR) | Русский | Español | Українська

DBLab Engine

twitter
⚡ Blazing-fast Postgres cloning and branching 🐘

🛠️ Build powerful dev/test environments.
🔃 Cover 100% of DB migrations with CI tests.
💡 Quickly verify ChatGPT ideas to get rid of hallucinations.

Available for any PostgreSQL, including self-managed and managed* like AWS RDS, GCP CloudSQL, Supabase, Timescale.

Can be installed and used anywhere: all clouds and on-premises.

Latest release

CI pipeline status Go report DepShield Badge

Contributor Covenant Community Slack Twitter Follow


*For managed PostgreSQL cloud services like AWS RDS or Heroku, direct physical connection and PGDATA access aren't possible. In these cases, DBLab should run on a separate VM within the same region. It will routinely auto-refresh its data, effectively acting as a database-as-a-service solution. This setup then offers thin database branching ideal for development and testing.

Why DBLab?

  • Build dev/QA/staging environments using full-scale, production-like databases.
  • Provide temporary full-size database clones for SQL query analysis and optimization (see also: SQL optimization chatbot Joe).
  • Automatically test database changes in CI/CD pipelines, minimizing risks of production incidents.
  • Rapidly validate ChatGPT or other LLM concepts, check for hallucinations, and iterate towards effective solutions.

For example, cloning a 1 TiB PostgreSQL database takes just about 10 seconds. On a single machine, you can have dozens of independent clones running simultaneously, supporting extensive development and testing activities without any added hardware costs.

Try it yourself right now:

How it works

Thin cloning is fast because it is based on Copy-on-Write (CoW). DBLab employs two technologies for enabling thin cloning: ZFS (default) and LVM.

Using ZFS, DBLab routinely takes new snapshots of the data directory, managing a collection of them and removing old or unused ones. When requesting a fresh clone, users have the option to select their preferred snapshot.

Read more:

Where to start

Case studies

Features

  • Speed & scale
    • Blazing-fast cloning of Postgres databases – clone in seconds, irrespective of database size
    • Theoretical max of snapshots/clones: 264 (ZFS, default)
    • Maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2128 bytes (ZFS, default)
  • Support & technologies
    • Supported PostgreSQL versions: 9.6–17
    • Thin cloning (CoW) technologies: ZFS and LVM
    • UI for manual tasks and API & CLI for automation
    • Packaged in Docker containers for all components
  • Postgres containers
    • Popular extensions including contrib modules, pgvector, HypoPG and many others (docs)
    • Customization capabilities for containers (docs)
    • Docker container and Postgres config parameters in DBLab config
  • Source database requirements
    • Location flexibility: self-managed Postgres, AWS RDS, GCP CloudSQL, Azure, etc. No source adjustments needed
    • No ZFS or Docker requirements for source databases
  • Data provisioning & retrieval
    • Physical (pg_basebackup, WAL-G, pgBackRest) and logical (dump/restore) provisioning
    • Partial data retrieval in logical mode (specific databases/tables)
    • Continuous update in physical mode
    • Periodic full refresh in logical mode without downtime
  • Recovery & management
    • Fast Point in Time Recovery (PITR) for physical mode
    • Auto-deletion of unused clones
    • Snapshot retention policies in DBLab configuration
  • Clones
    • "Deletion protection" for preventing clone deletion
    • Persistent clones withstand DBLab restarts
    • "Reset" command for data version switching
    • Resource quotas: CPU, RAM
  • Monitoring & security
    • /healthz API endpoint (no auth), extended /status endpoint (API docs)
    • Netdata module for insights

How to contribute

Support us on GitHub/GitLab

The simplest way to show your support is by giving us a star on GitHub or GitLab! ⭐

Add a star

Spread the word

  • Shoot out a tweet and mention @Database_Lab
  • Share this repo's link on your favorite social media platform

Share your experience

If DBLab has been a vital tool for you, tell the world about your journey. Use the logo from the ./assets folder for a visual touch. Whether it's in documents, presentations, applications, or on your website, let everyone know you trust and use DBLab.

HTML snippet for lighter backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-light-background.svg" />
</a>

For darker backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-dark-background.svg" />
</a>

Propose an idea or report a bug

Check out our contributing guide for more details.

Participate in development

Check out our contributing guide for more details.

Reference guides

How-to guides

More you can find in the "How-to guides" section of the docs.

Miscellaneous

License

DBLab source code is licensed under the OSI-approved open source license Apache 2.0.

Reach out to the Postgres.ai team if you want a trial or commercial license that does not contain the GPL clauses: Contact page.

Community & Support

Contributor Covenant

Many thanks to our amazing contributors!

Translations

Making DBLab more accessible to engineers around the globe is a great help for the project. Check details in the translation section of contributing guide.

This README is available in the following translations:

👉 How to make a translation contribution