To view as a website, see: http://zhangyushao.site/awesome-database-testing/.
This is a curitated list of resources on database testing.
What covered: papers, blogs, notes, tools and etc. for whoever wants to test a database mangement system (DBMS).
What is not: using SQLs, learning database basic knowledge, implementing a database system.
Note: this is NOT an exhaustive list of materials, you may want to find more accroding to references of each item.
Here we put materials with general discussions.
NOTE: We only include peer-reviewed and published papers here, the same also applies to other entries. You may want to find the authors' free-version of the papers on their personal page.
-
Understanding the query optimization Query Optimization in Database Systems [1984]
-
Understanding the query optimization - a more recent view An Overview of Query Optimization [1998]
-
What are the problems from the indutrial view Testing SQL Server's Query Optimizer : Challenges , Techniques and Experiences
-
Optimizer Evaluation How good are query optimizers, really?
- What are the problems from the academic view Is Query Optimization a “Solved” Problem?
- Query generation with template substitution Generating Thousand Benchmark Queries in Seconds
- Language for customized data generator Flexible Database Generators
- RAGS Microsoft SQL Server Massive Stochastic Testing of SQL
- Generate data regarding the query constraints QAGen: Generating Query-Aware Test Databases
- How could symbolic execution help Qex: Symbolic SQL Query Explorer
- Data generation as a search problem Search-based test data generation for SQL queries
- For more valid inputs! SQUIRREL: Testing Database Management Systems with Language Validity and Coverage Feedback [CCS 2021]
- Randgen MySQL version (not maintained) doc, MariaDB version, PingCAP version
- pquery - stress testing MySQL
- Sysbench - MySQL benchmarks
- SQLsmith - fuzzing style tool
- Squirrel - tool for CCS 2021 paper
- mtr_to_sql.sh@Percona-QA - a script to extract mysql-test cases as a sql file, with engine replacement and shuffles according to your need
Here we mainly refer to the case where the engine returns wrong query result.
- RAGS with system-differential testing Massive Stochastic Testing of SQL
- Data generation with contraints sovling Query-Aware Test Generation Using a Relational Constraint Solver
- Differential testing for the optimizing rules A framework for testing query transformation rules
- Criteria on integrtity testing The Effectiveness of Test Coverage Criteria for Relational Database Schema Integrity Constraints
- Parallelism on the old machines Quickly generating billion-record synthetic databases
- Verified database system Toward a Verified Relational Database Management System
- Reply the transactions Debugging transactions and tracking their provenance with reenactment
- Search-based test data generation for SQL queries [2018]
- Automated verification of query equivalence using satisfiability modulo theories [2019]
- Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction [2020]
- Testing query execution engines with mutations [2020]
- SQLancer@ETH_ZURICH
- go-sqlancer@PingCAP
- pquery@Percona-QA - a tool providing the ability to combine correctness and crash recovery testing on both single-node and clustered instances
Here we focus on detecting and debugging performance issues.
- Very first system Efficient testing of high performance transaction processing systems
- Is the exact estimation possible Exact Cardinality Query Optimization for Optimizer Testing
- How to identify issues on plan generation On the stability of plan costs and the costs of plan stability
- What's wrong with the models Testing cardinality estimation models in SQL server
- Generate and compare the plans Testing the accuracy of query optimizers
- TPC Benchmarking
- APOLLO@gatech
We list idea and tools which are adopted in the industry.
- Chaos Mesh from PingCAP
What you can contribute:
- papers (only peer-reviewed papers will be accepted to this repo)
- talks/slides/articles
- tools
Please also add yourself to Contributors in the PR.