yuniql

*** LATEST UPDATE: BETA version v0.465.0 has been released! This release covers all the major features expected to be part of v1.0. Try out with samples described below. - may.27.2020

yuniql (yuu-nee-kel) is an open source schema versioning and database migration engine for SqlServer, PostgreSql and others. Improve your Data Platform DevOps discipline with repeatable deployment, plain SQL scripts, bulk import, integrated CI/CD pipelines, and Docker-based migrations.

Inspired by Evolutionary Database Design by Martin Fowler and Pramod Sadalage.

Why yuniql?

It's raw SQL. Yuniql follows database-first approach in versioning your database. Versions are normal directories or folders. Scripts are series of plain old .sql files. No special tool or language required.
It's .NET Core Native. Released as a self-contained .NET Core 3.0 application. Yuniql doesn't require any dependencies or CLR installed on the developer machine or CI/CD server. For windows, yuniql.exe is ready-for-use on day 1.
Bulk Import CSV. Load up your master data and lookup tables from CSV files. A powerful feature when provisioning fresh developer databases or when taking large block of master data as part of a new version.
DevOps Friendly. Azure Pipeline Tasks available in the Market Place. Use Yuniql task acquires a specific version of the Yuniql. Run Yuniql task runs database migrations with Yuniql CLI using version acquired earlier.
Cloud Ready. Platform tested for Azure SQL Database, Amazon RDS and Google Cloud SQL. Plugins for Snowflake, Aurora and Azure Synapse Analytics are lined up for development.
Docker Support. Each project is prepared for containerized execution using Yuniql base images. A dockerized database project is cheap way to run migration on any CI/CD platform.
Cross-platform. Works with Windows and major Linux distros.
Open Source. Released under Apache License version 2.0. Absolutely free for personal or commercial use.

Working with CLI

Manage local db versions and run database migrations from your CLI tool. Perform local migration run or verify with uncommitted runs to test your scripts. Install yuniql CLI with Chocolatey or use alternative ways listed here https://yuniql.io/docs/install-yuniql

choco install yuniql --version 0.465.0

Run migrations for SQL Server

docker run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=P@ssw0rd!" -p 1400:1433 -d mcr.microsoft.com/mssql/server:2017-latest
SETX YUNIQL_CONNECTION_STRING "Server=localhost,1400;Database=yuniqldb;User Id=SA;Password=P@ssw0rd!"

git clone https://github.com/rdagumampan/yuniql.git c:\temp\yuniql-cli
cd c:\temp\yuniql-cli\samples\basic-sqlserver-sample

yuniql run -a
yuniql info

Run migrations for PostgreSql, MySql and others

docker run -e POSTGRES_USER=sa -e POSTGRES_PASSWORD=P@ssw0rd! -e POSTGRES_DB=yuniqldb -p 5432:5432 postgres
SETX YUNIQL_CONNECTION_STRING "Host=localhost;Port=5432;Username=sa;Password=P@ssw0rd!;Database=yuniqldb"

cd c:\temp\yuniql-cli\samples\basic-postgresql-sample

yuniql run -a --platform postgresql
yuniql info --platform postgresql

Working with Azure DevOps Pipelines Tasks

Run your database migration from Azure DevOps Pipelines. The tasks downloads package and cache it for later execution just like how Use .NET Core or Use Node tasks works. Find Yuniql on Azure DevOps MarketPlace. Developer guide is available here https://yuniql.io/docs/migrate-via-azure-devops-pipelines.

Working with Docker Container

Run your database migration thru a Docker container. This is specially helpful on Linux environments and CI/CD pipelines running on Linux Agents as it facilitates your migration without having to worry any local installations or runtime dependencies. Developer guide is available here https://yuniql.io/docs/migrate-via-docker-container.

git clone https://github.com/rdagumampan/yuniql.git c:\temp\yuniql-docker
cd c:\temp\yuniql-docker\samples\basic-sqlserver-sample

docker build -t sqlserver-example .
docker run sqlserver-example -c "<your-connection-string>" -a --platform sqlserver

Working with ASP.NET Core

Run your database migration when your ASP.NET Core host service starts up. This ensures that database is always at latest compatible state before operating the service. Applies to Worker and WebApp projects. Developer guide is available here https://yuniql.io/docs/migrate-via-aspnetcore-application.

dotnet add package Yuniql.AspNetCore

using Yuniql.AspNetCore;
...
...

//docker run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=P@ssw0rd!" -p 1400:1433 -d mcr.microsoft.com/mssql/server:2017-latest
var traceService = new ConsoleTraceService { IsDebugEnabled = true };
app.UseYuniql(traceService, new YuniqlConfiguration
{
	WorkspacePath = Path.Combine(Environment.CurrentDirectory, "_db"),
	ConnectionString = "Server=localhost,1400;Database=yuniqldb;User Id=SA;Password=P@ssw0rd!",
	AutoCreateDatabase = true,
	Tokens = new List<KeyValuePair<string, string>> {
		new KeyValuePair<string, string>("VwColumnPrefix1","Vw1"),
		new KeyValuePair<string, string>("VwColumnPrefix2","Vw2"),
		new KeyValuePair<string, string>("VwColumnPrefix3","Vw3"),
		new KeyValuePair<string, string>("VwColumnPrefix4","Vw4")
	}
});

Working with Console Application

Run your database migration when Console App starts. Developer guide is available here https://yuniql.io/docs/migrate-via-netcore-console-application.

dotnet add package Yuniql.Core

using Yuniql.Core;
...
...

static void Main(string[] args)
{
	//docker run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=P@ssw0rd!" -p 1400:1433 -d mcr.microsoft.com/mssql/server:2017-latest
	var traceService = new ConsoleTraceService { IsDebugEnabled = true };
	var configuration = new YuniqlConfiguration
	{
		WorkspacePath = Path.Combine(Environment.CurrentDirectory, "_db"),
		ConnectionString = "Server=localhost,1400;Database=yuniqldb;User Id=SA;Password=P@ssw0rd!",
		AutoCreateDatabase = true
	};

	var migrationServiceFactory = new MigrationServiceFactory(traceService);
	var migrationService = migrationServiceFactory.Create();
	migrationService.Initialize(configuration.ConnectionString);
	migrationService.Run(
		configuration.WorkspacePath,
		configuration.TargetVersion,
		configuration.AutoCreateDatabase,
		configuration.Tokens,
		configuration.VerifyOnly,
		configuration.BulkSeparator);
}

Advanced use cases

Contributing & asking for help

Please submit ideas for improvement or report a bug by creating an issue.
Alternatively, tag #yuniql on Twitter or drop me a message rdagumampanATgmail.com.

If this is your first time to participate in an open source initiative, you may look at issues labeled as first timer friendly issues. If you found an interesting case, you can fork this repository, clone to your dev machine, create a local branch, and make Pull Requests (PR) so I can review and merge your changes.

To prepare your dev machine, please visit https://github.com/rdagumampan/yuniql/wiki/Setup-development-environment

Supported databases and platform tests

For running migration from docker container, see instructions here

Platform	Build Status	Description
SqlServer		Sql Server 2017, Azure SQL Database
PostgreSql		PostgreSql v9.6, v12.1
MySql		MySql v5.7, v8.0
Docker image linux-x64		`docker pull rdagumampan/yuniql:linux-x64-latest`
Docker imiage win-x64		`docker pull rdagumampan/yuniql:win-x64-latest`

Amazon RDS Aurora ***
Snowflake Data Warehouse ***
Azure Synapse Analytics (Azure DW)***

*** planned or being evaluated/developer/tested

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Credits

Yuniql relies on many open-source projects and we would like to thanks:

CommandlineParser for CLI commands
CsvTextFieldParser for CSV file parsing
Npgsql for PostgreSql drivers
Shouldly for unit tests
Moq for unit test mocks
Evolutionary database design by Martin Fowler and Pramod Sadalage
Microsoft, Oracle, for everything in dotnetcore seems open source now :)
All the free devops tools! GitHub, AppVeyor, Docker, Shields.io ++

mikependon/yuniql