dunmashiku/kusto-loco

C# KQL query engine with flexible I/O layers and visualization

If you are using this project please give it a ⭐ to show your appreciation - thanks!

Kusto-Loco

Kusto-Loco is a set of libraries and applications based around the Kusto Query Language (KQL). KQL is normally used against data held in Azure Data Explorer but Kusto-Loco allows you to query in-memory data held in your own applications.

For a super-quick introduction to KQL see this wiki page but to give you a flavour here's a simple query that calculates the average rating of all PCs in a product table and renders the results as a chart.

products 
| where Category=='PC' 
| join reviews on ProductId 
| summarize Rating=avg(Score) by ProductId,ProductName 
| order by Rating
| project ProductName, Rating
| render columnchart

Kusto-Loco makes it easy to load data from CSV, JSON or Parquet files or from sets of POCOs held in memory. Query results can be serialised back to files or objects or rendered to HTML charts using the Vega-Lite charting library.

Loading data, running a query and rendering the results to a chart can be done in a few lines of code after referencing the appropriate Nuget Packages.

var context = new KustoQueryContext()
                  .CopyDataIntoTable("products", productRows); // load data from a set of POCOs
var result = await context.RunQuery("products | summarize count() by Category | render piechart");

var datatable = result.ToDataTable(); // create a datatable to dump into a datagrid
webview.NavigateToString(KustoResultRenderer.RenderToHtml(result)); //render chart

If you just want to get started playing around with KQL on your own file-based data you can use the supplied LokqlDX application.

Kusto-Loco even comes with a Powershell module that allows you use KQL queries in a Powershell pipeline.

Quick Starts

• For application developers who want to consume the libraries
• LokqlDX - a simple data explorer for local files
• Powershell integration

Status

The current version is 1.0.1. The project is still in active development and core APIs may change. However, the core engine is stable and and is actively used in a production environment. Some KQL operators and functions are not yet implemented.

Credits and Contributors

KustoLoco is a fork of the BabyKusto engine created by DavidNx,Vicky Li and David Nissimoff who appear to have developed the core engine as part of a Microsoft Hackathon.

Since then the engine has been heavily extended and optimised by NeilMacMullen with additional contributions from Vartika Gupta and Kosta Demoore.

The project leans heavily on a number of open source libraries including:

• Benchmark.Net used for performance testing
• CommandLineParser used for command parsing within LokqlDX
• CsvHelper used for CSV serialisation
• Fashenstein use for fuzzy string matching
• FluentAssertions used for testing
• geohash-dotnet for geohash encoding
• NotNullStrings for basic string extensions
• Parquet.Net for Parquet serialisation
• Spectre.Console for nice table rendering
• T-Digest.net is used internally for some aggregation functions
• VegaGenerator is used to simplify rendering of charts

Contributing

If you'd like to help with the project please see the Contributing page.