New functions of clickhouse. Direction of infi

[emakarov]

Hi @ishirav

ClickHouse is moving forward, and in this issue I wanted to understand what is your vision on keeping with CH changes.

In particular, new features of CH (implemented and upcoming) are related to:

• User permissions (GRANT, etc..)
• RBAC control
• more functions, including ML (simple regressions for now but can be much more in future). I've read that work on functions is in progress
• Constraints and templates for data insertions
• ClickHouse internal tables for statistics and performance/monitoring

And many other features

What is your vision on infi.clickhouse_orm.
Should support of various such features be incorporated into it or no?
Is there any criteria what should be included into ORM, what should not be?

[ishirav]

Theoretically, the ORM should cover as many database features as possible. Realistically, the ClickHouse developers are adding features a lot faster than I can keep up with, given the limited time I have for working on this project. Of course, I'm always open to pull requests.

My current plan is to release version 2.0, which will support functions and expressions everywhere - a big change in terms of the ORM's expressiveness. It will also support IPv4 and IPv6 fields, but probably no other new database features (since I don't want to delay the release, better to get it out there). Version 2.0 will no longer support Python 2.7, but should be otherwise backwards-compatible.

After v2.0 I'll work on v2.1+ which will cover new database features. The ones I had in mind are:

• New MergeTree features
• Column comments
• TTL expressions
• Delete rows
• Update rows

Perhaps also:

• Column constraints
• ORDER BY ... WITH FILL

Note that this is a very tentative list, and I'm open to discussion. Generally I implement features by public demand, ease of implementation, having a good fit with the ORM, and apparent usefulness.

As for other database features you mentioned:

• Permissions and RBAC: I don't see any documentation about these, so I don't know yet.
• Templates for insertion: I'll need to look into this, although it seems less useful when working in a Python environment since you can just parse the data using Python and convert it into ORM instances before inserting it.
• Internal tables: it seems to me that the structure of these tables changes often between database versions, so it might be problematic to create models for them. Perhaps I could make something dynamic that builds a model for system tables based on their definition, e.g. you would call database.getSystemModel("part_log") and get a model class that you can use for queries. That might be cool.

[emakarov]

Thanks for such detailed response!

Permissions and RBAC: I don't see any documentation about these, so I don't know yet.

Yes, this is still in progress but they want to release it in December or early 2020

[ishirav]

Some of the features listed above were added to v2.1.0:

• Support for model constraints
• Support for data skipping indexes
• Support for mutations: QuerySet.update and QuerySet.delete