Create a syntax package for LogQL
cyriltovena opened this issue · 11 comments
Looking at what's coming, I think we should in the future extract the syntax code from the LogQL package and move it to a logql/syntax package so it can be reused by other without having to embed the whole engine. There is a lot places where we need to just parse logql.
This will means also export each supported expression.
/cc @owen-d
Hi! 👋 We would love to have this and we're wondering when / if it'll be available.
We want to parse some LogQL ingressing to a go back-end service. If this isn't going to be delivered anytime soon, is there a current workaround to reliably parse LogQL?
It might be a bit until this is prioritized, but you can take a look at how we parse it in the logql package for now: https://github.com/grafana/loki/blob/master/pkg/logql/parser.go
Having this is a standalone package would be very handy.
I ended up having a go at extracting enough of logcli, and of logql to use in a standard app (our deployment app lets users get logs and stream them during deploys), and though it would be worth leaving some notes here on the experience.
logging client:
- there are deps on logproto and loghttp for some relatively trivial constants, these were easy to refactor out.
- small dependencies on prometheus common/model which were fairly easy to extract out.
logql parser:
This was much harder.
- similar depdencies on loghttp and logproto
- dependencies on promql, which is a core prometheus package, so brings in many of its deps. These were fairly easy to hack out.
- depdency
- The parser and the code for actual evaluation of logql are mixed. The parser also performs some small optimisations that required some depdencies, though I removed those.
- The bigger problem was that the final parser was hard to work with. The interface based approach means the simple usecase I had would have required adding more public methods to the interface, or making some types public.
My use case was, parse a logql expression (possibly just a pipline), and update / add a selector to include some labels grabbed from the deployment config. The selector matches are extractable, as is the pipeline, but there's actually no way to updated them or construct a new query expression.
In the end I'm using the extracted client, but have dropped the logql parser in favour of some crude string munging.
My use case was, parse a logql expression (possibly just a pipline), and update / add a selector to include some labels grabbed from the deployment config. The selector matches are extractable, as is the pipeline, but there's actually no way to updated them or construct a new query expression.
You should be able to parse the AST, inject what you need then use the String function.
you need to:
switch e:= expr.(type) {
case *matchersExpr:
....
case *pipelineExpr:
...
case *vectorAggregationExpr:
case *range....
}
But yeah I understand that those are private. I guess we could change that.
Yes, the fact they are private types was the bulk of the pain, FWIW, the actual type I got back was either a matchersExpr (I think), or a *multiStage. WIthout modifying tthe types, neither was super amenable to having the matcher updated, since both the types and fields are private, and the accessors for the private fields weren't up to the task.
One of the other smaller struggles, but slightly related, was that whilst the overall expresioons (e.g. multistage), have a String method, the response to Matches() and Pipline didn't, so I couldn't stringify them separately either, so I could parse the expression, and then restringify it, but that doesn't win me anything.
As it happens I just look if the user passed in a full query with a matcher (starts with '{'), and then manually trim that add it to the end of my discovered matcher. it's ugly, but it works.
If people just need to validate and update, without the evaluation, my guess is that it's probably easiest to just grab the grammar the the lexer, and build a new package on those, with a simplified AST (more geared to direct manipulation). I don't /think/ it'd be a huge task, but I've not got the time to have go at the moment, maybe later in December.
Appreciated the feedback!
Gonna keep this open as there is some relevance here to helping some new users learn more about Loki, if we don't do this in say the next few months we should close this.
I’ve recently tried something similar as @tcolgate to get (temporary) label based permissions.
I built a prototype logql/syntax package to see how my two use-cases inserting labels and building queries could look like.
I tried reusing just the expr.y and moving the lexer to a separate package, but these seem to be quite intertwined. I think the easiest way to deduplicate the code is the logql packaga just wrapping the logql/syntax package.
Should have some time to finish it up over the next weeks.
Feedback appreciated!