suzuki-shunsuke/go-graylog

System/Grok : Manage grok patterns

Closed this issue · 11 comments

Luubosi commented

would it be possible to add new feature - System/Grok : Manage grok patterns ?
i thing api is ok for this, :9000/api/api-browser/#!/System/Grok and application of this grok in extractor.
Thanks for your work and reply.

suzuki-shunsuke commented

Thank you for your feedback.
I don't know the structure of content_pack.
Do you know it?

http://127.0.0.1:9000/api/api-browser/#!/System/Grok/listGrokPatterns_get_0

    {
      "name": "HTTPD_ERRORLOG",
      "pattern": "%{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}",
      "content_pack": null,
      "id": "5d6a0af36df4af000de157e2"
    }
suzuki-shunsuke commented

I have created the PR #160 .

Luubosi commented

Hi
there are errors, please check. Provider i have 7.1.0

[terragrunt] [C:\graylog-terraform] 2019/09/04 14:58:06 Running command: terraform apply -input=false -auto-approve
?[31m
?[1m?[31mError: ?[0m?[0m?[1mInvalid template control keyword?[0m
?[0m on 007-grok_pattern.tf line 5, in resource "graylog_grok_pattern" "amc_grok_type4":
5: pattern = "ERROR_COUNT=[{]Value=%{?[4mDATA?[0m:ERROR_COUNT}, LAST_REQUEST"
?[0m
"DATA" is not a valid template control keyword.
?[0m?[0m

[terragrunt] [C:\graylog-terraform] 2019/09/04 14:58:53 Running command: terraform apply -input=false -auto-approve
?[31m
?[1m?[31mError: ?[0m?[0m?[1mInvalid template control keyword?[0m
?[0m on 007-grok_pattern.tf line 5, in resource "graylog_grok_pattern" "amc_grok_type4":
5: pattern = "ERROR_COUNT=[{]Value=%{?[4mGREEDYDATA?[0m:ERROR_COUNT}, LAST_REQUEST"
?[0m
"GREEDYDATA" is not a valid template control keyword.
?[0m?[0m

[terragrunt] [C:\graylog-terraform] 2019/09/04 14:59:36 Running command: terraform apply -input=false -auto-approve
?[31m
?[1m?[31mError: ?[0m?[0m?[1mInvalid template control keyword?[0m
?[0m on 007-grok_pattern.tf line 5, in resource "graylog_grok_pattern" "amc_grok_type4":
5: pattern = "%{?[4mDATE?[0m}[- ]%{TIME}"
?[0m
"DATE" is not a valid template control keyword.
?[0m?[0m

suzuki-shunsuke commented

Please share terraform configuration files.
I think your pattern parameter is invalid.

You can test the pattern parameter with Web UI or API Browser.

suzuki-shunsuke commented

To be honest I'm not familiar with the Grok pattern because I don't use this feature.
So if you have any questions about the Grok pattern, please ask at https://community.graylog.org/ .

http://docs.graylog.org/en/3.0/pages/extractors.html#using-grok-patterns-to-extract-data

suzuki-shunsuke commented

I succeeded to create a Grok pattern with the following Terraform configuration.

resource "graylog_grok_pattern" "test" {
  name    = "test"
  pattern = "test"
}
Luubosi commented

Hi, for example.
i use grok pattern.
terraform configuration

resource "graylog_grok_pattern" "amc_grok_type4" {
name = "AMCJSON"
pattern = "test %{TIME}"
}

in new grok pattern you can user grok patterns in list /system/grokpatterns
for example

COMMONAPACHELOG | %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

is using existing grok pattern DATA, NUMBER, USER.

suzuki-shunsuke commented

Oh, I got it. I can reproduce it.

$ cat grok_pattern.tf
resource "graylog_grok_pattern" "test2" {
  name    = "AMCJSON"
  pattern = "test %{TIME}"
}

$ terraform plan

Error: Invalid template control keyword

  on grok_pattern.tf line 3, in resource "graylog_grok_pattern" "test2":
   3:   pattern = "test %{TIME}"

"TIME" is not a valid template control keyword.

You should escape "%".

https://github.com/hashicorp/hcl2/blob/master/hcl/hclsyntax/spec.md#template-literals

resource "graylog_grok_pattern" "test2" {
  name    = "AMCJSON"
  pattern = "test %%{TIME}"
}

The interpolation and directive introductions are escaped by doubling their leading characters.
The ${ sequence is escaped as $${ and the %{ sequence is escaped as %%{.

Luubosi commented

Hi, thanks for clarification. Now it works correct and grok patterns is ok.
thanks for documentation, this is solution - pattern = "%%{DATE}[- ]%%{TIME}"