the volume for host logstash configuration folder does not match the container folder.

Question

the volume for host logstash configuration folder does not match the container folder.

jcastillo725 opened this issue 4 years ago · 17 comments

Describe the bug
I followed the steps for setting up PFELK on docker but I don't see logs coming in. It looks like the correct container folder for .conf files is in "/usr/share/logstash/pipeline" so I updated the volume in the docker compose yml file and was able to get logs but logstash shuts down after a few seconds.

Original volume container folder: /etc/pfelk/conf.d:ro
What I changed it to: /usr/share/logstash/pipeline:ro

To Reproduce
Steps to reproduce the behavior:

Install fresh ELK on docker using latest version

Screenshots
If applicable, add screenshots to help explain your problem.

Operating System (please complete the following information):

OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
Linux 5.8.0-48-generic x86_64
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Version of Docker (docker -v): Docker version 19.03.8, build afacb8b7f0
Version of Docker-Compose (docker-compose -v): docker-compose version 1.25.0, build unknown

Elasticsearch, Logstash, Kibana (please complete the following information):

Version of ELK (cat /docker-pfelk/.env) 7.11

**Service logs

docker-compose logs pfelk01
docker-compose logs pfelk02
docker-compose logs pfelk03
docker-compose logs logstash
logstash | Using bundled JDK: /usr/share/logstash/jdk
logstash | Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
logstash | [INFO ] 2021-03-24 12:15:02.768 [main] runner - Starting Logstash {"logstash.version"=>"7.11.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +jit [linux-x86_64]"}
logstash | [INFO ] 2021-03-24 12:15:02.844 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
logstash | [INFO ] 2021-03-24 12:15:02.878 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
logstash | [INFO ] 2021-03-24 12:15:04.238 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"e49647e6-91e5-4042-bace-5479b6fe76c0", :path=>"/usr/share/logstash/data/uuid"}
logstash | [WARN ] 2021-03-24 12:15:05.083 [LogStash::Runner] pipelineregisterhook - Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
logstash | Please configure Metricbeat to monitor Logstash. Documentation can be found at:
logstash | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
logstash | [WARN ] 2021-03-24 12:15:05.860 [LogStash::Runner] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash | [INFO ] 2021-03-24 12:15:07.623 [LogStash::Runner] licensereader - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
logstash | [WARN ] 2021-03-24 12:15:08.269 [LogStash::Runner] licensereader - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://es01:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://es01:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash | [WARN ] 2021-03-24 12:15:08.436 [LogStash::Runner] licensereader - Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://es01:9200/][Manticore::SocketException] Connection refused (Connection refused) {:url=>http://es01:9200/, :error_message=>"Elasticsearch Unreachable: [http://es01:9200/][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash | [ERROR] 2021-03-24 12:15:08.465 [LogStash::Runner] licensereader - Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://es01:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash | [ERROR] 2021-03-24 12:15:08.512 [LogStash::Runner] internalpipelinesource - Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
logstash | [INFO ] 2021-03-24 12:15:08.641 [Agent thread] configpathloader - No config files found in path {:path=>"/etc/pfelk/conf.d/.conf"}
logstash | [ERROR] 2021-03-24 12:15:08.643 [Agent thread] sourceloader - No configuration found in the configured sources.
logstash | [INFO ] 2021-03-24 12:15:09.042 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
logstash | [INFO ] 2021-03-24 12:15:13.812 [LogStash::Runner] runner - Logstash shut down.
logstash | Using bundled JDK: /usr/share/logstash/jdk
logstash | Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
logstash | [INFO ] 2021-03-24 12:16:03.525 [main] runner - Starting Logstash {"logstash.version"=>"7.11.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +jit [linux-x86_64]"}
logstash | [WARN ] 2021-03-24 12:16:04.090 [LogStash::Runner] pipelineregisterhook - Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
logstash | Please configure Metricbeat to monitor Logstash. Documentation can be found at:
logstash | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
logstash | [WARN ] 2021-03-24 12:16:04.260 [LogStash::Runner] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash | [INFO ] 2021-03-24 12:16:04.738 [LogStash::Runner] licensereader - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
logstash | [WARN ] 2021-03-24 12:16:04.975 [LogStash::Runner] licensereader - Restored connection to ES instance {:url=>"http://es01:9200/"}
logstash | [INFO ] 2021-03-24 12:16:05.278 [LogStash::Runner] licensereader - ES Output version determined {:es_version=>7}
logstash | [WARN ] 2021-03-24 12:16:05.279 [LogStash::Runner] licensereader - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
logstash | [INFO ] 2021-03-24 12:16:05.358 [LogStash::Runner] internalpipelinesource - Monitoring License OK
logstash | [INFO ] 2021-03-24 12:16:05.359 [LogStash::Runner] internalpipelinesource - Validated license for monitoring. Enabling monitoring pipeline.
logstash | [INFO ] 2021-03-24 12:16:05.403 [Agent thread] configpathloader - No config files found in path {:path=>"/etc/pfelk/conf.d/.conf"}
logstash | [INFO ] 2021-03-24 12:16:06.457 [Converge PipelineAction::Create<.monitoring-logstash>] Reflections - Reflections took 53 ms to scan 1 urls, producing 23 keys and 47 values
logstash | [WARN ] 2021-03-24 12:16:06.734 [Converge PipelineAction::Create<.monitoring-logstash>] elasticsearchmonitoring - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash | [INFO ] 2021-03-24 12:16:06.769 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
logstash | [WARN ] 2021-03-24 12:16:06.775 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Restored connection to ES instance {:url=>"http://es01:9200/"}
logstash | [INFO ] 2021-03-24 12:16:06.785 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - ES Output version determined {:es_version=>7}
logstash | [WARN ] 2021-03-24 12:16:06.785 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
logstash | [INFO ] 2021-03-24 12:16:06.857 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://es01:9200"]}
logstash | [WARN ] 2021-03-24 12:16:06.858 [[.monitoring-logstash]-pipeline-manager] javapipeline - 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
logstash | [INFO ] 2021-03-24 12:16:06.910 [[.monitoring-logstash]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x60e35d64@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
logstash | [INFO ] 2021-03-24 12:16:08.002 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.08}
logstash | [INFO ] 2021-03-24 12:16:08.010 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>".monitoring-logstash"}
logstash | [INFO ] 2021-03-24 12:16:08.046 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[]}
logstash | [INFO ] 2021-03-24 12:16:08.199 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
logstash | [INFO ] 2021-03-24 12:16:09.862 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
logstash | [INFO ] 2021-03-24 12:16:10.135 [LogStash::Runner] runner - Logstash shut down.
logstash | Using bundled JDK: /usr/share/logstash/jdk
logstash | Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
logstash | [INFO ] 2021-03-24 12:16:37.265 [main] runner - Starting Logstash {"logstash.version"=>"7.11.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +jit [linux-x86_64]"}
logstash | [WARN ] 2021-03-24 12:16:38.002 [LogStash::Runner] pipelineregisterhook - Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
logstash | Please configure Metricbeat to monitor Logstash. Documentation can be found at:
logstash | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
logstash | [WARN ] 2021-03-24 12:16:38.162 [LogStash::Runner] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash | [INFO ] 2021-03-24 12:16:38.477 [LogStash::Runner] licensereader - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
logstash | [WARN ] 2021-03-24 12:16:38.642 [LogStash::Runner] licensereader - Restored connection to ES instance {:url=>"http://es01:9200/"}
logstash | [INFO ] 2021-03-24 12:16:38.935 [LogStash::Runner] licensereader - ES Output version determined {:es_version=>7}
logstash | [WARN ] 2021-03-24 12:16:38.937 [LogStash::Runner] licensereader - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
logstash | [INFO ] 2021-03-24 12:16:39.035 [LogStash::Runner] internalpipelinesource - Monitoring License OK
logstash | [INFO ] 2021-03-24 12:16:39.039 [LogStash::Runner] internalpipelinesource - Validated license for monitoring. Enabling monitoring pipeline.
logstash | [INFO ] 2021-03-24 12:16:39.072 [Agent thread] configpathloader - No config files found in path {:path=>"/etc/pfelk/conf.d/.conf"}
logstash | [INFO ] 2021-03-24 12:16:40.005 [Converge PipelineAction::Create<.monitoring-logstash>] Reflections - Reflections took 60 ms to scan 1 urls, producing 23 keys and 47 values
logstash | [WARN ] 2021-03-24 12:16:40.177 [Converge PipelineAction::Create<.monitoring-logstash>] elasticsearchmonitoring - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash | [INFO ] 2021-03-24 12:16:40.201 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
logstash | [WARN ] 2021-03-24 12:16:40.209 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Restored connection to ES instance {:url=>"http://es01:9200/"}
logstash | [INFO ] 2021-03-24 12:16:40.223 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - ES Output version determined {:es_version=>7}
logstash | [WARN ] 2021-03-24 12:16:40.226 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
logstash | [INFO ] 2021-03-24 12:16:40.275 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://es01:9200"]}
logstash | [WARN ] 2021-03-24 12:16:40.278 [[.monitoring-logstash]-pipeline-manager] javapipeline - 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
logstash | [INFO ] 2021-03-24 12:16:40.359 [[.monitoring-logstash]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x4f08277@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
logstash | [INFO ] 2021-03-24 12:16:41.464 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.1}
logstash | [INFO ] 2021-03-24 12:16:41.485 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>".monitoring-logstash"}
logstash | [INFO ] 2021-03-24 12:16:41.564 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[]}
logstash | [INFO ] 2021-03-24 12:16:41.689 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
logstash | [INFO ] 2021-03-24 12:16:43.273 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
logstash | [INFO ] 2021-03-24 12:16:43.643 [LogStash::Runner] runner - Logstash shut down.
logstash | Using bundled JDK: /usr/share/logstash/jdk
logstash | Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
logstash | [INFO ] 2021-03-24 12:17:05.330 [main] runner - Starting Logstash {"logstash.version"=>"7.11.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +jit [linux-x86_64]"}
logstash | [WARN ] 2021-03-24 12:17:06.011 [LogStash::Runner] pipelineregisterhook - Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
logstash | Please configure Metricbeat to monitor Logstash. Documentation can be found at:
logstash | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
logstash | [WARN ] 2021-03-24 12:17:06.178 [LogStash::Runner] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash | [INFO ] 2021-03-24 12:17:06.485 [LogStash::Runner] licensereader - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
logstash | [WARN ] 2021-03-24 12:17:06.646 [LogStash::Runner] licensereader - Restored connection to ES instance {:url=>"http://es01:9200/"}
logstash | [INFO ] 2021-03-24 12:17:06.918 [LogStash::Runner] licensereader - ES Output version determined {:es_version=>7}
logstash | [WARN ] 2021-03-24 12:17:06.919 [LogStash::Runner] licensereader - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
logstash | [INFO ] 2021-03-24 12:17:07.003 [LogStash::Runner] internalpipelinesource - Monitoring License OK
logstash | [INFO ] 2021-03-24 12:17:07.005 [LogStash::Runner] internalpipelinesource - Validated license for monitoring. Enabling monitoring pipeline.
logstash | [INFO ] 2021-03-24 12:17:07.041 [Agent thread] configpathloader - No config files found in path {:path=>"/etc/pfelk/conf.d/.conf"}
logstash | [INFO ] 2021-03-24 12:17:07.940 [Converge PipelineAction::Create<.monitoring-logstash>] Reflections - Reflections took 77 ms to scan 1 urls, producing 23 keys and 47 values
logstash | [WARN ] 2021-03-24 12:17:08.095 [Converge PipelineAction::Create<.monitoring-logstash>] elasticsearchmonitoring - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash | [INFO ] 2021-03-24 12:17:08.131 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
logstash | [WARN ] 2021-03-24 12:17:08.141 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Restored connection to ES instance {:url=>"http://es01:9200/"}
logstash | [INFO ] 2021-03-24 12:17:08.150 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - ES Output version determined {:es_version=>7}
logstash | [WARN ] 2021-03-24 12:17:08.150 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
logstash | [INFO ] 2021-03-24 12:17:08.213 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://es01:9200"]}
logstash | [WARN ] 2021-03-24 12:17:08.215 [[.monitoring-logstash]-pipeline-manager] javapipeline - 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
logstash | [INFO ] 2021-03-24 12:17:08.340 [[.monitoring-logstash]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x5395f7d0@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
logstash | [INFO ] 2021-03-24 12:17:09.470 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.13}
logstash | [INFO ] 2021-03-24 12:17:09.479 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>".monitoring-logstash"}
logstash | [INFO ] 2021-03-24 12:17:09.522 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[]}
logstash | [INFO ] 2021-03-24 12:17:09.605 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
logstash | [INFO ] 2021-03-24 12:17:11.211 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
logstash | [INFO ] 2021-03-24 12:17:11.600 [LogStash::Runner] runner - Logstash shut down.
docker-compose logs kibana

Additional context
Add any other context about the problem here.

Answer 1 · 2021-03-24T23:04:17.000Z

Take a look at issue #28 - I plan to reorganize but am also fiddling merging the docker and host installation into one repo.

Answer 2 · 2021-03-25T03:49:22.000Z

Logstash doesn’t see any configuration file on the path “/etc/pfelk/conf.d/*.conf”

logstash | [INFO ] 2021-03-25 03:03:17.111 [Agent thread] configpathloader - No config files found in path {:path=>"/etc/pfelk/conf.d/*.conf"}

The host folder it’s referencing to is “./etc/logstash/conf.d/”
- ./etc/logstash/conf.d/:/etc/pfelk/conf.d:ro

But it does not contain the .conf files.

The .conf files are located in “/etc/pfelk/conf.d” in the host (This is where it was extracted when I unzipped pfelkdocker.zip)

I copied the files to “./etc/logstash/conf.d/” and ran into some filter errors. I copied databases and patterns to the logstash/conf.d folder and got rid of the filter errors but now I'm getting different messages:

elastic@ubuntu:~/pfelk/etc/pfelk/conf.d$ sudo docker attach logstash
[WARN ] 2021-03-25 03:55:35.576 [Converge PipelineAction::Create] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-03-25 03:55:35.614 [Converge PipelineAction::Create] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-03-25 03:55:35.642 [Converge PipelineAction::Create] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-03-25 03:55:35.675 [Converge PipelineAction::Create] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-03-25 03:55:35.710 [Converge PipelineAction::Create] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-03-25 03:55:35.736 [Converge PipelineAction::Create] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-03-25 03:55:35.772 [Converge PipelineAction::Create] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-03-25 03:55:35.789 [Converge PipelineAction::Create] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2021-03-25 03:55:35.800 [Converge PipelineAction::Create] elasticsearch - Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2021-03-25 03:55:35.836 [[pfelk]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
[WARN ] 2021-03-25 03:55:35.844 [[pfelk]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es01:9200/"}
[INFO ] 2021-03-25 03:55:35.860 [[pfelk]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2021-03-25 03:55:35.860 [[pfelk]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2021-03-25 03:55:35.917 [[pfelk]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es01:9200"]}
[INFO ] 2021-03-25 03:55:35.938 [[pfelk]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
[WARN ] 2021-03-25 03:55:35.957 [[pfelk]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es01:9200/"}
[INFO ] 2021-03-25 03:55:35.984 [[pfelk]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2021-03-25 03:55:36.002 [[pfelk]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2021-03-25 03:55:36.057 [[pfelk]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es01:9200"]}
[INFO ] 2021-03-25 03:55:36.064 [[pfelk]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
[WARN ] 2021-03-25 03:55:36.078 [[pfelk]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es01:9200/"}
[INFO ] 2021-03-25 03:55:36.089 [[pfelk]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2021-03-25 03:55:36.105 [[pfelk]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2021-03-25 03:55:36.154 [[pfelk]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es01:9200"]}
[INFO ] 2021-03-25 03:55:36.159 [[pfelk]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
[WARN ] 2021-03-25 03:55:36.174 [[pfelk]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es01:9200/"}
[INFO ] 2021-03-25 03:55:36.201 [[pfelk]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2021-03-25 03:55:36.208 [[pfelk]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2021-03-25 03:55:36.275 [[pfelk]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es01:9200"]}
[INFO ] 2021-03-25 03:55:36.283 [[pfelk]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
[WARN ] 2021-03-25 03:55:36.317 [[pfelk]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es01:9200/"}
[INFO ] 2021-03-25 03:55:36.323 [[pfelk]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2021-03-25 03:55:36.324 [[pfelk]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2021-03-25 03:55:36.369 [[pfelk]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es01:9200"]}
[INFO ] 2021-03-25 03:55:36.379 [[pfelk]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
[WARN ] 2021-03-25 03:55:36.402 [[pfelk]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es01:9200/"}
[INFO ] 2021-03-25 03:55:36.433 [[pfelk]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2021-03-25 03:55:36.442 [[pfelk]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2021-03-25 03:55:36.471 [[pfelk]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es01:9200"]}
[INFO ] 2021-03-25 03:55:36.477 [[pfelk]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
[WARN ] 2021-03-25 03:55:36.488 [[pfelk]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es01:9200/"}
[INFO ] 2021-03-25 03:55:36.508 [[pfelk]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2021-03-25 03:55:36.509 [[pfelk]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2021-03-25 03:55:36.551 [[pfelk]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es01:9200"]}
[INFO ] 2021-03-25 03:55:36.556 [[pfelk]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
[WARN ] 2021-03-25 03:55:36.580 [[pfelk]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es01:9200/"}
[INFO ] 2021-03-25 03:55:36.606 [[pfelk]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2021-03-25 03:55:36.606 [[pfelk]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2021-03-25 03:55:36.651 [[pfelk]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es01:9200"]}
[INFO ] 2021-03-25 03:55:36.656 [[pfelk]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es01:9200/]}}
[WARN ] 2021-03-25 03:55:36.677 [[pfelk]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es01:9200/"}
[INFO ] 2021-03-25 03:55:36.683 [[pfelk]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2021-03-25 03:55:36.683 [[pfelk]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2021-03-25 03:55:36.737 [[pfelk]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es01:9200"]}
[INFO ] 2021-03-25 03:55:36.779 [Ruby-0-Thread-31: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.1-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:130] elasticsearch - Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[INFO ] 2021-03-25 03:55:36.832 [Ruby-0-Thread-31: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.1-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:130] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@Version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[INFO ] 2021-03-25 03:55:36.974 [[pfelk]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-City.mmdb"}
[INFO ] 2021-03-25 03:55:37.049 [[pfelk]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-City.mmdb"}
[INFO ] 2021-03-25 03:55:37.125 [[pfelk]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-ASN.mmdb"}
[INFO ] 2021-03-25 03:55:37.286 [[pfelk]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-ASN.mmdb"}
[ERROR] 2021-03-25 03:55:37.321 [[pfelk]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"pfelk", :exception=>#<LogStash::Filters::Dictionary::DictionaryFileError: Translate: Missing or stray quote in line 1 when loading dictionary file at /etc/pfelk/databases/service-names-port-numbers.csv>, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1899:in block in shift'", "org/jruby/RubyArray.java:1809:in each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1867:in block in shift'", "org/jruby/RubyKernel.java:1442:in loop'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1821:in shift'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/csv_file.rb:20:in block in read_file_into_dictionary'", "org/jruby/RubyIO.java:3511:in foreach'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/csv_file.rb:18:in read_file_into_dictionary'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:101:in merge_dictionary'", "org/jruby/RubyMethod.java:115:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:66:in load_dictionary'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:53:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:19:in create'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/translate.rb:166:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in block in register_plugins'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:586:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in block in start'"], "pipeline.sources"=>["/etc/pfelk/conf.d/01-inputs.conf", "/etc/pfelk/conf.d/02-types.conf", "/etc/pfelk/conf.d/03-filter.conf", "/etc/pfelk/conf.d/05-apps.conf", "/etc/pfelk/conf.d/20-interfaces.conf", "/etc/pfelk/conf.d/30-geoip.conf", "/etc/pfelk/conf.d/35-rules-desc.conf", "/etc/pfelk/conf.d/36-ports-desc.conf", "/etc/pfelk/conf.d/37-enhanced_user_agent.conf", "/etc/pfelk/conf.d/38-enhanced_url.conf", "/etc/pfelk/conf.d/45-cleanup.conf", "/etc/pfelk/conf.d/49-enhanced_private.conf", "/etc/pfelk/conf.d/50-outputs.conf"], :thread=>"#<Thread:0x7d02add9@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[INFO ] 2021-03-25 03:55:37.323 [[pfelk]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"pfelk"}
[ERROR] 2021-03-25 03:55:37.331 [Converge PipelineAction::Create] agent - Failed to execute action {:id=>:pfelk, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[INFO ] 2021-03-25 03:55:37.445 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2021-03-25 03:55:38.644 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[INFO ] 2021-03-25 03:55:39.417 [LogStash::Runner] runner - Logstash shut down.

Answer 3 · 2021-03-25T11:32:13.000Z

Where is the docker-compose.yml file located in reference to the pfelk files?

The docker-compose.yml is referencing those paths based on a relative path (i.e. the preceding dot). Based on your previous response your path is /home/elastic/pfelk/ so to get docker-compose to recognize this path the docker compose will need to be placed within /home/elastic/pfelk/

/home/elastic/pfelk/
├── docker-compose.yml
├── conf.d
│   ├── 01-inputs.conf
│   ├── 02-types.conf
│   ├── 03-filter.conf
│   ├── 05-apps.conf
│   ├── 20-interfaces.conf
│   ├── 30-geoip.conf
│   ├── 35-rules-desc.conf
│   ├── 36-ports-desc.conf
│   ├── 45-cleanup.conf
│   └── 50-outputs.conf
├── config
│   ├── logstash.yml
│   └── pipelines.yml
├── databases
│   ├── private-hostnames.csv
│   ├── rule-names.csv
│   └── service-names-port-numbers.csv
└── patterns
    ├── openvpn.grok
    └── pfelk.grok

Otherwise and alternatively you may amend the docker-compose.yml to specify the absolute path:

      - /home/elastic/pfelk/etc/logstash/config/:/usr/share/logstash/config:ro       
      - /home/elastic/pfelk/etc/logstash/conf.d/:/etc/pfelk/conf.d:ro
      - /home/elastic/pfelk/etc/logstash/conf.d/patterns/:/etc/pfelk/patterns:ro
      - /home/elastic/pfelk/etc/logstash/conf.d/databases/:/etc/pfelk/databases:ro

Note: the preceding dot was omitted, specifying the absolute path versus relative path

Linux Paths:
/ absolute path
. Relative path - current directory
.. Relative path - parent directory

Answer 4 · 2021-03-25T16:42:55.000Z

But the extracted the folder /etc/logstash does not contain the subfolder conf.d

The directory /home/elastic/pfelk/ is what I created to unzip the file and the created directory looks more like:

/home/elastic/pfelk/
├── docker-compose.yml
├── elasticsearch
│ ├── Dockerfile
├── kibana
│ ├── Dockerfile
├── logstash
│ ├── Dockerfile
├── etc
│ ├── logstash
│ │ ├── config
│ │ │ ├── logstash.yml
│ │ │ └── pipelines.yml
│ ├── pfelk
│ │ ├── conf.d
│ │ │ ├── 01-inputs.conf
│ │ │ ├── 02-types.conf
│ │ │ ├── 03-filter.conf
│ │ │ ├── 05-apps.conf
│ │ │ ├── 20-interfaces.conf
│ │ │ ├── 30-geoip.conf
│ │ │ ├── 35-rules-desc.conf
│ │ │ ├── 36-ports-desc.conf
│ │ │ ├── 45-cleanup.conf
│ │ │ └── 50-outputs.conf
│ │ ├── databases
│ │ │ ├── private-hostnames.csv
│ │ │ ├── rule-names.csv
│ │ │ └── service-names-port-numbers.csv
│ │ ├── patterns
│ │ │ ├── openvpn.grok (Missing)
│ │ │ └── pfelk.grok

so the docker-compes.yml is indeed in /home/elastic/pfelk together with the other extracted folders described above but the .conf.d folder is not in the ./etc/logstash folder but in ./etc/pfelk while the pipeline.yml is pointing it to ./etc/logstash.

btw please forgive me for my stubbornness and thanks for your patience... I know there's just something that I still don't understand with what you're explaining but can't see it yet =(

Answer 5 · 2021-03-25T21:07:28.000Z

Got it...the docker-compose.yml had an incorrect reference which was corrected.

I would download or update to the latest docker-compose.yml and try again - sorry for the inconvenience.

The pipelines

Answer 6 · 2021-03-26T03:49:14.000Z

Not an inconvenience at all bro . The work you're putting on this is awesome!

The last time I copied the files to what I thought to be the correct references Logstash shutdown again with this error:

[ERROR] 2021-03-25 03:55:37.331 [Converge PipelineAction::Create] agent - Failed to execute action {:id=>:pfelk, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}

The whole log is in my 2nd comment. I will try again with the updated zip file later or tomorrow.

Answer 7 · 2021-03-26T05:15:45.000Z

Just a few more minor adjustments for the following:

      - ./etc/pfelk/conf.d/patterns/:/etc/pfelk/patterns:ro
      - ./etc/pfelk/conf.d/databases/:/etc/pfelk/databases:ro

[ERROR] 2021-03-26 04:52:51.453 [Converge PipelineAction::Create] translate - Invalid setting for translate filter plugin:

filter {
translate {
# This setting must be a path
# File does not exist or cannot be opened /etc/pfelk/databases/rule-names.csv
dictionary_path => "/etc/pfelk/databases/rule-names.csv"
...
}
}

The databases and patterns folders within conf.d do not have the files.. The files are in the databases and patterns folders with pfelk folder alongside the conf.d folder.

So I copied the files to where they are referencing and seems to wrok but got this new error now:

[ERROR] 2021-03-26 05:07:13.699 [[pfelk]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"pfelk", :exception=>#<LogStash::Filters::Dictionary::DictionaryFileError: Translate: Missing or stray quote in line 1 when loading dictionary file at /etc/pfelk/databases/service-names-port-numbers.csv>, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1899:in block in shift'", "org/jruby/RubyArray.java:1809:in each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1867:in block in shift'", "org/jruby/RubyKernel.java:1442:in loop'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1821:in shift'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/csv_file.rb:20:in block in read_file_into_dictionary'", "org/jruby/RubyIO.java:3511:in foreach'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/csv_file.rb:18:in read_file_into_dictionary'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:101:in merge_dictionary'", "org/jruby/RubyMethod.java:115:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:66:in load_dictionary'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:53:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:19:in create'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/translate.rb:166:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in block in register_plugins'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:586:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in block in start'"], "pipeline.sources"=>["/etc/pfelk/conf.d/01-inputs.conf", "/etc/pfelk/conf.d/02-types.conf", "/etc/pfelk/conf.d/03-filter.conf", "/etc/pfelk/conf.d/05-apps.conf", "/etc/pfelk/conf.d/20-interfaces.conf", "/etc/pfelk/conf.d/30-geoip.conf", "/etc/pfelk/conf.d/35-rules-desc.conf", "/etc/pfelk/conf.d/36-ports-desc.conf", "/etc/pfelk/conf.d/37-enhanced_user_agent.conf", "/etc/pfelk/conf.d/38-enhanced_url.conf", "/etc/pfelk/conf.d/45-cleanup.conf", "/etc/pfelk/conf.d/49-enhanced_private.conf", "/etc/pfelk/conf.d/50-outputs.conf"], :thread=>"#<Thread:0x652a68db@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}

PS: Can you check the new pfelkdocker.zip. It looks like it still has the old docker-compose.yml volumes

Answer 8 · 2021-03-26T12:05:35.000Z

The zip file was updated and I just tested on my system - working w/o issues.

Answer 9 · 2021-03-26T15:50:21.000Z

hmm that's weird. I downloaded the new one, unzipped and still had to copy the databases and patterns from /pflek to /pfelk/conf.d. Anyway I was still having the errors with translate on service-names-port-numbers.csv and rule-names.csv. I ended up removing filters 35 and 36 and it worked.

I think the errors were weird though because I checked both CSVs and there were no extra quotation marks (") whatsoever. It's working now but unfortunately, I can't use the 2 enrichments but I guess its really ok.

See logs and screenshots below for reference:

Errors:

Error: Translate: Missing or stray quote in line 1 when loading dictionary file at /etc/pfelk/databases/service-names-port-numbers.csv>, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1899:in block in shift'", "org/jruby/RubyArray.java:1809:in each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1867:in block in shift'", "org/jruby/RubyKernel.java:1442:in loop'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1821:in shift'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/csv_file.rb:20:in block in read_file_into_dictionary'", "org/jruby/RubyIO.java:3511:in foreach'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/csv_file.rb:18:in read_file_into_dictionary'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:101:in merge_dictionary'", "org/jruby/RubyMethod.java:115:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:66:in load_dictionary'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:53:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:19:in create'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/translate.rb:166:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in block in register_plugins'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:586:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in block in start'"], "pipeline.sources"=>["/etc/pfelk/conf.d/01-inputs.conf", "/etc/pfelk/conf.d/02-types.conf", "/etc/pfelk/conf.d/03-filter.conf", "/etc/pfelk/conf.d/05-apps.conf", "/etc/pfelk/conf.d/20-interfaces.conf", "/etc/pfelk/conf.d/30-geoip.conf", "/etc/pfelk/conf.d/35-rules-desc.conf", "/etc/pfelk/conf.d/36-ports-desc.conf", "/etc/pfelk/conf.d/37-enhanced_user_agent.conf", "/etc/pfelk/conf.d/38-enhanced_url.conf", "/etc/pfelk/conf.d/45-cleanup.conf", "/etc/pfelk/conf.d/49-enhanced_private.conf", "/etc/pfelk/conf.d/50-outputs.conf"], :thread=>"#<Thread:0x75350788@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}

[ERROR] 2021-03-26 15:24:19.829 [[pfelk]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"pfelk", :exception=>#<LogStash::Filters::Dictionary::DictionaryFileError: Translate: Missing or stray quote in line 1 when loading dictionary file at /etc/pfelk/databases/rule-names.csv>, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1899:in block in shift'", "org/jruby/RubyArray.java:1809:in each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1867:in block in shift'", "org/jruby/RubyKernel.java:1442:in loop'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/csv.rb:1821:in shift'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/csv_file.rb:20:in block in read_file_into_dictionary'", "org/jruby/RubyIO.java:3511:in foreach'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/csv_file.rb:18:in read_file_into_dictionary'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:101:in merge_dictionary'", "org/jruby/RubyMethod.java:115:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:66:in load_dictionary'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:53:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/dictionary/file.rb:19:in create'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-translate-3.2.3/lib/logstash/filters/translate.rb:166:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in block in register_plugins'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:586:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in block in start'"], "pipeline.sources"=>["/etc/pfelk/conf.d/01-inputs.conf", "/etc/pfelk/conf.d/02-types.conf", "/etc/pfelk/conf.d/03-filter.conf", "/etc/pfelk/conf.d/05-apps.conf", "/etc/pfelk/conf.d/20-interfaces.conf", "/etc/pfelk/conf.d/30-geoip.conf", "/etc/pfelk/conf.d/35-rules-desc.conf", "/etc/pfelk/conf.d/37-enhanced_user_agent.conf", "/etc/pfelk/conf.d/38-enhanced_url.conf", "/etc/pfelk/conf.d/45-cleanup.conf", "/etc/pfelk/conf.d/49-enhanced_private.conf", "/etc/pfelk/conf.d/50-outputs.conf"], :thread=>"#<Thread:0x68a190ce@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}

Renamed the extensions of filters 35 and 36 from .conf to .copy

Logstash working after the 2 filters were disabled.

Thanks for the support!

Answer 10 · 2021-03-26T17:03:22.000Z

Let me test it again on a fresh instance (purge all docker containers/volumes)... I had issues before with the database look-ups but my current setup is running fine with them.

Answer 11 · 2021-03-27T05:55:37.000Z

Sorry this is off topic, for suricata do I need to install syslog-ng or the logs will be sent to the firewall's system logs? currently, I don't see surricata logs coming in.

Answer 12 · 2021-03-27T13:01:08.000Z

@jcastillo725 - that depends are you running OPNsense or pfSense?

OPNsense is the simplest as it utilizes syslog-ng natively.
pfSense is a bit wonky but this guide should help it getting it configured and setup

Answer 13 · 2021-03-27T15:49:09.000Z

tcp("logstash.local"
do i change this to the ip since its on another host? I think I did but still not getting logs

Answer 14 · 2021-03-27T16:37:48.000Z

This screenshot is from issue #276 running pfSense 2.5.0. I currently use OPNsense but I know multiple people have been able to get it working with the provided wiki instructions.

I would apply the following:}

{
   tcp("logstash.local"
   port(5040)
   );
};

and amend logstash.local with the host or IP of where pfelk is installed.

Answer 15 · 2021-03-28T13:52:01.000Z

I ended up using opnsense instead and got all the logs. Have you had the chance to test the lookups on a fresh instance?

Answer 16 · 2021-03-28T14:11:15.000Z

I am currently running with a fresh instance of:

docker-compose.yml from here
pfelk files from here

I plan to merge the two repos and squash the docker repo but that'll be a future endeavor (i.e. once I have additional free time). Let me know if you need or want assistance with setting up this method.

Answer 17 · 2021-03-28T14:23:51.000Z

And you're not getting any errors for service-names-port-numbers.csv and rule-names.csv? I will try that on another VM but I think we can close this issue now.