Using configuration

Question

Using configuration

marcellodesales opened this issue 9 years ago · 30 comments

Hi there,

I'm new to Splunk and the OPS team gave me the following files under the /opt/splunkforwarder/etc/system/local/:

$ ls -la /opt/splunkforwarder/etc/system/local/
total 16
drwx------ 2 root root  74 Jan  9 00:29 .
drwx------ 3 root root  19 Jan  8 22:29 ..
-rw------- 1 root root 171 Jan  8 22:29 inputs.conf
-rwx------ 1 root root 195 Jan  8 22:29 outputs.conf
-r-------- 1 root root 265 Jan  8 22:29 README
-rw------- 1 root root 375 Jan  8 22:29 server.conf

According to http://blogs.splunk.com/2015/08/24/collecting-docker-logs-and-stats-with-splunk/, I'd like to configure a single SplunkForwarder container to collect syslog for all the containers... I'd like also to disclose:

The company below is a BIG Splunk customer, but so far nobody from OPS team supports Docker
We are still on Splunk 6.2.x and so, we cannot use the Docker Native Driver http://blogs.splunk.com/2015/12/16/splunk-logging-driver-for-docker/

This is a host installation with the splunk-forwarder 6.2.x... The content of the files is as follows:

inputs.conf

[root@pe2enpmas300 npmo-server]# cat inputs.conf
[default]
host = pe2enpmas300.corp.company.net

[monitor:///var/log/messages]
disabled = false
sourcetype = syslog
_blacklist = \.(gz)$
index= sp-njsnginx-reference-e2eidx

outputs.conf

[root@pe2enpmas300 npmo-server]# cat outputs.conf
[tcpout]
defaultGroup = primary_indexers

[tcpout:primary_indexers]
server = oe2esstlg310.corp.company.net:9997, oe2esstlg311.corp.company.net:9997, oe2esstlg312.corp.company.net:9997
autoLB = true

server.conf

[root@pe2enpmas300 npmo-server]# cat server.conf
[sslConfig]
sslKeysfilePassword = $1$Of8JPJZlRRS2

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[general]
pass4SymmKey = $1$brNdYNMjDka2
serverName = pe2enpmas300.corp.company.net

Questions

Can I just mount the settings in the Forwarder server?
How can I debug this?

For instance, I just copied the files from the splunk forward from the host and I'm mounting in the data container... Let's start with the container in a separate docker-compose...

docker-compose-monitoring.yml

Since I'm running docker as root (with SSL enabled), then I opened port 514....

splunkforwarder:
  image: outcoldman/splunk:6.2.4-forwarder
  restart: always
  environment:
    - SPLUNK_FORWARD_SERVER="oe2esstlg310.corp.company.net:9997,oe2esstlg311.corp.company.net:9997,oe2esstlg312.corp.company.net:9997"
  ports:
    - 514:514/udp

docker-compose.yml

I just use extensions https://docs.docker.com/compose/extends/ and the ${HOSTNAME} variable substitution https://docs.docker.com/compose/compose-file/#variable-substitution

$ echo $HOSTNAME
pe2enpmas300.corp.company.net

For the configuration, I'm mounting it...

[root@pe2enpmas300 npmo-server]# ls -la monitor/splunk/
total 20
drwx------ 2 polkitd ssh_keys  95 Jan  9 01:00 .
drwx------ 3 root    root      19 Jan  9 01:40 ..
-rw------- 1 polkitd ssh_keys 171 Jan  8 22:29 inputs.conf
-rw------- 1 polkitd ssh_keys  45 Jan  9 01:00 migration.conf
-rwx------ 1 polkitd ssh_keys 195 Jan  8 22:29 outputs.conf
-r-------- 1 polkitd ssh_keys 265 Jan  9 00:51 README
-rw------- 1 polkitd ssh_keys 375 Jan  8 22:29 server.conf

Here's the file Dockerfile

splunkforwarderData:
  image: busybox
  volumes:
    - ./monitor/splunk:/opt/splunk/etc/system/local

splunkforwarder:
  extends:
    file: docker-compose-monitoring.yml
    service: splunkforwarder
  volumes_from:
    - "splunkforwarderData"

newww:
  build: roles/newww
  restart: always
  env_file: .env
  expose:
    - "5005"
  ports:
    - "80:8081"
  log_driver: "syslog"
  log_opt:
    syslog-tag: "newww"
    syslog-address: udp://${HOSTNAME}

Docker inspect command shows the mounted settings...

$ docker compose 
    "Mounts": [
        {
            "Source": "/npmo-data/npmo-server/monitor/splunk",
            "Destination": "/opt/splunk/etc/system/local",
            "Mode": "rw",
            "RW": true
        }
    ],

I verified that the file is in the container as well...

root@pe2enpmas300 npmo-server]# docker exec -ti npmoserver_splunkforwarder_1 bash

root@bb3bb53aba78:/opt/splunk# ls -la
total 96
drwxr-xr-x  9 splunk splunk  4096 Jan  9 09:53 .
drwxr-xr-x  3 root   root      19 Oct 28 14:28 ..
drwxr-xr-x  3 splunk splunk  4096 Jun 26  2015 bin
-r--r--r--  1 splunk splunk    57 Jun 26  2015 copyright.txt
drwxr-xr-x 13 splunk splunk  4096 Jan  9 08:52 etc
drwxr-xr-x  2 splunk splunk    26 Jun 26  2015 include
drwxr-xr-x  4 splunk splunk  4096 Jun 26  2015 lib
-r--r--r--  1 splunk splunk 52503 Jun 26  2015 license-eula.txt
drwxr-xr-x  3 splunk splunk    55 Jun 26  2015 openssl
-r--r--r--  1 splunk splunk   842 Jun 26  2015 README-splunk.txt
drwxr-xr-x  3 splunk splunk    39 Jun 26  2015 share
-r--r--r--  1 splunk splunk 17634 Jun 26  2015 splunkforwarder-6.2.4-271043-linux-2.6-x86_64-manifest
drwxr-xr-x  6 splunk splunk    48 Jan  9 08:51 var

root@bb3bb53aba78:/opt/splunk# ls -la etc/system/local/
total 20
drwx------ 2 splunk splunk  95 Jan  9 09:00 .
drwxr-xr-x 7 splunk splunk  73 Jan  9 08:51 ..
-rw------- 1 splunk splunk 171 Jan  9 06:29 inputs.conf
-rw------- 1 splunk splunk  45 Jan  9 09:00 migration.conf
-rwx------ 1 splunk splunk 195 Jan  9 06:29 outputs.conf
-r-------- 1 splunk splunk 265 Jan  9 08:51 README
-rw------- 1 splunk splunk 375 Jan  9 06:29 server.conf

Any help is appreciated...

outcoldman commented 9 years ago

👍

Answer 1 · 2016-01-11T15:47:12.000Z

Looks like the config you received from Ops does not do what you thought it would do. See this snippet from your inputs.conf:

[monitor:///var/log/messages]

That stanza instructs a Splunk forwarder to tail a file on disk, not listen on a TCP/UDP port. As documented here, to listen on a UDP port, you'll need to do this:

[udp://514]

I suggest going back to your Ops folks to confirm the settings before baking them in.

To your questions:

If I understand what you are asking, yes you can mount a docker volume to effect configuration changes. Mount to the whole $SPLUNK_HOME/etc folder.

Troubleshooting steps. If you have access to Splunk, then you can search it for host=pe2enpmas300*, and failing that, host=pe2enpmas300* index=_internal. If the former sees nothing, you are not collecting data. If the latter sees nothing, then the forwarder is not forwarding data at all, not even the internal debug logs.

If you don't have Splunk access, fix that, because Splunk is cool. In the meantime, inspect the container's /var volume. /var/log/splunk/splunkd.log maps to what you should be seeing in that _internal line above.

Answer 2 · 2016-01-11T15:49:52.000Z

Note that our best practice is to not use Splunk as a syslog server, even though it works fine. For production, we suggest setting up a syslog server (could be in a container...), and have a Splunk Forwarder ingest the syslog files on disk, so going back to using a line like the monitor://... stuff. This excellent blog post lays out the reasons why.

Answer 3 · 2016-01-11T15:53:57.000Z

@marcellodesales at least one issue I see, please take a look on README.md

Splunk processes are running under splunk user.

We are using 1514 instead of standard 514 syslog port because ports below 1024 are reserved for root access only. See Run Splunk Enterprise as a different or non-root user.

I understand that you are running your docker daemon with root, but in the container I start splunk with splunk user. if you really want to use port 514 inside of the container - you should start splunk as root as well, for that you can specify environment variables SPLUNK_USER=root, SPLUNK_GROUP=root.

Also you have not enabled listening on port 514, see http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports

Another small issue - you do not need to specify SPLUNK_FORWARD_SERVER as you specify them on your own in outputs.conf. SPLUNK_FORWARD_SERVER exists only for people who does not have conf files and they want to configure it with this variable.

About the configuration files there are few options:

The right way will be to use Deployment Server, see http://docs.splunk.com/Documentation/Splunk/6.2.8/Updating/Aboutdeploymentserver So you will need only to specify deployment server and forwarder should pick up all the configurations from it. I should probably add a script which will allow to automatically set deployment server in the same way I do SPLUNK_FORWARD_SERVER
You can extend the image, build on top of my image: something like

FROM outcoldman/splunk:forwarder-6.2

COPY inputs.conf $SPLUNK_HOME/etc/system/local/
COPY outputs.conf $SPLUNK_HOME/etc/system/loca/
.... and so on ...

You can just mount to the $SPLUNK_HOME/etc/, don't worry about default settings - they will be copied over on startup.

Debugging

For debugging I would recommend to look inside of the splunkd.log, see

docker exec tail -f $SPLUNK_HOME/var/log/splunk/splunkd.log

Answer 4 · 2016-01-11T20:27:13.000Z

@halr9000 and @outcoldman, thank you so much for all the response here! I will go through today and I will place comments about our findings...

@halr9000
[udp://514]

This one of the reasons why it might not have worked... So, let me try that...

@outcoldman
SPLUNK_USER=root, SPLUNK_GROUP=root

I will be using the following on my ops-services-docker-compose.yml

splunkforwarder:
  image: outcoldman/splunk:6.2.4-forwarder
  restart: always
  ports:
    - 514:514/udp
  environment:
    - SPLUNK_USER=root
    - SPLUNK_GROUP=root

Is that all I need? The processes are now run by root.

[root@pe2enpmas300 npmo-server]# docker exec -ti npmoserver_splunkforwarder_1 bash
root@a59c0fdf35e9:/opt/splunk#

root@a59c0fdf35e9:/opt/splunk# ps aux | grep splunk
root        66  0.1  0.6 209580 99612 ?        Sl   20:19   0:03 splunkd -p 8089 start
root        67  0.0  0.0  53640  7768 ?        Ss   20:19   0:00 [splunkd pid=66] splunkd -p 8089 start [process-runner]
root        97  0.0  0.0  47580  1852 ?        S    20:19   0:00 sudo -HEu root tail -f /opt/splunk/var/log/splunk/splunkd_stderr.log
root        98  0.0  0.0   5952   616 ?        S    20:19   0:00 tail -f /opt/splunk/var/log/splunk/splunkd_stderr.log
root       140  0.0  0.0  10460   940 ?        S+   21:06   0:00 grep --color=auto splunk

I will monitor the splunk logs inside the container... I see that the container connected to the index servers:

$ docker exec -ti npmoserver_splunkforwarder_1 sh
[root@pe2enpmas300 npmo-server]# docker exec -ti npmoserver_splunkforwarder_1 bash
root@a59c0fdf35e9:/opt/splunk# tail -f $SPLUNK_HOME/var/log/splunk/splunkd.log
01-11-2016 20:19:32.399 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/splunkd_stdout.log'.
01-11-2016 20:19:32.400 +0000 INFO  WatchedFile - Will begin reading at offset=183 for file='/opt/splunk/var/log/splunk/splunkd_stderr.log'.
01-11-2016 20:19:32.412 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.115:9997
01-11-2016 20:19:32.432 +0000 INFO  WatchedFile - Will begin reading at offset=955414 for file='/opt/splunk/var/log/splunk/metrics.log'.

I can see it connected to the hosts...

01-10-2016 14:56:43.229 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.129:9997
01-10-2016 14:57:14.093 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.113:9997
01-10-2016 14:57:43.309 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.115:9997

However, I see a few errors around SSL connectivity. Do I need to have the CERTS? or can I disable it?

01-11-2016 20:19:31.787 +0000 INFO  ServerConfig - My hostname is "a59c0fdf35e9".
01-11-2016 20:19:31.797 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
01-11-2016 20:19:31.797 +0000 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
01-11-2016 20:19:31.797 +0000 INFO  ServerConfig - Setting HTTP server compression state=on
01-11-2016 20:19:31.797 +0000 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
01-11-2016 20:19:31.797 +0000 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
01-11-2016 20:19:31.801 +0000 INFO  LicenseMgr - Initing LicenseMgr

Thanks for any help!

Answer 5 · 2016-01-12T01:44:48.000Z

Hi @halr9000 @outcoldman

Good news! The setup worked!!!

My question now is more related to the events... There's a mix of txt and json output... Is there a way to identify and differenciate among the events in syslog?

Thanks a lot

Answer 6 · 2016-01-12T02:13:12.000Z

Nice! Should we close this issue now?

Answer 7 · 2016-01-12T02:35:48.000Z

@outcoldman Actually I have to take it back... There was a splunkforwarder instance running on the host... :'( Here's what I have in the containers so far...

Docker-compose

Mounts the dir /var/log containing the /var/log/messages file. This HOST is writing the docker container logs to /var/log/messages... This is volume-mounted in the docker container splunkforwarder.

splunkforwarderData:
  image: busybox
  volumes:
    - ./monitor/splunk:/opt/splunk/etc/system/local
    - /var/log:/var/log

splunkforwarder:
  image: outcoldman/splunk:6.2.4-forwarder
  restart: always
  environment:
    - SPLUNK_USER=root
    - SPLUNK_GROUP=root
  volumes_from:
    - "splunkforwarderData"

frontdoor:
  build: roles/registry
  restart: always
  env_file: .env
  ports:
    - "8080:8080"
  log_driver: "syslog"
  log_opt:
    syslog-tag: "frontdoor"

Logs

Can you confirm that this output below is good? I cannot see new messages going to splunk anymore :( Is there any other logs I should be looking at? I see in the bottom of the logs that the splunk forwarder is indeed connected to the 2 hosts...

root@f750b9a56059:/opt/splunk# ls -la /var/log/messages
-rw------- 1 root root 2842029 Jan 12 02:14 /var/log/messages
root@f750b9a56059:/opt/splunk# vim
bash: vim: command not found
root@f750b9a56059:/opt/splunk# vi $SPLUNK_HOME/var/log/splunk/splunkd.log
01-12-2016 02:14:18.906 +0000 INFO  ServerConfig - Will generate GUID, as none found on this server.
01-12-2016 02:14:18.907 +0000 INFO  ServerConfig - My newly generated GUID is 0E7035A5-7F3B-42ED-8735-7BEA3435BA74
01-12-2016 02:14:18.907 +0000 INFO  ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-12-2016 02:14:18.907 +0000 INFO  ServerConfig - Found no site defined in server.conf
01-12-2016 02:14:18.907 +0000 INFO  ServerConfig - My hostname is "f750b9a56059".
01-12-2016 02:14:18.917 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFi
nal_ex:bad decrypt.
01-12-2016 02:14:18.917 +0000 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
01-12-2016 02:14:18.917 +0000 INFO  ServerConfig - Setting HTTP server compression state=on
01-12-2016 02:14:18.917 +0000 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
01-12-2016 02:14:18.917 +0000 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
01-12-2016 02:14:18.928 +0000 INFO  LicenseMgr - Initing LicenseMgr
01-12-2016 02:14:18.928 +0000 INFO  LMConfig - serverName=pe2enpmas300.corp.company.net guid=0E7035A5-7F3B-42ED-8735-7BEA3435BA74
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - connection_timeout=30
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - send_timeout=30
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - receive_timeout=30
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - squash_threshold=2000
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - strict_pool_quota=1
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-12-2016 02:14:18.929 +0000 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-12-2016 02:14:18.929 +0000 INFO  LMStackMgr - closing stack mgr
01-12-2016 02:14:18.929 +0000 INFO  LMSlaveInfo - all slaves cleared
01-12-2016 02:14:18.929 +0000 INFO  LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
01-12-2016 02:14:18.929 +0000 INFO  LMStackMgr - added pool auto_generated_pool_free to stack free
01-12-2016 02:14:18.929 +0000 INFO  ServerRoles - Declared role=license_master.
01-12-2016 02:14:18.929 +0000 INFO  LMStackMgr - init completed [0E7035A5-7F3B-42ED-8735-7BEA3435BA74,Forwarder,runContext_splunkd=true]
01-12-2016 02:14:18.929 +0000 INFO  LicenseMgr - StackMgr init complete...
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - init'ing slaveId=0E7035A5-7F3B-42ED-8735-7BEA3435BA74 label=pe2enpmas300.corp.company.net [30,30,self]
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - enabling implicit feature set
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - attempting to ping master=self from slave=0E7035A5-7F3B-42ED-8735-7BEA3435BA74
01-12-2016 02:14:18.930 +0000 INFO  LMSlaveInfo - new slave='0E7035A5-7F3B-42ED-8735-7BEA3435BA74' created
01-12-2016 02:14:18.930 +0000 INFO  LMSlaveInfo - Detected that masterTimeFromSlave(ZERO_TIME) < lastRolloverTime(Tue Jan 12 00:00:00 2016), meaning that the master has a
lready rolled over. Ignore slave persisted usage.
"/opt/splunk/var/log/splunk/splunkd.log" 167 lines, 19388 characters
01-12-2016 02:14:19.128 +0000 INFO  IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db' (No such fi
le or directory).  This is normal when splunk is first starting up.
01-12-2016 02:14:19.130 +0000 WARN  DistributedPeerManager - feature=DistSearch not enabled for your license level
01-12-2016 02:14:19.130 +0000 INFO  IndexProcessor - running splunkd specific init
01-12-2016 02:14:19.131 +0000 INFO  loader - Initializing from configuration
01-12-2016 02:14:19.132 +0000 INFO  PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-12-2016 02:14:19.134 +0000 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-12-2016 02:14:19.134 +0000 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-12-2016 02:14:19.282 +0000 INFO  TcpOutputProc - Initializing with fwdtype=lwf
01-12-2016 02:14:19.282 +0000 INFO  ServerRoles - Declared role=lightweight_forwarder.
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-12-2016 02:14:19.287 +0000 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-12-2016 02:14:19.287 +0000 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-12-2016 02:14:19.287 +0000 INFO  PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-12-2016 02:14:19.287 +0000 INFO  PipelineComponent - Launching the pipelines.
01-12-2016 02:14:19.288 +0000 INFO  loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 02:14:19.288 +0000 INFO  loader - Limiting REST HTTP server to 2646 threads
01-12-2016 02:14:19.293 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFi
nal_ex:bad decrypt.
01-12-2016 02:14:19.293 +0000 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
01-12-2016 02:14:19.293 +0000 ERROR HTTPServer - SSL will not be enabled
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - TailWatcher initializing...
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 02:14:19.342 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 02:14:19.342 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 02:14:19.342 +0000 INFO  TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 02:14:19.342 +0000 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 02:14:19.371 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.113:9997
01-12-2016 02:14:49.131 +0000 WARN  AuthenticationManagerSplunk - Seed file is not present. Defaulting to generic username/pass pair.
01-12-2016 02:14:50.006 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.115:9997
01-12-2016 02:15:50.067 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.129:9997
01-12-2016 02:16:20.096 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.115:9997
01-12-2016 02:17:19.158 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.129:9997

Answer 8 · 2016-01-12T02:44:17.000Z

@marcellodesales looks good for me. I see that you are monitoring /var/log/messages, also see that you could connect to the indexers. Try to less on var/log/messages and see that when you get new messages here you will get them in splunk as well.

I do not thing that it is good idea to mount /var/log into container in the same path, who knows what container can expect from this path (I do not run anything, which depends on it, but I cannot be 100% sure about that).

Answer 9 · 2016-01-12T03:33:11.000Z

@outcoldman I changed the settings to the following...

splunkforwarderData:
  image: busybox
  volumes:
    - ./monitor/splunk:/opt/splunk/etc/system/local
    - /var/log/messages:/var/log/messages

That only loads the /var/log/messages... However, I still don't see anything in Splunk... The files under /opt/splunk/etc/system/local are loaded, correct? Because we see the dir being watched...

From inside the container, I see the logs coming to the /var/log/messages if I tail it... However, nothing is showing up in Splunk, although it says connected... :( Is there anything else I should be looking at?

root@be6fe6af3549:/opt/splunk# tail -f /var/log/messages
Jan 11 19:30:01 pe2enpmas300 systemd: Starting user-0.slice.
Jan 11 19:30:01 pe2enpmas300 systemd: Started Session 1186 of user root.
Jan 11 19:30:01 pe2enpmas300 systemd: Starting Session 1186 of user root.
Jan 11 19:30:01 pe2enpmas300 systemd: Removed slice user-0.slice.
Jan 11 19:30:01 pe2enpmas300 systemd: Stopping user-0.slice.
Jan 11 19:30:01 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:01.996Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:04 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:03.999Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:04 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:04.559Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:03:30:04 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 11 19:30:05 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:05.921Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:03:30:05 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 11 19:30:06 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:06.000Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:08 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:08.003Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:09 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:09.592Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:03:30:09 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 11 19:30:10 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:10.008Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:10 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:10.874Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:03:30:10 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 11 19:30:12 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:12.011Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:14 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:14.012Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:14 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:14.588Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:03:30:14 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 11 19:30:15 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:15.965Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:03:30:15 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 11 19:30:16 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:16.015Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:18 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:18.018Z: 0 queued changes, 0 open requests, 0 retries pending

Answer 10 · 2016-01-12T03:49:27.000Z

The docker-compose logs show that the container tries to create the certs and initializes...

postgres_1            | taching: configured logging reader does not support reading
splunkforwarder_1     |
splunkforwarder_1     | This appears to be your first time running this version of Splunk.
splunkforwarder_1     |
splunkforwarder_1     | Splunk> Now with more code!
splunkforwarder_1     |
splunkforwarder_1     | Checking prerequisites...
nginx_1               | taching: configured logging reader does not support reading
splunkforwarder_1     |     Checking mgmt port [8089]: New certs have been generated in '/opt/splunk/etc/auth'.
elasticsearch_1       | taching: configured logging reader does not support reading
splunkforwarder_1     | Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
splunkforwarder_1     | Couldn't initialize SSL Context for HTTPClient in ServerConfig
splunkforwarder_1     | Generating a 1024 bit RSA private key
splunkforwarder_1     | ...............................++++++
splunkforwarder_1     | ........................................++++++
auth_1                | taching: configured logging reader does not support reading
splunkforwarder_1     | writing new private key to 'privKeySecure.pem'
redis_1               | taching: configured logging reader does not support reading
splunkforwarder_1     | -----
splunkforwarder_1     | Signature ok
splunkforwarder_1     | subject=/CN=be6fe6af3549/O=SplunkUser
splunkforwarder_1     | Getting CA Private Key
splunkforwarder_1     | writing RSA key
splunkforwarder_1     | open
splunkforwarder_1     |         Creating: /opt/splunk/var/lib/splunk
splunkforwarder_1     |         Creating: /opt/splunk/var/run/splunk
splunkforwarder_1     |         Creating: /opt/splunk/var/run/splunk/appserver/i18n
splunkforwarder_1     |         Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
splunkforwarder_1     |         Creating: /opt/splunk/var/run/splunk/upload
splunkforwarder_1     |         Creating: /opt/splunk/var/spool/splunk
splunkforwarder_1     |         Creating: /opt/splunk/var/spool/dirmoncache
splunkforwarder_1     |         Creating: /opt/splunk/var/lib/splunk/authDb
splunkforwarder_1     |         Creating: /opt/splunk/var/lib/splunk/hashDb
splunkforwarder_1     |     Checking conf files for problems...
splunkforwarder_1     |     Done
splunkforwarder_1     | All preliminary checks passed.
splunkforwarder_1     |
splunkforwarder_1     | Starting splunk server daemon (splunkd)...
splunkforwarder_1     | Done
splunkforwarder_1     |
splunkforwarder_1     | 2016-01-12 03:18:38.574 +0000 splunkd started (build 271043)

Then, I can see a few errors in the container

root@be6fe6af3549:/opt/splunk# grep -R "ERROR" var/log/splunk/splunkd.log
01-12-2016 03:18:38.572 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
01-12-2016 03:18:38.572 +0000 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
01-12-2016 03:18:38.931 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
01-12-2016 03:18:38.931 +0000 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
01-12-2016 03:18:38.931 +0000 ERROR HTTPServer - SSL will not be enabled

Maybe this is the reason the container cannot forward the logs?

Answer 11 · 2016-01-12T03:55:29.000Z

@marcellodesales sorry, I don't see anything unusual.

Just few recommendations:

Try the same setup without docker.
Check logs on indexers, if there are anything suspicious
Are you sure that blacklist works?
Are you sure that index exists?

Answer 12 · 2016-01-12T04:07:55.000Z

@outcoldman

The host setup works, that's how I got the screenshot above...
Where can I find the logs on indexes?
As per 1, it works (from the host setup)
As per 1, it works (from the host setup)

The only delta I found was the SSL errors shown on the container... Here's the logs from the HOST without ANY ERROR...

[root@pe2enpmas300 npmo-server]# grep -R "ERROR" /opt/splunkforwarder/var/log/splunk/splunkd.log | wc -l
0


01-08-2016 20:01:27.070 -0800 INFO  ServerConfig - Will generate GUID, as none found on this server.
01-08-2016 20:01:27.070 -0800 INFO  ServerConfig - My newly generated GUID is 6748789D-54FE-4389-B00D-F144248A0683
01-08-2016 20:01:27.071 -0800 INFO  ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-08-2016 20:01:27.071 -0800 INFO  ServerConfig - Found no site defined in server.conf
01-08-2016 20:01:27.071 -0800 INFO  ServerConfig - My hostname is "pe2enpmas300.corp.company.net".
01-08-2016 20:01:27.080 -0800 INFO  ServerConfig - Setting HTTP server compression state=on
01-08-2016 20:01:27.081 -0800 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
01-08-2016 20:01:27.081 -0800 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
01-08-2016 20:01:27.084 -0800 INFO  LicenseMgr - Initing LicenseMgr
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - serverName=pe2enpmas300.corp.company.net guid=6748789D-54FE-4389-B00D-F144248A0683
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - connection_timeout=30
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - send_timeout=30
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - receive_timeout=30
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - squash_threshold=2000
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - strict_pool_quota=1
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-08-2016 20:01:27.084 -0800 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-08-2016 20:01:27.084 -0800 INFO  LMStackMgr - closing stack mgr
01-08-2016 20:01:27.084 -0800 INFO  LMSlaveInfo - all slaves cleared
01-08-2016 20:01:27.140 -0800 INFO  LMConfig - created default pool=auto_generated_pool_forwarder for stack=forwarder
01-08-2016 20:01:27.140 -0800 INFO  LMStackMgr - added default pool=auto_generated_pool_forwarder for stack=forwarder
01-08-2016 20:01:27.142 -0800 INFO  LMConfig - created default pool=auto_generated_pool_free for stack=free
01-08-2016 20:01:27.142 -0800 INFO  LMStackMgr - added default pool=auto_generated_pool_free for stack=free
01-08-2016 20:01:27.142 -0800 INFO  ServerRoles - Declared role=license_master.
01-08-2016 20:01:27.142 -0800 INFO  LMStackMgr - init completed [6748789D-54FE-4389-B00D-F144248A0683,Forwarder,runContext_splunkd=true]
01-08-2016 20:01:27.142 -0800 INFO  LicenseMgr - StackMgr init complete...
01-08-2016 20:01:27.142 -0800 INFO  LMTracker - init'ing slaveId=6748789D-54FE-4389-B00D-F144248A0683 label=pe2enpmas300.corp.company.net [30,30,self]
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - enabling implicit feature set
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - attempting to ping master=self from slave=6748789D-54FE-4389-B00D-F144248A0683
01-08-2016 20:01:27.143 -0800 INFO  LMSlaveInfo - new slave='6748789D-54FE-4389-B00D-F144248A0683' created
01-08-2016 20:01:27.143 -0800 INFO  LMSlaveInfo - Detected that masterTimeFromSlave(ZERO_TIME) < lastRolloverTime(Fri Jan  8 00:00:00 2016), meaning that the master has a
lready rolled over. Ignore slave persisted usage.
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=AdvancedXML state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=Alerting state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=AllowDuplicateKeys state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=CustomRoles state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=DeployClient state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=DeployServer state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=DistSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=GuestPass state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=KVStore state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=LDAPAuth state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=LocalSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=MultisiteClustering state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=NontableLookups state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=RcvData state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=RcvSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=RollingWindowAlerts state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ScheduledAlerts state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ScheduledReports state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ScheduledSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=SearchheadPooling state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=SigningProcessor state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=SplunkWeb state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=SyslogOutputProcessor state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=UnisiteClustering state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - setting masterGuid='6748789D-54FE-4389-B00D-F144248A0683'
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - attempting to contact master=self from slave=6748789D-54FE-4389-B00D-F144248A0683 success
01-08-2016 20:01:27.143 -0800 INFO  LicenseMgr - Tracker init complete...
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: virtual address space size: unlimited
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: data segment size: unlimited
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: resident memory size: unlimited
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited]
01-08-2016 20:01:27.146 -0800 WARN  ulimit - Core file generation disabled
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: data file size: unlimited
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: open files: 4096 files
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: user processes: 63441 processes
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: cpu time: unlimited
01-08-2016 20:01:27.147 -0800 INFO  ulimit - Linux transparent hugetables support, enabled="always" defrag="always"
01-08-2016 20:01:27.150 -0800 INFO  loader - Splunkd starting (build 271043).
01-08-2016 20:01:27.150 -0800 INFO  loader - System info: Linux, pe2enpmas300.corp.company.net, 3.10.0-327.3.1.el7.x86_64, #1 SMP Fri Nov 20 05:40:26 EST 2015, x86_64.
01-08-2016 20:01:27.150 -0800 INFO  loader - Detected 4 (virtual) CPUs, 4 CPU cores, and 15880MB RAM
01-08-2016 20:01:27.150 -0800 INFO  loader - Maximum number of threads (approximate): 7940
01-08-2016 20:01:27.150 -0800 INFO  loader - Arguments are: "-p" "8089" "start"
01-08-2016 20:01:27.150 -0800 INFO  loader - Getting configuration data from: /opt/splunkforwarder/etc/myinstall/splunkd.xml
01-08-2016 20:01:27.151 -0800 INFO  loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunkforwarder/etc/modules
01-08-2016 20:01:27.151 -0800 INFO  loader - loading modules from /opt/splunkforwarder/etc/modules
01-08-2016 20:01:27.152 -0800 INFO  loader - Writing out composite configuration file: /opt/splunkforwarder/var/run/splunk/composite.xml
01-08-2016 20:01:27.156 -0800 INFO  ServerRoles - Declared role=universal_forwarder.
01-08-2016 20:01:27.156 -0800 INFO  BundlesSetup - Setup stats for /opt/splunkforwarder/etc: wallclock_elapsed_msec=6, cpu_time_used=0.004996, shared_services_generation=
1, shared_services_population=1
01-08-2016 20:01:27.178 -0800 INFO  loader - Setting SSL configuration.
01-08-2016 20:01:27.178 -0800 INFO  loader - Server supporting SSL versions SSL3,TLS1.0,TLS1.1,TLS1.2
01-08-2016 20:01:27.178 -0800 INFO  loader - Using cipher suite TLSv1+HIGH:@STRENGTH
01-08-2016 20:01:27.178 -0800 INFO  loader - ECDH curve not configured
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "MonitorNoHandle://" with 2 parameters: disabled, index
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "WinEventLog://" with 30 parameters: start_from, current_only, checkpointInter
val, disabled, evt_resolve_ad_obj, evt_dc_name, evt_dns_name, index, whitelist, blacklist, whitelist1, whitelist2, whitelist3, whitelist4, whitelist5, whitelist6, whiteli
st7, whitelist8, whitelist9, blacklist1, blacklist2, blacklist3, blacklist4, blacklist5, blacklist6, blacklist7, blacklist8, blacklist9, suppress_text, renderXml
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "WinHostMon://" with 4 parameters: type, interval, disabled, index
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "WinNetMon://" with 15 parameters: remoteAddress, process, user, addressFamily
, packetType, direction, protocol, readInterval, driverBufferSize, userBufferSize, mode, multikvMaxEventCount, multikvMaxTimeMs, disabled, index
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "WinPrintMon://" with 4 parameters: type, baseline, disabled, index
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "WinRegMon://" with 7 parameters: proc, hive, type, baseline, baseline_interva
l, disabled, index
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "admon://" with 7 parameters: targetDc, startingNode, monitorSubtree, disabled
, index, printSchema, baseline
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "perfmon://" with 11 parameters: object, counters, instances, interval, mode,
samplingInterval, stats, disabled, index, showZeroValue, useEnglishOnly
01-08-2016 20:01:27.357 -0800 WARN  ClusteringMgr - Ignoring clustering configuration, the active license disables this feature.
01-08-2016 20:01:27.358 -0800 INFO  SHPoolingMgr - initing shpooling with: ht=60 rf=3 ct=60 st=60 rt=60 rct=5 rst=5 rrt=10 rmst=600 rmrt=600 pe=1 im=0 is=0 mor=5 pb=5 rep
_port= pptr=10
01-08-2016 20:01:27.358 -0800 INFO  SHPoolingMgr - shpooling disabled
01-08-2016 20:01:27.358 -0800 INFO  DS_DC_Common - Initializing the PubSub system.
01-08-2016 20:01:27.358 -0800 INFO  DS_DC_Common - Initializing core facilities of PubSub system.
01-08-2016 20:01:27.366 -0800 INFO  DC:DeploymentClient - target-broker clause is missing.
01-08-2016 20:01:27.366 -0800 WARN  DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-08-2016 20:01:27.366 -0800 INFO  DS_DC_Common - Deployment Client not initialized.
01-08-2016 20:01:27.366 -0800 INFO  DS_DC_Common - Deployment Server not available on a dedicated forwarder.
01-08-2016 20:01:27.367 -0800 INFO  IntrospectionGenerator:disk_objects - Enabled: indexes|volumes|dispatch=false fishbucket=true partitions=false
01-08-2016 20:01:27.367 -0800 INFO  IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600s
01-08-2016 20:01:27.367 -0800 INFO  IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db' (N
o such file or directory).  This is normal when splunk is first starting up.
01-08-2016 20:01:27.369 -0800 WARN  DistributedPeerManager - feature=DistSearch not enabled for your license level
01-08-2016 20:01:27.369 -0800 INFO  IndexProcessor - running splunkd specific init
01-08-2016 20:01:27.370 -0800 INFO  loader - Initializing from configuration
01-08-2016 20:01:27.372 -0800 INFO  PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-08-2016 20:01:27.373 -0800 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-08-2016 20:01:27.373 -0800 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-08-2016 20:01:27.496 -0800 INFO  TcpOutputProc - Initializing with fwdtype=lwf
01-08-2016 20:01:27.496 -0800 INFO  ServerRoles - Declared role=lightweight_forwarder.
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-08-2016 20:01:27.501 -0800 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-08-2016 20:01:27.501 -0800 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-08-2016 20:01:27.501 -0800 INFO  PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-08-2016 20:01:27.501 -0800 INFO  PipelineComponent - Launching the pipelines.
01-08-2016 20:01:27.502 -0800 INFO  loader - Limiting REST HTTP server to 1365 sockets
01-08-2016 20:01:27.502 -0800 INFO  loader - Limiting REST HTTP server to 1365 threads
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - TailWatcher initializing...
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-08-2016 20:01:27.547 -0800 INFO  TailingProcessor - Adding watch on path: /opt/splunkforwarder/etc/splunk.version.
01-08-2016 20:01:27.547 -0800 INFO  TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/log/splunk.
01-08-2016 20:01:27.547 -0800 INFO  TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/spool/splunk.
01-08-2016 20:01:27.547 -0800 INFO  TailingProcessor - Adding watch on path: /var/log/messages.
01-08-2016 20:01:27.547 -0800 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-08-2016 20:01:27.600 -0800 INFO  TcpOutputProc - Connected to idx=10.153.194.113:9997
01-08-2016 20:01:57.371 -0800 WARN  AuthenticationManagerSplunk - Seed file is not present. Defaulting to generic username/pass pair.
01-08-2016 20:01:57.703 -0800 INFO  TcpOutputProc - Connected to idx=10.153.194.129:9997
01-08-2016 20:02:28.450 -0800 INFO  TcpOutputProc - Connected to idx=10.153.194.115:9997
01-08-2016 20:02:57.479 -0800 INFO  TcpOutputProc - Connected to idx=10.153.194.129:9997
01-08-2016 20:03:27.509 -0800 INFO  TcpOutputProc - Connected to idx=10.153.194.115:9997
01-08-2016 20:03:57.538 -0800 INFO  TcpOutputProc - Connected to idx=10.153.194.129:9997
01-08-2016 20:04:27.560 -0800 INFO  TcpOutputProc - Connected to idx=10.153.194.115:9997
01-08-2016 20:04:57.576 -0800 INFO  TcpOutputProc - Connected to idx=10.153.194.129:9997
01-08-2016 20:05:27.593 -0800 INFO  TcpOutputProc - Connected to idx=10.153.194.113:9997
01-08-2016 20:06:57.638 -0800 INFO  TcpOutputProc - Connected to idx=10.153.194.129:9997
01-08-2016 20:07:27.652 -0800 INFO  TcpOutputProc - Connected to idx=10.153.194.113:9997

Answer 13 · 2016-01-12T04:43:14.000Z

HOST: Works with the same configuration shared before...

- `CONTAINER`: DOES NOT WORK so far...

Is there any Debug statement I can turn on in the container?

Answer 14 · 2016-01-12T06:36:34.000Z

Hi @outcoldman

I got a Docker image based on CENTOS that works like the following:

We have a cached version of the rpm stored at http://sds-repo-int.qdc.company.com:8081/nexus/content/repositories/CTO.OPS-releases/com/company/CTO/OPS-releases/splunkforwarder/6.2.4/splunkforwarder-6.2.4-271043-lin ux-2.6-x86_64.rpm

Dockerfile

FROM richxsl/rhel7
MAINTAINER marcello.desales@gmail.com

ENV SPLUNK_HOME /opt/splunkforwarder
ENV SPLUNK_GROUP splunk
ENV SPLUNK_USER splunk
ENV SPLUNK_BACKUP_DEFAULT_ETC /var/opt/splunk

RUN groupadd -r ${SPLUNK_GROUP} \
    && useradd -r -m -g ${SPLUNK_GROUP} ${SPLUNK_USER}

COPY ./INTU-LATEST.repo /etc/yum.repos.d/

RUN yum remove -y subscription-manager
ENV LANG en_US.utf8

ADD http://sds-repo-int.qdc.company.com:8081/nexus/content/repositories/CTO.OPS-releases/com/company/CTO/OPS-releases/splunkforwarder/6.2.4/splunkforwarder-6.2.4-271043-lin
ux-2.6-x86_64.rpm .

RUN yum install -y splunkforwarder-6.2.4-271043-linux-2.6-x86_64.rpm && /opt/splunkforwarder/bin/splunk enable boot-start --accept-license --answer-yes && rm -f splunkfor
warder-6.2.4-271043-linux-2.6-x86_64.rpm

COPY entrypoint.sh /sbin/entrypoint.sh
RUN chmod +x /sbin/entrypoint.sh

WORKDIR /opt/splunkforwarder

# Configurations folder, var folder for everyting (indexes, logs, kvstore)
VOLUME [ "/opt/splunkforwarder/etc", "/opt/splunkforwarder/var" ]

ENTRYPOINT ["/sbin/entrypoint.sh"]

entrypoint.sh

Only for the forwarder running as non-daemon...

#!/bin/bash

set -e

${SPLUNK_HOME}/bin/splunk start --nodaemon --accept-license --answer-yes --no-prompt

Is there anything that this setup differs from yours? I noticed you do other things while setting up the server in entrypoint.sh.

Here's the logs of the container itself:

splunkforwarder_1     |
splunkforwarder_1     | Splunk> Now with more code!
splunkforwarder_1     |
splunkforwarder_1     | Checking prerequisites...
splunkforwarder_1     |     Checking mgmt port [8089]: New certs have been generated in '/opt/splunkforwarder/etc/auth'.
splunkforwarder_1     | Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
splunkforwarder_1     | Couldn't initialize SSL Context for HTTPClient in ServerConfig
splunkforwarder_1     | Generating a 1024 bit RSA private key
splunkforwarder_1     | .....++++++
splunkforwarder_1     | .......................++++++
splunkforwarder_1     | writing new private key to 'privKeySecure.pem'
splunkforwarder_1     | -----
splunkforwarder_1     | Signature ok
splunkforwarder_1     | subject=/CN=pe2enpmas300.corp.company.net/O=SplunkUser
splunkforwarder_1     | Getting CA Private Key
splunkforwarder_1     | writing RSA key
splunkforwarder_1     | 2016-01-12 08:53:24.785 +0000 splunkd started (build 271043)

Answer 15 · 2016-01-12T07:52:25.000Z

Another false alarm... It is NOT working... I had Splunk running in the host.

[deploy@pe2enpmas300 ~]$ sudo su
[root@pe2enpmas300 deploy]# cd /npmo-data/npmo-server/
[root@pe2enpmas300 npmo-server]# service splunk status
Splunk status:
splunkd is running (PID: 25381).
splunk helpers are running (PIDs: 25382).
[root@pe2enpmas300 npmo-server]# service splunk stop

I restarted all the containers again and splunk stopped recieving events...

Potential problems

Is there anything in the Docker level that I need to verify?

Both docker images present this same problem: https://answers.splunk.com/answers/39497/unable-to-access-the-webui-and-im-getting-this-error-in-my-splunkd-log-anyone-know-how-to-fix-this.html

I'm not sure if this is caused by Docker images or if we need to share the server's cert.

Other Potential solution is to automate the creation of the SSL Certs... Maybe?

https://answers.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certificates-and-authentication.html

Answer 16 · 2016-01-12T14:58:02.000Z

@marcellodesales in your server.conf you have sslKeysfilePassword setup, do you actually map any your own certificates? Could you try to remove this line from conf?

Answer 17 · 2016-01-12T15:14:07.000Z

@outcoldman Let me try that! I don't think I do...

Answer 18 · 2016-01-12T15:26:09.000Z

@outcoldman I get the same exact error I reported before and they are the same between the 2 docker images (yours and ours)...

docker logs

splunkforwarder_1 |
splunkforwarder_1 | Splunk> Now with more code!
splunkforwarder_1 |
splunkforwarder_1 | Checking prerequisites...
splunkforwarder_1 |     Checking mgmt port [8089]: New certs have been generated in '/opt/splunkforwarder/etc/auth'.
splunkforwarder_1 | Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
splunkforwarder_1 | Couldn't initialize SSL Context for HTTPClient in ServerConfig
splunkforwarder_1 | Generating a 1024 bit RSA private key
splunkforwarder_1 | ..............................................................++++++
splunkforwarder_1 | ..........................++++++
splunkforwarder_1 | writing new private key to 'privKeySecure.pem'
splunkforwarder_1 | -----
splunkforwarder_1 | Signature ok
splunkforwarder_1 | subject=/CN=pe2enpmas300.corp.company.net/O=SplunkUser
splunkforwarder_1 | Getting CA Private Key
splunkforwarder_1 | writing RSA key
splunkforwarder_1 | 2016-01-12 15:20:40.571 +0000 splunkd started (build 271043)

splunkd.log

01-12-2016 15:20:40.922 +0000 INFO  loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 15:20:40.922 +0000 INFO  loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:20:40.927 +0000 ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=101077092 error:06065064:digital envelope rout
ines:EVP_DecryptFinal_ex:bad decrypt.
01-12-2016 15:20:40.927 +0000 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
01-12-2016 15:20:40.927 +0000 ERROR HTTPServer - SSL will not be enabled

Answer 19 · 2016-01-12T15:38:20.000Z

@marcellodesales and please remove pass4SymmKey as well

Answer 20 · 2016-01-12T15:44:09.000Z

@outcoldman ok Let me try it...

Answer 21 · 2016-01-12T15:50:54.000Z

@outcoldman It finally solved the ERROR events with your image... I'm recreating all the containers and verify...

[root@pe2enpmas300 npmo-server]# docker exec -ti npmoserver_splunkforwarder_1 bash
root@cd9d03dfce3b:/opt/splunk# vi var/log/
introspection/ splunk/
root@cd9d03dfce3b:/opt/splunk# vi var/log/splunk/splunkd.log
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - Will generate GUID, as none found on this server.
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My newly generated GUID is C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - Found no site defined in server.conf
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My hostname is "cd9d03dfce3b".
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Setting HTTP server compression state=on
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - Initing LicenseMgr
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - serverName=pe2enpmas300.corp.company.net guid=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - connection_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - send_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - receive_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - squash_threshold=2000
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - strict_pool_quota=1
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - closing stack mgr
01-12-2016 15:45:34.604 +0000 INFO  LMSlaveInfo - all slaves cleared
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - added pool auto_generated_pool_free to stack free
01-12-2016 15:45:34.604 +0000 INFO  ServerRoles - Declared role=license_master.
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - init completed [C3A35716-9824-4BDC-83D8-4C67FE8077DD,Forwarder,runContext_splunkd=true]
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - StackMgr init complete...
01-12-2016 15:45:34.604 +0000 INFO  LMTracker - init'ing slaveId=C3A35716-9824-4BDC-83D8-4C67FE8077DD label=pe2enpmas300.corp.company.net [30,30,self]
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - enabling implicit feature set
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - attempting to ping master=self from slave=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.605 +0000 INFO  LMSlaveInfo - new slave='C3A35716-9824-4BDC-83D8-4C67FE8077DD' created
01-12-2016 15:45:34.605 +0000 INFO  LMSlaveInfo - Detected that masterTimeFromSlave(ZERO_TIME) < lastRolloverTime(Tue Jan 12 00:00:00 2016), meaning that the mas
ter has already rolled over. Ignore slave persisted usage.
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
:1
01-12-2016 15:45:34.805 +0000 INFO  DS_DC_Common - Initializing core facilities of PubSub system.
01-12-2016 15:45:34.813 +0000 INFO  DC:DeploymentClient - target-broker clause is missing.
01-12-2016 15:45:34.813 +0000 WARN  DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-12-2016 15:45:34.813 +0000 INFO  DS_DC_Common - Deployment Client not initialized.
01-12-2016 15:45:34.813 +0000 INFO  DS_DC_Common - Deployment Server not available on a dedicated forwarder.
01-12-2016 15:45:34.813 +0000 INFO  IntrospectionGenerator:disk_objects - Enabled: indexes|volumes|dispatch=false fishbucket=true partitions=false
01-12-2016 15:45:34.813 +0000 INFO  IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600s
01-12-2016 15:45:34.814 +0000 INFO  IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db' (N
o such file or directory).  This is normal when splunk is first starting up.
01-12-2016 15:45:34.815 +0000 WARN  DistributedPeerManager - feature=DistSearch not enabled for your license level
01-12-2016 15:45:34.815 +0000 INFO  IndexProcessor - running splunkd specific init
01-12-2016 15:45:34.816 +0000 INFO  loader - Initializing from configuration
01-12-2016 15:45:34.817 +0000 INFO  PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-12-2016 15:45:34.819 +0000 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-12-2016 15:45:34.819 +0000 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-12-2016 15:45:34.966 +0000 INFO  TcpOutputProc - Initializing with fwdtype=lwf
01-12-2016 15:45:34.966 +0000 INFO  ServerRoles - Declared role=lightweight_forwarder.
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Launching the pipelines.
01-12-2016 15:45:34.973 +0000 INFO  loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 15:45:34.973 +0000 INFO  loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:45:35.018 +0000 INFO  TailingProcessor - TailWatcher initializing...
01-12-2016 15:45:35.018 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 15:45:35.031 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.115:9997
:1
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - Will generate GUID, as none found on this server.
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My newly generated GUID is C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - Found no site defined in server.conf
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My hostname is "cd9d03dfce3b".
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Setting HTTP server compression state=on
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - Initing LicenseMgr
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - serverName=pe2enpmas300.corp.company.net guid=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - connection_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - send_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - receive_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - squash_threshold=2000
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - strict_pool_quota=1
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - closing stack mgr
01-12-2016 15:45:34.604 +0000 INFO  LMSlaveInfo - all slaves cleared
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - added pool auto_generated_pool_free to stack free
01-12-2016 15:45:34.604 +0000 INFO  ServerRoles - Declared role=license_master.
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - init completed [C3A35716-9824-4BDC-83D8-4C67FE8077DD,Forwarder,runContext_splunkd=true]
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - StackMgr init complete...
01-12-2016 15:45:34.604 +0000 INFO  LMTracker - init'ing slaveId=C3A35716-9824-4BDC-83D8-4C67FE8077DD label=pe2enpmas300.corp.company.net [30,30,self]
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - enabling implicit feature set
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - attempting to ping master=self from slave=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.605 +0000 INFO  LMSlaveInfo - new slave='C3A35716-9824-4BDC-83D8-4C67FE8077DD' created
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedXML state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Alerting state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AllowDuplicateKeys state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=CustomRoles state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=DistSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=GuestPass state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=KVStore state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=LDAPAuth state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=RcvSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=ScheduledAlerts state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=UnisiteClustering state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LicenseMgr - Tracker init complete...
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: virtual address space size: unlimited
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: data segment size: unlimited
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: resident memory size: unlimited
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: core file size: unlimited
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: data file size: unlimited
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: open files: 1048576 files
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: user processes: 1048576 processes
01-12-2016 15:45:34.611 +0000 INFO  loader - Splunkd starting (build 271043).
01-12-2016 15:45:34.611 +0000 INFO  loader - System info: Linux, cd9d03dfce3b, 3.10.0-327.3.1.el7.x86_64, #1 SMP Fri Nov 20 05:40:26 EST 2015, x86_64.
01-12-2016 15:45:34.611 +0000 INFO  loader - Detected 4 (virtual) CPUs, 4 CPU cores, and 15880MB RAM
01-12-2016 15:45:34.611 +0000 INFO  loader - Maximum number of threads (approximate): 7940
01-12-2016 15:45:34.611 +0000 INFO  loader - Arguments are: "-p" "8089" "start"
01-12-2016 15:45:34.611 +0000 INFO  loader - Getting configuration data from: /opt/splunk/etc/myinstall/splunkd.xml
01-12-2016 15:45:34.612 +0000 INFO  loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunk/etc/modules
01-12-2016 15:45:34.612 +0000 INFO  loader - loading modules from /opt/splunk/etc/modules
01-12-2016 15:45:34.613 +0000 INFO  loader - Writing out composite configuration file: /opt/splunk/var/run/splunk/composite.xml
01-12-2016 15:45:34.617 +0000 INFO  ServerRoles - Declared role=universal_forwarder.
01-12-2016 15:45:34.643 +0000 INFO  loader - Setting SSL configuration.
01-12-2016 15:45:34.643 +0000 INFO  loader - Server supporting SSL versions SSL3,TLS1.0,TLS1.1,TLS1.2
01-12-2016 15:45:34.643 +0000 INFO  loader - Using cipher suite TLSv1+HIGH:@STRENGTH
01-12-2016 15:45:34.643 +0000 INFO  loader - ECDH curve not configured
01-12-2016 15:45:34.803 +0000 INFO  SpecFiles - Found external scheme definition for stanza "MonitorNoHandle://" with 2 parameters: disabled, index
01-12-2016 15:45:34.803 +0000 INFO  SpecFiles - Found external scheme definition for stanza "WinHostMon://" with 4 parameters: type, interval, disabled, index
01-12-2016 15:45:34.803 +0000 INFO  SpecFiles - Found external scheme definition for stanza "WinPrintMon://" with 4 parameters: type, baseline, disabled, index
01-12-2016 15:45:34.804 +0000 WARN  ClusteringMgr - Ignoring clustering configuration, the active license disables this feature.
01-12-2016 15:45:34.805 +0000 INFO  SHPoolingMgr - shpooling disabled
01-12-2016 15:45:34.805 +0000 INFO  DS_DC_Common - Initializing the PubSub system.
01-12-2016 15:45:34.805 +0000 INFO  DS_DC_Common - Initializing core facilities of PubSub system.
01-12-2016 15:45:34.813 +0000 INFO  DC:DeploymentClient - target-broker clause is missing.
01-12-2016 15:45:34.813 +0000 WARN  DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-12-2016 15:45:34.813 +0000 INFO  DS_DC_Common - Deployment Client not initialized.
01-12-2016 15:45:34.813 +0000 INFO  DS_DC_Common - Deployment Server not available on a dedicated forwarder.
01-12-2016 15:45:34.813 +0000 INFO  IntrospectionGenerator:disk_objects - Enabled: indexes|volumes|dispatch=false fishbucket=true partitions=false
01-12-2016 15:45:34.813 +0000 INFO  IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600s
01-12-2016 15:45:34.814 +0000 INFO  IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db' (N
o such file or directory).  This is normal when splunk is first starting up.
01-12-2016 15:45:34.815 +0000 WARN  DistributedPeerManager - feature=DistSearch not enabled for your license level
01-12-2016 15:45:34.815 +0000 INFO  IndexProcessor - running splunkd specific init
01-12-2016 15:45:34.816 +0000 INFO  loader - Initializing from configuration
01-12-2016 15:45:34.817 +0000 INFO  PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-12-2016 15:45:34.819 +0000 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-12-2016 15:45:34.819 +0000 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-12-2016 15:45:34.966 +0000 INFO  TcpOutputProc - Initializing with fwdtype=lwf
01-12-2016 15:45:34.966 +0000 INFO  ServerRoles - Declared role=lightweight_forwarder.
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Launching the pipelines.
01-12-2016 15:45:34.973 +0000 INFO  loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 15:45:34.973 +0000 INFO  loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:45:35.018 +0000 INFO  TailingProcessor - TailWatcher initializing...
01-12-2016 15:45:35.018 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 15:45:35.031 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.115:9997

Answer 22 · 2016-01-12T16:02:54.000Z

@outcoldman

Nothing yet... Although I don't see any errors in the logs...

01-12-2016 15:55:02.774 +0000 INFO  loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - TailWatcher initializing...
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 15:55:02.841 +0000 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 15:55:02.853 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.113:9997
01-12-2016 15:55:32.622 +0000 WARN  AuthenticationManagerSplunk - Seed file is not present. Defaulting to generic username/pass pair.
01-12-2016 15:55:32.975 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.129:9997
01-12-2016 15:56:02.927 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.115:9997
01-12-2016 15:56:33.254 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.113:9997
01-12-2016 15:57:03.284 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.129:9997
01-12-2016 15:57:33.315 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.115:9997

The container can see the current logs in the /var/log/messages...

root@51e26f57eb48:/opt/splunk# tail -f /var/log/messages
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.690Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 301: change for package lodash._isiterateecall"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.693Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 302: change for package lodash.templatesettings"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.696Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 303: change for package lodash.escape"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.697Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 304: change for package lodash._getnative"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.702Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 305: change for package lodash.keys"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.706Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 306: change for package lodash.isarray"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.709Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 307: change for package lodash.isarguments"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.710Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 308: change for package clone-stats"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.729Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 309: change for package clone"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.732Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 310: change for package flagged-respawn"}

Nothing significant found in the logs either... :(

root@51e26f57eb48:/opt/splunk# ls -la var/log/splunk/
total 92
drwx------ 2 root root  4096 Jan 12 16:01 .
drwx--x--x 4 root root    39 Jan 12 15:55 ..
-rw------- 1 root root   140 Jan 12 15:55 audit.log
-rw------- 1 root root     0 Jan 12 15:55 btool.log
-rw------- 1 root root   307 Jan 12 16:00 conf.log
-rw------- 1 root root    64 Jan 12 15:55 first_install.log
-rw------- 1 root root     0 Jan 12 15:55 license_audit.log
-rw------- 1 root root     0 Jan 12 15:55 license_usage.log
-rw------- 1 root root 47883 Jan 12 16:01 metrics.log
-rw------- 1 root root     0 Jan 12 15:55 mongod.log
-rw------- 1 root root     0 Jan 12 15:55 remote_searches.log
-rw------- 1 root root     0 Jan 12 15:55 scheduler.log
-rw------- 1 root root     0 Jan 12 15:55 searchhistory.log
-rw------- 1 root root     0 Jan 12 15:55 splunkd_access.log
-rw------- 1 root root 20262 Jan 12 16:01 splunkd.log
-rw------- 1 root root    61 Jan 12 15:55 splunkd_stderr.log
-rw------- 1 root root     0 Jan 12 15:55 splunkd_stdout.log
-rw------- 1 root root     0 Jan 12 15:55 splunkd_ui_access.log
-rw------- 1 root root  1125 Jan 12 15:55 splunkd-utility.log
root@51e26f57eb48:/opt/splunk# cat var/log/splunk/splunkd_stderr.log
2016-01-12 15:55:02.408 +0000 splunkd started (build 271043)

Maybe turning on DEBUG mode of the forward can show something else?

Answer 23 · 2016-01-12T16:12:26.000Z

@marcellodesales sure we can turn on DEBUG for something, but I have no clue where we lose data.

Another suggestion - try to use splunk standalone instance just to try if you will get these messages indexed inside Splunk without actually forwarding them.

Answer 24 · 2016-01-12T16:24:35.000Z

@outcoldman

Suddenly, things started working this time, as I see it picked up the files... The only thing I did was to add a debug line as specified in step 6 of https://answers.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certificates-and-authentication.html... But that should not affect anything I'm guessing...

01-12-2016 16:16:49.475 +0000 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 16:16:49.494 +0000 INFO  WatchedFile - Will begin reading at offset=19716950 for file='/var/log/messages'.
01-12-2016 16:16:49.497 +0000 INFO  WatchedFile - Will begin reading at offset=1125 for file='/opt/splunk/var/log/splunk/splunkd-utility.log'.
01-12-2016 16:16:49.499 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/btoo
l.log'.
01-12-2016 16:16:49.502 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/sear
chhistory.log'.
01-12-2016 16:16:49.503 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/splu
nkd_access.log'.
01-12-2016 16:16:49.505 +0000 INFO  WatchedFile - Will begin reading at offset=48771 for file='/opt/splunk/var/log/splunk/audit.log'.
01-12-2016 16:16:49.507 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/mong
od.log'.
01-12-2016 16:16:49.509 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/lice
nse_usage.log'.
01-12-2016 16:16:49.510 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/lice
nse_audit.log'.
01-12-2016 16:16:49.513 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/remo
te_searches.log'.
01-12-2016 16:16:49.515 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/sche
duler.log'.
01-12-2016 16:16:49.516 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/splu
nkd_ui_access.log'.
01-12-2016 16:16:49.517 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/splu
nkd_stdout.log'.
01-12-2016 16:16:49.519 +0000 INFO  WatchedFile - Will begin reading at offset=61 for file='/opt/splunk/var/log/splunk/splunkd_stderr.log'.
01-12-2016 16:16:49.558 +0000 INFO  TcpOutputProc - Connected to idx=10.153.194.113:9997

The logs event date in Splunk are using a different time...

root@51e26f57eb48:/opt/splunk# date
Tue Jan 12 16:18:49 UTC 2016
root@51e26f57eb48:/opt/splunk# tail -f /var/log/messages
Jan 12 08:21:44 pe2enpmas300 docker/policyfollower[1358]: storing document admn at 640999
Jan 12 08:21:44 pe2enpmas300 docker/policyfollower[1358]: done processing admn at 640999
Jan 12 08:21:45 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:45.186Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:46 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:46.533Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:16:21:46 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 12 08:21:47 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:47.189Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:47 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:47.335Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:16:21:47 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 12 08:21:49 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:49.192Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:51 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:51.195Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:51 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:51.467Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:16:21:51 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 12 08:21:52 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:52.385Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:16:21:52 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 12 08:21:53 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:53.197Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:55 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:55.200Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:56 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:56.525Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:16:21:56 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 12 08:21:57 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:57.203Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:57 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:57.302Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:16:21:57 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}

Where there's of offset in the Web UI of Less 8 hours... We are very close now 👍

Answer 25 · 2016-01-12T16:26:58.000Z

How can I correlate the Time column with the time displayed in the Event column???

I noticed that the timezone in the container is UTC

root@51e26f57eb48:/opt/splunk# date
Tue Jan 12 16:26:24 UTC 2016

Answer 26 · 2016-01-12T16:31:07.000Z

@marcellodesales do you mean that it shows you wrong time or not expected TimeZone? I guess the difference of 8 hours is expected if your indexers show you time in your timezone and forwarder collects everything in UTC. If you are in Pacific Time zone - looks like everything is working.

Answer 27 · 2016-01-12T16:45:22.000Z

@outcoldman Everything is working!!! The only problem is that we are in the PST timezone, but the Time shown in the Time column is less 8 hrs... Is there a way to set the forwarder to collect everything in PST instead?

I'm tring to set the time inside the container, but no luck so far... What I tried was:

Set the environment var TZ
Mount the /etc/timezone

Mounting the timezone did not work because my host is RHEL7 and I did not find the same files in your container... (not sure...)

Answer 28 · 2016-01-12T17:13:55.000Z

@marcellodesales no need to modify anything on system, in logs everything looks good, you have timestaps in UTC timezone and they show the right time.

Please read http://docs.splunk.com/Documentation/Splunk/6.2.8/Data/Configuretimestamprecognition

Take a look on etc/system/default/props.conf definition for syslog

[syslog]
pulldown_type = true
maxDist = 3
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
category = Operating System
description = Output produced by many syslog daemons, as described in RFC3164 by the IETF

As you can see it does not have TZ, so add props.conf inside etc/system/default/props.conf

[syslog]
TZ = UTC

And actually you have better time inside of the syslog message - I would recommend to set timezone parsing of these values instead of timestamps from syslog.

Answer 29 · 2016-01-12T17:21:21.000Z

@outcoldman Sounds good and thanks for all your help... The deployment on the second host worked... Here's what I have so far:

splunkforwarderData:
  image: busybox
  volumes:
    - ./monitor/splunk:/opt/splunk/etc/system/local
    - /var/log/messages:/var/log/messages:ro
    - /usr/share/zoneinfo/America/Los_Angeles:/etc/localtime:ro

The server.conf without any SSL setting... Since the container needs to write the values for the server, the volume is mounted with write permission...

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[general]
serverName = pe2enpmas300.company.intuit.net

Thanks a lot for more pointers... I will verify with OPS how the proceed with that...