permission denied errors in container logs for splunk

[aneesh786]

Discussed in #1252

^{Originally posted by aneesh786 November 10, 2023}
Hi Team,

Iam trying to install a cluster manager by using spark operator 2.4.0. Containers are failing with below error.

STDOUT:

Updating /opt/splunk/etc

STDERR:

tar: anonymizer: Cannot mkdir: Permission denied
tar: anonymizer: Cannot mkdir: Permission denied
tar: anonymizer/anonymizer-time.ini: Cannot open: No such file or directory
tar: anonymizer: Cannot mkdir: Permission denied
tar: anonymizer/dictionary.txt: Cannot open: No such file or directory
tar: anonymizer: Cannot mkdir: Permission denied
tar: anonymizer/names.txt: Cannot open: No such file or directory
tar: anonymizer: Cannot mkdir: Permission denied
tar: anonymizer/private-terms.txt: Cannot open: No such file or directory
tar: anonymizer: Cannot mkdir: Permission denied
tar: anonymizer/public-terms.txt: Cannot open: No such file or directory
tar: apps: Cannot mkdir: Permission denied
tar: apps: Cannot mkdir: Permission denied
tar: apps/alert_logevent: Cannot mkdir: No such file or directory
tar: apps: Cannot mkdir: Permission denied

tar: system/static/moreAppsIcon.png: Cannot open: No such file or directory
tar: system: Cannot mkdir: Permission denied
tar: system/static/splunkrc_cmds.xml: Cannot open: No such file or directory
tar: users: Cannot mkdir: Permission denied
tar: users: Cannot mkdir: Permission denied
tar: users/users.ini.default: Cannot open: No such file or directory
tar: Exiting with failure status due to previous errors

also looked at the container and looks like /opt/splunk/etc does n't have permission.
[splunk@splunk-cm-cluster-manager-0 splunk]$ ls -lrt
total 3220
-r--r--r--. 1 splunk splunk 85405 Aug 25 17:02 license-eula.txt
-r--r--r--. 1 splunk splunk 57 Aug 25 17:02 copyright.txt
-r--r--r--. 1 splunk splunk 520 Aug 25 17:05 README-splunk.txt
drwxr-xr-x. 3 splunk splunk 45 Aug 25 17:24 include
drwxr-xr-x. 2 splunk splunk 66 Aug 25 17:24 cmake
drwxr-xr-x. 2 splunk splunk 55 Aug 25 17:24 quarantined_files
drwxr-xr-x. 3 splunk splunk 58 Aug 25 17:24 openssl
-rw-r--r--. 1 splunk splunk 0 Aug 25 17:24 ftr
drwxr-xr-x. 2 splunk splunk 54 Aug 25 17:24 swidtag
-r--r--r--. 1 splunk splunk 3200489 Aug 25 17:25 splunk-9.1.1-64e843ea36b1-linux-2.6-x86_64-manifest
drwxr-xr-x. 1 splunk splunk 20 Sep 8 18:14 share
drwxr-xr-x. 1 splunk splunk 43 Sep 8 18:14 lib
drwxr-xr-x. 1 splunk splunk 169 Sep 8 18:14 bin
drwxrwxrwx. 2 root root 6 Nov 9 18:39 var
drwxrwxrwx. 2 root root 6 Nov 9 18:39 etc
[splunk@splunk-cm-cluster-manager-0 splunk]$
drwxrwxrwx. 2 root root 6 Nov 10 07:20 etc
[splunk@splunk-cm-cluster-manager-0 splunk]$ cd etc
[splunk@splunk-cm-cluster-manager-0 etc]$ ls
ls: cannot open directory '.': Permission denied

I tried standalone deployment and that too failed with same error. Do we have any fix for this ?

[vivekr-splunk]

@aneesh786. can you help us investigate this issue further, which platform are you trying this. which storage class are you trying it on. thank you

[aneesh786]

@vivekr-splunk
While trying to create cluster manager or any resource, my container was unable to mount /etc and /var.
I had configured persistent storage using local-path. I believe this was the problem. Pod was unable to create volumes with this storage. Created another storage class using ceph cluster and made it as default storage class. This resolved the issue.
I was using K8S cluster to test splunk