Isilon/isilon_hadoop_tools

Issue with the Script

Opened this issue · 2 comments

arjunnandy commented

The Isilon_create_directories script has an issue where it did not create the warehouse/tablespace/external. Mentioned below is a dump of the debug after executing the script

[2021-04-23 09:44:18,076] isilon_hadoop_tools.directories [INFO] mkdir '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external'
[2021-04-23 09:44:18,076] isilon_hadoop_tools.directories [INFO] chmod '775' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external'
[2021-04-23 09:44:18,076] isilon_hadoop_tools.directories [INFO] chown 'hdfs:supergroup' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external'
[2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] mkdir '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed'
[2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chmod '775' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed'
[2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chown 'hdfs:supergroup' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed'
[2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] mkdir '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external/hive'
[2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chmod '1775' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external/hive'
[2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chown 'hive:hive' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external/hive'
[2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] mkdir '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed/hive'
[2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chmod '1775' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed/hive'
[2021-04-23 09:44:18,077] isilon_hadoop_tools.directories [INFO] chown 'hive:hive' '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/managed/hive'
[2021-04-23 09:47:27,628] isilon_hadoop_tools.directories [INFO] mkdir '/ifs/data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external'
"message" : "A component of path 'data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external' does not exist."
[2021-04-23 09:47:28,496] root [ERROR] A component of path 'data/UAT_SAAP/NCW/HDFS/warehouse/tablespace/external' does not exist.

After running the script we went to the Isilon and verified that the tablespace directory was missing under warehouse.
ncwdh5icl11-1# cd /ifs/data/UAT_SAAP/NCW/HDFS/warehouse/
ncwdh5icl11-1# ls
ncwdh5icl11-1# pwd

The customer had to create the tablespace directory manually. I too have noticed the same behavior. Could you correct the script so that the error does not come up.

shirleylxie commented

This issue should be fixed with v4.1.3. Please have a try at the newest version.

Here is the log of v4.1.3.

[INFO] chown 'hdfs:supergroup' '/ifs/data/warehouse/tablespace'
[INFO] mkdir '/ifs/data/warehouse/tablespace/external'
[INFO] chmod '775' '/ifs/data/warehouse/tablespace/external'
[INFO] chown 'hdfs:supergroup' '/ifs/data/warehouse/tablespace/external'
[INFO] mkdir '/ifs/data/warehouse/tablespace/managed'
[INFO] chmod '775' '/ifs/data/warehouse/tablespace/managed'
[INFO] chown 'hdfs:supergroup' '/ifs/data/warehouse/tablespace/managed'
[INFO] mkdir '/ifs/data/warehouse/tablespace/external/hive'
[INFO] chmod '1775' '/ifs/data/warehouse/tablespace/external/hive'
[INFO] chown 'hive:hive' '/ifs/data/warehouse/tablespace/external/hive'
[INFO] mkdir '/ifs/data/warehouse/tablespace/managed/hive'
[INFO] chmod '1775' '/ifs/data/warehouse/tablespace/managed/hive'
[INFO] chown 'hive:hive' '/ifs/data/warehouse/tablespace/managed/hive'
  • 👍1
tucked commented

Duplicate of #94