Is it running it on full cluster or single node
thak123 opened this issue · 7 comments
I am unable to see the load running on other nodes on my spark cluster. I am having a 10 node cluster with standalone set up.
How can I check if it is utilizing full cluster ?
UPDATE
I added --master on line 90 & 420 . I see they are getting logged in spark UI. But I see
Caused by: java.io.FileNotFoundException: File file:/home/hadoop-user/spark-tpc-ds-performance-test/gendata/call_center.dat does not exist
even after running step 2. When I check it is there initially. But during subsequent process at 4% it throws error saying not present.
@bomeng @dilipbiswal please advise
I tried cloning on each node the git repo ." File file:/home/hadoop-user/spark-tpc-ds-performance-test/gendata/call_center.dat does not exist" this error was resolved but I get error on query execution stage saying that IOEXCEPTION: spark-warehouse doesnt have permission
@thak123 In this pattern, we only explore how to set this up on a single node (laptop). We have a section "Considerations while increasing the scale factor" where we mention briefly about steps that needs to be performed. If while trying to make it work on a multi node setup, you find some steps missing, please feel free to submit a PR. So in case of a multi node cluster operating on top of a distributed file system (HDFS), we have to make sure that data files are available on all the nodes. So we need to copy the data files in HDFS and modify the load scripts to work off the copied location.
@dilipbiswal I am able to push the data into HDFS . I am able to run steps from 1 and 2 . The create table step is able to create the tables structure but none of the tables are populated with data . So the number of rows returned are zero. The parquet table created using the temp csv_table is having zero rows. The issue is not faced in the local system. Any Idea on fixing this issue.
I am using spark 2.3 and hive 0.11.
Should the spark be compiled from the scratch in the first place ?
Was able to fix the issue. Able to successfully run tpc-ds in my cluster
@thak123 Glad it worked for you :-)
@thak123 Can you please explain how you were able to run it on your cluster. I am running it on my Spark cluster and I have the same 0 rows error. Thanks.