Adding support for ppc64le
Closed this issue · 9 comments
ENV:
arch=ppc64le
os=RHEL7.6
As mentioned in the issue prestodb/presto#14549, in order to get the tests for presto-hive, presto-geospatial, presto-spark-base, etc. to PASS and to get the presto-server execute successfully with hive catalog, I needed to replace the libhadoop.so, libsnappy.so in hadoop-apache2-2.7.4-7.jar.
So, in order to get that fixed at the source, I need to get the libraries which are compatible with RHEL/CentOS 7.6 added to this repo. I have a query regarding that:
@electrum as a part of this commit, the two libraries were added to this repo (maybe compatible with ubuntuxx.xx, but the don't work for RHEL 7.6). I'd like to request some guidance from you as to what would be the best way to get the libraries compatible with RHEL/CentOS 7.6 added to this repo?
@electrum as a part of this commit, the two libraries were added to this repo (maybe compatible with ubuntuxx.xx, but the don't work for RHEL 7.6). I'd like to request some guidance from you as to what would be the best way to get the libraries compatible with RHEL/CentOS 7.6 added to this repo?
@amitsadaphule, @electrum does not work on this repo anymore.
If you want his guidance, you should file an issue at https://github.com/prestosql/presto-hadoop-apache
or reach out on Presto community slack
Thanks for the feedback @findepi ! I'll reach out on Presto community slack
I was able to validate the build and tests on RHEL 7.6 as well as Ubuntu 16.04, using libhadoop.so, libsnappy.so extracted from HDP 2.x rpm, with the following steps:
mkdir hdp; mkdir hadoop; cd hdp
wget http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.1.0/HDP-2.6.1.0-centos7-ppc-rpm.tar.gz
tar xzf HDP-2.6.1.0-centos7-ppc-rpm.tar.gz
cd ..
cp ./hdp/HDP/centos7-ppc/hadoop/hadoop_2_6_1_0_129-2.7.3.2.6.1.0-129.ppc64le.rpm ./hadoop/
cd hadoop; rpm2cpio hadoop_2_6_1_0_129-2.7.3.2.6.1.0-129.ppc64le.rpm | cpio -idm
cd ..
git clone https://github.com/prestodb/presto-hadoop-apache2.git
rm -rf presto-hadoop-apache2/src/main/resources/nativelib/Linux-ppc64le/libhadoop.so presto-hadoop-apache2/src/main/resources/nativelib/Linux-ppc64le/libsnappy.so
cp hadoop/usr/hdp/2.6.1.0-129/hadoop/lib/native/libhadoop.so.1.0.0 presto-hadoop-apache2/src/main/resources/nativelib/Linux-ppc64le/libhadoop.so
cp hadoop/usr/hdp/2.6.1.0-129/hadoop/lib/native/libsnappy.so.1.1.4 presto-hadoop-apache2/src/main/resources/nativelib/Linux-ppc64le/libsnappy.so
cd presto-hadoop-apache2
mvn package
Could someone from the community please add these 2 libraries to the repo? Thanks in advance!
@amitsadaphule since you have them extracted, would you like to open a pull request with these changes?
Thanks @rschlussel for the feedback! The problem here is that I don’t have necessary approvals from my management to publish binaries externally. That's why I couldn't raise the PR.
@rschlussel I've set up a repo and travis-ci which extracts the libraries from HDP rpm and has the resultant libraries (libhadoop.so, libsnappy.so) @ https://github.com/amitsadaphule/HDP-ppc64le-bin-factory/releases.
I've verified the test results on RHEL 7.6 as well as Ubuntu 16.04 for presto-hadoop-apache2. Here's the log from Ubuntu 16.04 container:
testuser@cf0cf92e3f11:~/presto-hadoop-apache2$ mvn package
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO] ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO]
[INFO] -------------< com.facebook.presto.hadoop:hadoop-apache2 >--------------
[INFO] Building hadoop-apache2 2.7.4-9-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (default) @ hadoop-apache2 ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- dependency-scope-maven-plugin:0.8:check (default) @ hadoop-apache2 ---
[INFO] No test dependency scope issues found
[INFO]
[INFO] --- license-maven-plugin:2.3:check (default) @ hadoop-apache2 ---
[INFO] Checking licenses...
[INFO]
[INFO] --- maven-checkstyle-plugin:3.0.0:check (checkstyle) @ hadoop-apache2 ---
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (default) @ hadoop-apache2 ---
[INFO] argLine set to -javaagent:/home/testuser/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/testuser/presto-hadoop-apache2/target/jacoco.exec
[INFO]
[INFO] --- git-commit-id-plugin:2.1.13:revision (default) @ hadoop-apache2 ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hadoop-apache2 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 7 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-apache2 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 11 source files to /home/testuser/presto-hadoop-apache2/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hadoop-apache2 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/testuser/presto-hadoop-apache2/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hadoop-apache2 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 3 source files to /home/testuser/presto-hadoop-apache2/target/test-classes
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ hadoop-apache2 ---
[INFO] Skipping plugin execution
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:analyze-duplicate (default) @ hadoop-apache2 ---
[INFO] Skipping plugin execution
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:analyze-dep-mgt (default) @ hadoop-apache2 ---
[INFO] Skipping plugin execution
[INFO]
[INFO] --- duplicate-finder-maven-plugin:1.2.1:check (default) @ hadoop-apache2 ---
[INFO] Skipping duplicate-finder execution!
[INFO]
[INFO] --- modernizer-maven-plugin:1.7.1:modernizer (modernizer) @ hadoop-apache2 ---
[INFO] Skipping modernizer execution!
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hadoop-apache2 ---
[INFO] Surefire report directory: /home/testuser/presto-hadoop-apache2/target/surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestSuite
Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNGMapConfigurator@6631f5ca
2020-07-24T08:33:45.884+0000 WARNING Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-07-24T08:33:47.119+0000 INFO Successfully loaded & initialized native-bzip2 library system-native
2020-07-24T08:33:47.125+0000 INFO Successfully loaded & initialized native-zlib library
2020-07-24T08:33:47.176+0000 INFO Got brand-new compressor [.bz2]
2020-07-24T08:33:47.200+0000 INFO Got brand-new decompressor [.bz2]
2020-07-24T08:33:47.203+0000 INFO Got brand-new compressor [.deflate]
2020-07-24T08:33:47.205+0000 INFO Got brand-new decompressor [.deflate]
2020-07-24T08:33:47.207+0000 INFO Got brand-new compressor [.gz]
2020-07-24T08:33:47.208+0000 INFO Got brand-new decompressor [.gz]
2020-07-24T08:33:47.211+0000 INFO Got brand-new compressor [.lz4]
2020-07-24T08:33:47.213+0000 INFO Got brand-new decompressor [.lz4]
2020-07-24T08:33:47.215+0000 INFO Got brand-new compressor [.snappy]
2020-07-24T08:33:47.217+0000 INFO Got brand-new decompressor [.snappy]
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.381 sec
Results :
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.2:report (report) @ hadoop-apache2 ---
[INFO] Loading execution data file /home/testuser/presto-hadoop-apache2/target/jacoco.exec
[INFO] Analyzed bundle 'hadoop-apache2' with 16 classes
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hadoop-apache2 ---
[INFO] Building jar: /home/testuser/presto-hadoop-apache2/target/hadoop-apache2-2.7.4-9-SNAPSHOT.jar
[INFO]
[INFO] --- maven-jar-plugin:2.4:test-jar (attach-tests) @ hadoop-apache2 ---
[INFO] Building jar: /home/testuser/presto-hadoop-apache2/target/hadoop-apache2-2.7.4-9-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-source-plugin:2.2.1:jar-no-fork (attach-sources) @ hadoop-apache2 ---
[INFO] Building jar: /home/testuser/presto-hadoop-apache2/target/hadoop-apache2-2.7.4-9-SNAPSHOT-sources.jar
[INFO]
[INFO] --- maven-source-plugin:2.2.1:test-jar-no-fork (attach-sources) @ hadoop-apache2 ---
[INFO] Building jar: /home/testuser/presto-hadoop-apache2/target/hadoop-apache2-2.7.4-9-SNAPSHOT-test-sources.jar
[INFO]
[INFO] --- maven-shade-plugin:2.4.2:shade (default) @ hadoop-apache2 ---
[INFO] Including org.apache.hadoop:hadoop-common:jar:2.7.4 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-annotations:jar:2.7.4 in the shaded jar.
[INFO] Including com.google.guava:guava:jar:26.0-jre in the shaded jar.
[INFO] Including commons-cli:commons-cli:jar:1.2 in the shaded jar.
[INFO] Including commons-httpclient:commons-httpclient:jar:3.1 in the shaded jar.
[INFO] Including commons-codec:commons-codec:jar:1.4 in the shaded jar.
[INFO] Including commons-io:commons-io:jar:2.4 in the shaded jar.
[INFO] Including commons-net:commons-net:jar:3.1 in the shaded jar.
[INFO] Including commons-collections:commons-collections:jar:3.2.2 in the shaded jar.
[INFO] Including org.mortbay.jetty:jetty-util:jar:6.1.26 in the shaded jar.
[INFO] Including org.mortbay.jetty:jetty-sslengine:jar:6.1.26 in the shaded jar.
[INFO] Including commons-lang:commons-lang:jar:2.6 in the shaded jar.
[INFO] Including commons-configuration:commons-configuration:jar:1.6 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the shaded jar.
[INFO] Including org.apache.avro:avro:jar:1.7.4 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:2.5.0 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.2.4 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-auth:jar:2.7.4 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.2.5 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpcore:jar:4.2.4 in the shaded jar.
[INFO] Including org.apache.htrace:htrace-core:jar:3.1.0-incubating in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-hdfs:jar:2.7.4 in the shaded jar.
[INFO] Including io.netty:netty-all:jar:4.0.23.Final in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.4 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-azure:jar:2.7.4 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.9.8 in the shaded jar.
[INFO] Including com.microsoft.azure:azure-storage:jar:2.0.0 in the shaded jar.
[INFO] Including org.apache.commons:commons-lang3:jar:3.3.2 in the shaded jar.
[INFO] Including org.slf4j:jcl-over-slf4j:jar:1.7.13 in the shaded jar.
[INFO] Including org.slf4j:log4j-over-slf4j:jar:1.7.13 in the shaded jar.
[INFO] Including org.slf4j:slf4j-api:jar:1.7.13 in the shaded jar.
[INFO] Including org.slf4j:slf4j-jdk14:jar:1.7.13 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /home/testuser/presto-hadoop-apache2/target/hadoop-apache2-2.7.4-9-SNAPSHOT.jar with /home/testuser/presto-hadoop-apache2/target/hadoop-apache2-2.7.4-9-SNAPSHOT-shaded.jar
[INFO] Replacing original source artifact with shaded source artifact.
[INFO] Replacing /home/testuser/presto-hadoop-apache2/target/hadoop-apache2-2.7.4-9-SNAPSHOT-sources.jar with /home/testuser/presto-hadoop-apache2/target/hadoop-apache2-2.7.4-9-SNAPSHOT-shaded-sources.jar
[INFO] Dependency-reduced POM written at: /home/testuser/presto-hadoop-apache2/target/pom.xml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:14 min
[INFO] Finished at: 2020-07-24T08:34:40Z
[INFO] ------------------------------------------------------------------------
I've also verified that the tests for presto-hive, presto-geospatial, presto-jdbc, presto-spark-base PASS with the resultant hadoop-apache2-2.7.4-9-SNAPSHOT.jar copied as ~/.m2/repository/com/facebook/presto/hadoop/hadoop-apache2/2.7.4-7/hadoop-apache2-2.7.4-7.jar.
Could you please get these libraries added to src/main/resources/nativelib/Linux-ppc64le/? Thanks!
@rschlussel any thoughts on the above comment? Thanks in advance!
@rschlussel I've raised a PR: #45 that fixes this. Please review.