Support for CDH 5.8

Question

Support for CDH 5.8

Closed this issue 8 years ago · 20 comments

I pushed preliminary work (based on 4.8.0 RC2) to https://github.com/chiastic-security/phoenix-for-cloudera/tree/4.8-HBase-1.2-cdh5.8

Issues

You have to build this with mvn ... -Dcdh.flume.version=1.6.0 because the Flume artifacts specified by the CDH parent pom for 5.8.0 are not publicly available. The available version I find with search.maven.org is 1.6.0.
Phoenix 4.8.0 requires Tephra 0.8.0-incubating and this version of Tephra does not have a compat module for CDH 5.2 (HBase 1.2 plus CDH specific coprocessor and client API changes). I've disabled all transactional tests, see 3b745a8. The remaining tests in phoenix-core pass.
Several issues with the phoenix-hive module. I've temporarily disabled inclusion of this module in the build. See e6a751c

Anyway, it's possible to build binary artifacts minus the Hive integration:

$ mvn clean package -DskipTests -Dcdh.flume.version=1.6.0
[...]
[INFO] Apache Phoenix ..................................... SUCCESS [  1.502 s]
[INFO] Phoenix Core ....................................... SUCCESS [ 21.716 s]
[INFO] Phoenix - Flume .................................... SUCCESS [  0.836 s]
[INFO] Phoenix - Pig ...................................... SUCCESS [  0.939 s]
[INFO] Phoenix Query Server Client ........................ SUCCESS [ 12.380 s]
[INFO] Phoenix Query Server ............................... SUCCESS [  2.211 s]
[INFO] Phoenix - Pherf .................................... SUCCESS [  2.334 s]
[INFO] Phoenix - Spark .................................... SUCCESS [ 18.906 s]
[INFO] Phoenix Client ..................................... SUCCESS [01:04 min]
[INFO] Phoenix Server ..................................... SUCCESS [  2.971 s]
[INFO] Phoenix Assembly ................................... SUCCESS [ 14.171 s]
[INFO] Phoenix - Tracing Web Application .................. SUCCESS [  1.860 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

Spark tests look good.

Highly doubtful the flume integration will work, of course.

Answer 1 · 2016-08-07T22:31:45.000Z

FYI @busbey @JamesRTaylor

Answer 2 · 2016-08-07T22:50:42.000Z

Thanks for taking a look at this Andrew!

Which flume artifacts? We're supposed to publish everything, so that sounds like a bug.

For tephra, for the last labs packaging I rebuilt against CDH jars and that worked. I believe I already filed a JIRA against them for the issue, but rebuilding again should work.

Answer 3 · 2016-08-07T23:19:49.000Z

@busbey I saw the POM change you made for Tephra/CDH. That was for 0.7, Phoenix 4.8 needs 0.8.0-incubating. Tephra ships with a CDH compat module for HBase 1.0, not the newer stuff based on 1.2. Also, that labs version of Tephra is not publically available

As for flume, whatever is defined in the parent pom for 5.8.0 as cdh.flume.version. 0.9.4 something or other as I recall. Build failed unable to pull it from maven central or Cloudera artifactory. So I searched maven for something public, found 1.6.0, and used it.

Answer 4 · 2016-08-07T23:59:01.000Z

Hurm. The tephra repo was supposed to get published when the Phoenix one did. I'll figure out what happened. FWIW, it was just the vanilla 1.2 compat module with the versions changed. It's source compatible for CDH, just not binary. My guess is tephra 0.8.0's 1.2 module is the same.

Answer 5 · 2016-08-08T00:08:52.000Z

No 1.2 compat module for tephra in maven central: http://search.maven.org/#search%7Cga%7C1%7Ctephra . They have them for 0.96, 0.98, 1.0, 1.0-cdh, and 1.1.

Answer 6 · 2016-08-08T00:10:02.000Z

No compat module was required for 1.2 - the 1.1 compat module is fine for
1.2

On Sun, Aug 7, 2016 at 5:08 PM, Andrew Purtell notifications@github.com
wrote:

No 1.2 compat module for tephra in maven central:
http://search.maven.org/#search%7Cga%7C1%7Ctephra . They have them for
0.96, 0.98, 1.0, 1.0-cdh, and 1.1.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF4-K3bweTC3hovCAE3xscl8o0KPpOaiks5qdnOVgaJpZM4JemkT
.

Answer 7 · 2016-08-08T00:13:05.000Z

Then we need a 1.1-CDH @JamesRTaylor @busbey because the 1.1 compat module is not compatible with CDH. Errors from OperationWithAttributes for one.

Answer 8 · 2016-08-08T00:33:13.000Z

Poorna

Agreed, Andy.

Maybe that's what https://issues.apache.org/jira/browse/TEPHRA-176 will be?

On Sun, Aug 7, 2016 at 5:13 PM, Andrew Purtell notifications@github.com
wrote:

Then we need a 1.1-CDH @JamesRTaylor https://github.com/JamesRTaylor
@busbey https://github.com/busbey because the 1.1 compat module is not
compatible with CDH. Errors from OperationWithAttributes for one.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF4-K8W8UOQSqZmQhtuZzjm2V5Zn631Oks5qdnSSgaJpZM4JemkT
.

Answer 9 · 2016-08-08T00:48:12.000Z

Did you mean @Poorna @JamesRTaylor ?

Answer 10 · 2016-08-08T00:53:08.000Z

@poornachandra (https://github.com/poornachandra)

Answer 11 · 2016-08-08T16:58:42.000Z

FYI, I've filed an internal ticket to get the modified tephra repo published.

For the flume-ng-core artifact, can you try wiping out the local repo and refetching? Browsing the repository, at least, things look to be present (ref flume-ng-core:1.6.0-cdh5.8.0).

Answer 12 · 2016-08-09T18:02:07.000Z

I don't think it helps with needing tephra-incubating 4.8 support, but I got the modified tephra 0.7 published to the cloudera-labs repo

Answer 13 · 2016-08-09T18:19:01.000Z

I don't think it helps with needing tephra-incubating 4.8 support, but I got the modified tephra 0.7 published to the cloudera-labs repo

Thanks Sean. You're right, but it does mean the 4.7 labs release can be reproduced publicly now, which is great.

Answer 14 · 2016-08-09T18:23:46.000Z

Agreed. Thanks, Sean!

On Tue, Aug 9, 2016 at 11:19 AM, Andrew Purtell notifications@github.com
wrote:

I don't think it helps with needing tephra-incubating 4.8 support, but I
got the modified tephra 0.7 published to the cloudera-labs repo

Thanks Sean. You're right, but it does mean the 4.7 labs release can be
reproduced publicly now, which is great.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF4-K71wMNK6_n0wO9k5LJVBikwT6p6Fks5qeMSWgaJpZM4JemkT
.

Answer 15 · 2016-08-18T19:46:31.000Z

Did this get created/completed? I'm a little lost on where this topic ended up at.

Answer 16 · 2016-08-19T10:38:43.000Z

Hi @apurtell , this 4.8-HBase-1.2-cdh5.8 branch will supports for CDHv5.8 and HBasev1.2 for phoenix? Can I build and use the phoenix server/client jars in production?

Answer 17 · 2016-08-21T14:03:36.000Z

Please help.

Answer 18 · 2016-12-27T20:32:13.000Z

I'm trying to build a new parcel from this branch (4.8-HBase-1.2-cdh5.8).

There are any command to generate a new parcel?

I just saw instructions here https://github.com/cloudera/cm_ext/wiki/Building-a-parcel and the spark-assembly module from this repo.

Any idea?

Answer 19 · 2017-01-16T21:25:05.000Z

Is this stable enough for production use? I'm a bit worried about this. Especially since Cloudera seems to have abandoned their cloudera labs parcel.

Answer 20 · 2017-02-02T17:03:36.000Z

We have a branch for 4.9 that doesn't have these issues. Use that