drill
DRILL-8514: Bump zookeeper to 3.9.3
#2967
Open

DRILL-8514: Bump zookeeper to 3.9.3 #2967

letian-jiang wants to merge 3 commits into apache:master from letian-jiang:bump-zk
letian-jiang
letian-jiang150 days ago
letian-jiang DRILL-8514: Bump zookeeper to 3.9.3
08a0331e
pjfanning
pjfanning150 days ago

Upgrading zookeeper is not that easy. The CI build has already failed with an issue around protobuf.
This is a recent attempt to upgrade - #2893
That attempt had to raise the limit on the jar sizes and still hit test issues with HBase testing. It may not be feasible to upgrade Zookeeper until HBase and maybe other dependencies of Drill have uptaken newer versions of Zookeeper themselves.

letian-jiang
letian-jiang150 days ago

Upgrading zookeeper is not that easy. The CI build has already failed with an issue around protobuf. This is a recent attempt to upgrade - #2893 That attempt had to raise the limit on the jar sizes and still hit test issues with HBase testing. It may not be feasible to upgrade Zookeeper until HBase and maybe other dependencies of Drill have uptaken newer versions of Zookeeper themselves.

Ok. Thanks for the information.

letian-jiang letian-jiang closed this 150 days ago
cgivre cgivre reopened this 150 days ago
cgivre
cgivre150 days ago

@letian-jiang Why don't you try also updating the HBase client? Drill is currently running version 2.5 and there is a newer version available on MavenCentral.

pjfanning
pjfanning150 days ago👍 1

I created #2968 - maybe we can hold off with zookeeper upgrade until we know if that can be merged ok

cgivre
cgivre144 days ago

@pjfanning @letian-jiang All the other dependency libraries have been updated. Please rebase on current master and let's see if we can get this working.

cgivre cgivre added code-cleanup
cgivre cgivre added dependencies
letian-jiang Merge remote-tracking branch 'upstream/master' into bump-zk
457b7544
letian-jiang
letian-jiang144 days ago

@cgivre Done. Could you rerun the CI?

cgivre cgivre assigned letian-jiang letian-jiang 144 days ago
pjfanning
pjfanning143 days ago

The hadoop-2 build failure was because this setting is too small

<jdbc-all-jar.maxsize>55000000</jdbc-all-jar.maxsize>

Could you change it to this?

<jdbc-all-jar.maxsize>56000000</jdbc-all-jar.maxsize>
letian-jiang increase jar size
daf23e4f
cgivre cgivre added backport-to-stable
letian-jiang
letian-jiang143 days ago

Error: Errors:
Error: PhoenixTestSuite.initPhoenix:54 » Runtime Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterKeeperErrorCode = ConnectionLoss for /hbase
Error: SecuredPhoenixTestSuite.initPhoenix:55 » Runtime Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterKeeperErrorCode = ConnectionLoss for /hbase

letian-jiang
letian-jiang142 days ago

@pjfanning @cgivre Is it ok to simply rerun the CI again? This maybe an accidental error.

pjfanning
pjfanning142 days ago

The test issues seem related to the zookeeper upgrade.

pjfanning
pjfanning142 days ago

I think we should delay this and instead concentrate on upgrading the version of Phoenix that we use. The newer version of Phoenix seems to use newer HBase and Zookeeper versions.

pjfanning
pjfanning139 days ago

I don't know much about Phoenix and upgrading its version is not straightforward - #2972

Another option here is to look at shading the zookeeper classes and including those in one of the Drill jars so that we can upgrade the zookeeper version used by the Drill code but not affect the version used by Phoenix plugin.

cgivre
cgivre139 days ago

I think the shading option would work. If someone is using Phoenix and runs into issues we can revisit.

letian-jiang
letian-jiang136 days ago

Thanks for the advice. I will try this in the comming weeks.

cgivre
cgivre66 days ago

@letian-jiang Are you still working on this?

cgivre
cgivre20 days ago

@pjfanning @letian-jiang
We might be able to complete this PR once #2987 gets merged.

pjfanning
pjfanning19 days ago

The issue is more about Apache Phoenix and it relying on an old version of zookeeper. Unless they do a release that upgrades zookeeper then we are stuck with having to look at shading the zookeeper jar that we use in Drill code.

pjfanning
pjfanning19 days ago

It might be better if Drill advices people to keep their zookeeper installs protected from the public internet or untrusted users by using network allow lists.

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
Labels
Milestone