DRILL-8514: Bump zookeeper to 3.9.3 #2967
DRILL-8514: Bump zookeeper to 3.9.3
Upgrading zookeeper is not that easy. The CI build has already failed with an issue around protobuf.
This is a recent attempt to upgrade - #2893
That attempt had to raise the limit on the jar sizes and still hit test issues with HBase testing. It may not be feasible to upgrade Zookeeper until HBase and maybe other dependencies of Drill have uptaken newer versions of Zookeeper themselves.
Upgrading zookeeper is not that easy. The CI build has already failed with an issue around protobuf. This is a recent attempt to upgrade - #2893 That attempt had to raise the limit on the jar sizes and still hit test issues with HBase testing. It may not be feasible to upgrade Zookeeper until HBase and maybe other dependencies of Drill have uptaken newer versions of Zookeeper themselves.
Ok. Thanks for the information.
@letian-jiang Why don't you try also updating the HBase client? Drill is currently running version 2.5 and there is a newer version available on MavenCentral.
I created #2968 - maybe we can hold off with zookeeper upgrade until we know if that can be merged ok
@pjfanning @letian-jiang All the other dependency libraries have been updated. Please rebase on current master and let's see if we can get this working.
Merge remote-tracking branch 'upstream/master' into bump-zk
@cgivre Done. Could you rerun the CI?
The hadoop-2 build failure was because this setting is too small
<jdbc-all-jar.maxsize>55000000</jdbc-all-jar.maxsize>
Could you change it to this?
<jdbc-all-jar.maxsize>56000000</jdbc-all-jar.maxsize>
Error: Errors:
Error: PhoenixTestSuite.initPhoenix:54 » Runtime Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterKeeperErrorCode = ConnectionLoss for /hbase
Error: SecuredPhoenixTestSuite.initPhoenix:55 » Runtime Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterKeeperErrorCode = ConnectionLoss for /hbase
@pjfanning @cgivre Is it ok to simply rerun the CI again? This maybe an accidental error.
The test issues seem related to the zookeeper upgrade.
I think we should delay this and instead concentrate on upgrading the version of Phoenix that we use. The newer version of Phoenix seems to use newer HBase and Zookeeper versions.
I don't know much about Phoenix and upgrading its version is not straightforward - #2972
Another option here is to look at shading the zookeeper classes and including those in one of the Drill jars so that we can upgrade the zookeeper version used by the Drill code but not affect the version used by Phoenix plugin.
I think the shading option would work. If someone is using Phoenix and runs into issues we can revisit.
Thanks for the advice. I will try this in the comming weeks.
@letian-jiang Are you still working on this?
@pjfanning @letian-jiang
We might be able to complete this PR once #2987 gets merged.
The issue is more about Apache Phoenix and it relying on an old version of zookeeper. Unless they do a release that upgrades zookeeper then we are stuck with having to look at shading the zookeeper jar that we use in Drill code.
It might be better if Drill advices people to keep their zookeeper installs protected from the public internet or untrusted users by using network allow lists.
Login to write a write a comment.
Bump zookeeper to 3.9.3
https://issues.apache.org/jira/browse/DRILL-8514