DRILL-6090: While connecting to drill-bits using JDBC Driver through Zookeeper, a lot of "Curator-Framework-0" threads are created if connection to drill-bit is not successful(no drill-bits are up/reachable)
I am using Drill JDBC driver 1.12.0 to connect to MapR-DB. I am finding the available drill-bits using Zookeepers. When drill-bits are not up or not reachable, the connection is failed with exception: "Failure in connecting to Drill: oadd.org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client", which is expected, but number of threads created by ZKClusterCoordinator just keeps on increasing.
Steps to reproduce the issue
Setup a connection with a drill-bit using Apache Drill JDBC driver 1.12.0 through Zookeeper hosts(port 5181)
Now stop the drill-bit services or block the drill-bit IPs using iptable rules
Truncate catalina logs
Try to connect to the drill-bit/hit a code path that requires connection to drill-bits.
Take thread dump using kill -QUIT <java process id>
grep -c "Curator-Framework-0" catalina.out
Observe that the curator framework thread just keep on accumulating
RCA:
ZKClusterCoordinator creates curator threads in the constructor
ZKClusterCoordinator is instantiated by DrillClient.connect
DrillClient.connect is called in DrillConnectionImpl constructor
Fix:
Call DrillConnectionImpl .cleanup() from all the catch blocks in the DrillConnectionImpl constructor.
close apache/drill#1094
