drill

073ea681 - DRILL-5601: Rollup of external sort fixes an improvements

DRILL-5601: Rollup of external sort fixes an improvements - DRILL-5513: Managed External Sort : OOM error during the merge phase - DRILL-5519: Sort fails to spill and results in an OOM - DRILL-5522: OOM during the merge and spill process of the managed external sort - DRILL-5594: Excessive buffer reallocations during merge phase of external sort - DRILL-5597: Incorrect "bits" vector allocation in nullable vectors allocateNew() - DRILL-5602: Repeated List Vector fails to initialize the offset vector - DRILL-5617: Spill file name collisions when spill file is on a shared file system 0 DRILL-5445: bug in repeated map vector deserialization - Workaround for DRILL-5656: Streaming Agg Batch forces sort to retain in-memory batches past NONE - Fixes for the "record batch sizer" to handle for UNION, MAP, LIST types Fixes a longstanding bug in the deserialization of a repeated map vector read from a spill file. A few minor code cleanups also. All of the bugs have to do with handling low-memory conditions, and with correctly estimating the sizes of vectors, even when those vectors come from the spill file or from an exchange. Hence, the changes for all of the above issues are interrelated. Also includes some fixes for tests: * Certain tests require the ability to enforce the output size of the memory merge/sort. Restored this option. * Resolve issue with TestDrillbitResilience * A particular test injects a fault during in-memory sort, but used a single-batch input (which does not need the merge phase.) Rather than introduce a new config property (the earlier solution), altered the test to use input that returns more than one batch. Two fixes forBasicPhysicalOpUnitTest — a test that uses JMockit to create a “fake” fragment context. 1. No drillbit endpoint is available, so the SpillSet change that adds the node to the spill path failed. Solution was to omit the node path segment in such tests. 2. The Dynamic UDF registry is null causing a crash. This has nothing to do with sort. Perhaps some pre-existing error? Anyway, added a check for this condition. close #860