DRILL-2245: Clean up query setup and execution kickoff in Foreman/WorkManager in order to ensure consistent handling, and avoid hangs and races, with the goal of improving Drillbit robustness.
I did my best to keep these clean when I split them up, but this core commit
may depend on some minor changes in the hygiene commit that is also
associated with this bug, so either both should be applied, or neither.
The core commit should be applied first.
protocol/pom.xml
- updated protocol buffer compiler version to 2.6
- this made slight modifications to the formats of a few committed protobuf
files
AutoCloseables
- created org.apache.drill.common.AutoCloseables to handle closing these
quietly
BaseTestQuery, and derivatives
- factored out pieces into QueryTestUtil so they can be reused
DeferredException:
- created this so we can collect exceptions during the shutdown process
Drillbit
- uses AutoCloseables for the WorkManager and for the storeProvider
- allow start() to take a RemoteServiceSet
- private, final, formatting
Foreman
- added new state CANCELLATION_REQUESTED (via UserBitShared.proto) to represent
the time between request of a cancellation, and acknowledgement from all
remote endpoints running fragments on a query's behalf
- created ForemanResult to manage interleaving cleanup effects/failure with
query result state
- does not need to implement Comparable
- does not need to implement Closeable
- thread blocking fixes
- add resultSent flag
- add code to log plan fragments with endpoint assignments
- added finals, cleaned up formatting
- do queue management in acquireQuerySemaphore; local tests pass
- rename getContext() to getQueryContext()
- retain DrillbitContext
- a couple of exception injections for testing
- minor formatting
- TODOs
FragmentContext
- added a DeferredException to collect errors during startup/shutdown sequences
FragmentExecutor
- eliminated CancelableQuery
- use the FragmentContext's DeferredException for errors
- common subexpression elimination
- cleaned up
QueryContext
- removed unnecessary functions (with some outside classes tweaked for this)
- finals, formatting
QueryManager
- merge in QueryStatus
- affects Foreman, ../batch/ControlHandlerImpl,
and ../../server/rest/ProfileResources
- made some methods private
- removed unused imports
- add finals and formatting
- variable renaming to improve readability
- formatting
- comments
- TODOs
QueryStatus
- getAsInfo() private
- member renaming
- member access changes
- formatting
- TODOs
QueryTestUtil, BaseTestQuery, TestDrillbitResilience
- make maxWidth a parameter to server startup
SelfCleaningRunnable
- created org.apache.drill.common.SelfCleaningRunnable
SingleRowListener
- created org.apache.drill.SingleRowListener results listener
- use in TestDrillbitResilience
TestComparisonFunctions
- fix not to close the FragmentContext multiple times
TestDrillbitResilience
- created org.apache.drill.exec.server.TestDrillbitResilience to test drillbit
resilience in the face of exceptions and failures during queries
TestWithZookeeper
- factor out work into ZookeeperHelper so that it can be reused by
TestDrillbitResilience
UserBitShared
- get rid of unused UNKNOWN_QUERY
WorkEventBus
- rename methods, affects Foreman and ControlHandlerImpl
- remove unused WorkerBee reference
- most members final
- formatting
WorkManager
- Closeable to AutoCloseable
- removed unused incomingFragments Set
- eliminated unnecessary eventThread and pendingTasks by posting Runnables
directly to executor
- use SelfCleaningRunnable for Foreman management
- FragmentExecutor management uses SelfCleaningRunnable
- runningFragments to be a ConcurrentHashMap; TestTpchDistributed passes
- other improvements due to bee no longer needed in various places
- most members final
- minor formatting
- comments
- TODOs
(*) Created exception injection classes to simulate exceptions for testing
- ExceptionInjection
- ExceptionInjector
- ExceptionInjectionUtil
- TestExceptionInjection
DRILL-2245-hygiene: General code cleanup encountered while working on the rest
of this commit. This includes
- making members final whenever possible
- making members private whenever possible
- making loggers private
- removing unused imports
- removing unused private functions
- removing unused public functions
- removing unused local variables
- removing unused private members
- deleting unused files
- cleaning up formatting
- adding spaces before braces in conditionals and loop bodies
- breaking up overly long lines
- removing extra blank lines
While I tried to keep this clean, this commit may have minor dependencies on
DRILL-2245-core that I missed. The intention is just to break this up for
review purposes. Either both commits should be applied, or neither.