Read UC delta table (#773)
* initial commit
* use databricks-sql to read delta table and convert to json
* update
* update
* update
* add mocked unittest
* Fix lints
* update
* update
* restructure code
* Add timer for optimizing
* Add db-connect
* add wrapper
* update
* add install dbconnect
* update
* update
* patch dbconnect to allow multiple return formats
* update
* add arrow
* use compression
* clean up
* Add cluster rt check
* Fix lints
* remove patch.py for CI
* update
* update
* updat
* update
* fix tests
* fix lint
* update
* update
* Add more tests
* update
* update
* update
* change to download_json
* update
* fix lints
* Add decompressed option for arrow
* format json to jsonl
* Add comments
* Make cf_collect_type global option
* fix comments
* fix lints
* fix comments
* Fix lints
* change to use workspaceclient
* Add CPT support
* Rewire method assignment logic
* Fix bug in stripping https
* Add tests for rewired method assignment logic
* Fix lints
* Fix lints
* Removed logger set_level
* Remove pyspark. It conflicts with databricks-connect
* Update the comment
* skip cluster version check when cluster_id is serverless
* Add use_serverless flag
* update tests with use_serverless flag
* Fix lints
---------
Co-authored-by: Xiaohan Zhang <xiaohan.zhang@databricks.com>
