[CT-947] Test_incremental_strategy failed on main

[ChenyuLInx]

Describe the bug

tests/integration/incremental_strategies/test_incremental_strategies.py::TestInsertOverwrite along with some other tests are failing in current main branch. Noticed the failure while working on getting the tests working in #377. All other tests are passing now there but the TestInsertOverwrite is still failing.

Log for failure in main

Steps To Reproduce

I wasn't able to run the test locally but linked the fail log above

Expected behavior

Tests passing

[jtcohen6]

We switched over to using a high-concurrency Databricks cluster for CI, which is the only way to support both access control and Python + SQL.

https://docs.databricks.com/security/access-control/table-acls/table-acl.html#create-a-cluster-enabled-for-table-access-control

While setting up that new cluster, I forgot the very important detail that I had previously documented (https://docs.getdbt.com/reference/resource-configs/spark-configs#incremental-models):

If connecting via a Databricks cluster + ODBC driver (method: odbc + cluster), you must include set spark.sql.sources.partitionOverwriteMode DYNAMIC in the cluster Spark Config in order for dynamic partition replacement to work (incremental_strategy: insert_overwrite + partition_by).

Success on rerun for branch in #377: https://app.circleci.com/pipelines/github/dbt-labs/dbt-spark/1198/workflows/bf4ec4c2-dd2e-4228-b86e-1af5ee51d9c6/jobs/4642

[ChenyuLInx]

Test has moved back and now all tests on main passes