Add support for ODBC Server Side Parameters

[jethron]

Description

Allow passing through Cluster Configuration parameters via profiles.yml when using the odbc connection method.
This allows configuration of values that would need to be set via SET to be run in the Spark session and take effect on DBT models.

E.g.:

spark_dbt:
  target: default
  outputs:
    default:
      type: spark
      method: odbc
      driver: /opt/simba/spark/lib/64/libsparkodbc_sb64.so
      schema: default
      host: [hostid]
      token: [token]
      cluster: [clusterid]
      port: 443
      connect_retries: 3
      connect_timeout: 20
      server_side_parameters:
          "spark.databricks.delta.schema.autoMerge.enabled": True
          "spark.sql.jsonGenerator.ignoreNullFields": True

Checklist

• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

[jtcohen6]

Thanks for the PR @jethron! This seems like a clever way of managing a gnarly configuration process. Have you been able to test this locally, to confirm that the configuration supplied in the ODBC string takes effect?

[jethron]

Hey @jtcohen6, I have.

With this patch I've successfully used the spark.databricks.delta.schema.autoMerge.enabled setting above to use Delta Lake Automatic Schema Evolution to update a table schema with a MERGE statement from an incremental model:

• Without this patch the MERGE fails because columns are added/missing between source/target table during the merge insert.
• With the patch and above configuration, the MERGE succeeds, rows are inserted, and the schema evolves as expected.

I find specifying it here in profiles.yml much neater than the platform-specific instructions for adding it to your ODBC driver configuration, much easier to document for other users of the models.

[jtcohen6]

That's great to hear, and you named exactly the setting for which I imagined this might prove especially useful.

I'm going to update the Changelog on the master branch now. After I do, could you pull master and add a changelog entry? Then we can get this in for v0.21!

[jethron]

Sorry for the delay, didn't see the edit. Done!

[jtcohen6]

Thanks for the contribution @jethron!