v0.11.0 dbt_marketo

PR #33 includes the following updates:

Bug Fix

• Removed the use of ignore nulls statements in marketo__lead_history and marketo__change_data_scd, which was incompatible with PostgreSQL and Databricks Runtime. The logic has been updated with a new approach but produces the same results as before.
• Updated model marketo__change_data_pivot to use the activity_id as a tie-breaker to remove randomness when ordering events having the same activity_timestamp.
  • Previously if two events happened at the same timestamp, results would be inconsistent, which propagated to downstream models. Now, this model will produce consistent results.

Under the hood

• Added additional variable configurations to integration tests to account for a wider range of situations.

---

PR #32 and Marketo Source PR #35 include the following updates:

Feature Updates (includes 🚨 breaking changes 🚨)

• Ensures that stg_marketo__lead (and therefore marketo__leads) has and documents the below columns, all standard fields from Marketo. Previously, peristed all fields found in your LEAD source table but only ensured that the id, created_at, updated_at, email, first_name, last_name, and _fivetran_synced fields were included. If any of the following default columns are missing from your LEAD table, stg_marketo__lead will create a NULL version with the proper data type:
  • phone
  • main_phone
  • mobile_phone
  • company
  • inferred_company
  • address_lead
  • address
  • city
  • state
  • state_code
  • country
  • country_code
  • postal_code
  • billing_street
  • billing_city
  • billing_state
  • billing_state_code
  • billing_country
  • billing_country_code
  • billing_postal_code
  • inferred_city
  • inferred_state_region
  • inferred_country
  • inferred_postal_code
  • inferred_phone_area_code
  • anonymous_ip
  • unsubscribed -> aliased as is_unsubscribed (🚨 breaking change 🚨)
  • email_invalid -> aliased as is_email_invalid (🚨 breaking change 🚨)
  • do_not_call

Under the Hood

• Updated the maintainer PR template to resemble the most up to date format.
• Included auto-releaser GitHub Actions workflow to automate future releases.

Full Changelog: v0.10.0...v0.11.0

v0.10.0 dbt_marketo

🚨 Breaking Changes 🚨 (recommend --full-refresh):

PR #28 includes the following updates:

• The source package was updated in connection with the Fivetran Marketo connector's June 2023 and May 2023 releases. This affects the columns created for the following tables:
  • marketo__campaigns
  • marketo__email_sends
  • marketo__programs
  • marketo__leads
• See the source package changelog for more details.
• We recommend using dbt run --full-refresh the next time you run your project due to changes affecting incremental models.

🚘 Under the Hood:

PR #28 includes the following updates:

• Update documentation and testing seed data

PR #27 includes the following updates:

• Incorporated the new fivetran_utils.drop_schemas_automation macro into the end of each Buildkite integration test job.
• Updated the pull request templates.

Full Changelog: v0.9.1...v0.10.0

v0.9.1 dbt_marketo

Happy international ditch new years resolution day!

This year I wanted to cook dinner more often, looks like I am ordering in tonight! 🍔 One resolution I am not giving up is providing a comprehensive overview of the changes that took place within the latestet release of dbt_marketo. These changes include:

Bug Fix

PR #26 includes the following non-breaking change:

• Due to a "system glitch" within Marketo, the same lead_id can be deleted more than once. This may introduce multiple records for the same lead_id in the int_marketo__lead which can cause the downstream test marketo__leads.lead_id.unique to fail. The bug fix adds a distinct lead_id to the deleted_leads CTE in the int_marketo__lead model (#25).

Contributors

• @m-feeser (#26)

Full Changelog: v0.9.0...v0.9.1

dbt_marketo v0.9.0

🚨 Breaking Changes 🚨:

PR #24 includes the following breaking changes:

• Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically {{ dbt_utils.<macro> }} have been updated to {{ dbt.<macro> }} for the below macros:
  • any_value
  • bool_or
  • cast_bool_to_text
  • concat
  • date_trunc
  • dateadd
  • datediff
  • escape_single_quotes
  • except
  • hash
  • intersect
  • last_day
  • length
  • listagg
  • position
  • replace
  • right
  • safe_cast
  • split_part
  • string_literal
  • type_bigint
  • type_float
  • type_int
  • type_numeric
  • type_string
  • type_timestamp
  • array_append
  • array_concat
  • array_construct
• For current_timestamp and current_timestamp_in_utc macros, the dispatch AND the macro names have been updated to the below, respectively:
  • dbt.current_timestamp_backcompat
  • dbt.current_timestamp_in_utc_backcompat
• dbt_utils.surrogate_key has also been updated to dbt_utils.generate_surrogate_key. Since the method for creating surrogate keys differ, we suggest all users do a full-refresh for the most accurate data. For more information, please refer to dbt-utils release notes for this update.
• packages.yml has been updated to reflect new default fivetran/fivetran_utils version, previously [">=0.3.0", "<0.4.0"] now [">=0.4.0", "<0.5.0"].

dbt_marketo v0.8.0

Hello again, friends! 😸

PR #22 incorporates the following updates:

🚨 Breaking Changes 🚨

Some of the more complex transformation logic has been moved from the Marketo source package to the transform package. This was done so the delineation between staging and intermediate models is in line with Fivetran's other packages. This does not affect the final tables created by the transform package, but this will affect the staging tables as outlined below.

• Model stg_marketo__lead_base from dbt_marketo_source has been rolled into stg_marketo__lead.
• Parts from model stg_marketo__lead from dbt_marketo_source have been moved to a new model int_marketo__lead in dbt_marketo.
• Because of the above changes, model marketo__lead_adapter is now redundant and has been removed.

Features

• 🎉 Databricks and Postgres compatibility 🎉
• The starting date of the date range for the leads data can now be adjusted (see README for instructions).
• Ability to disable activity_delete_lead model if necessary (see README for instructions).

Under the Hood

• Updates structure of config default variables for enabling campaigns and program models to avoid conflicting with a user's settings.
• Updates the incremental strategy used by Postgres and Redshift adapters to delete+insert.

dbt_marketo v0.7.0

Bug Fixes

• Previously, merged_into_lead_id and lead_id were erroneously switched in stg_marketo__lead. This release switches them back, appropriately casting merged_into_lead_id as a string (it can have multiple comma-separated values) and lead_id as an integer (#17).

This is a BREAKING CHANGE as you will need to run a full refresh due to the columns' differing data types.

marketo 0.6.0

🎉 dbt v1.0.0 Compatibility 🎉

🚨 Breaking Changes 🚨

• Adjusts the require-dbt-version to now be within the range [">=1.0.0", "<2.0.0"]. Additionally, the package has been updated for dbt v1.0.0 compatibility. If you are using a dbt version <1.0.0, you will need to upgrade in order to leverage the latest version of the package.
  • For help upgrading your package, I recommend reviewing this GitHub repo's Release Notes on what changes have been implemented since your last upgrade.
  • For help upgrading your dbt project to dbt v1.0.0, I recommend reviewing dbt-labs upgrading to 1.0.0 docs for more details on what changes must be made.
• Upgrades the package dependency to refer to the latest dbt_marketo_source. Additionally, the latest dbt_marketo_source package has a dependency on the latest dbt_fivetran_utils. Further, the latest dbt_fivetran_utils package also has a dependency on dbt_utils [">=0.8.0", "<0.9.0"].
  • Please note, if you are installing a version of dbt_utils in your packages.yml that is not in the range above then you will encounter a package dependency error.

Under the Hood

• Redshift recently included pivot as a reserved word within the warehouse. As such, the pivot CTE within the marketo__change_data_pivot and marketo__change_data_details models have been changed to pivots to avoid the Redshift error.

dbt 0.20.0 Compatibility

🎉 dbt 0.20.0 Compatibility 🎉

🚨 This is a breaking change! 🚨 dbt v0.20.0 or greater is required for this release. If you are not ready to upgrade, consider using a previous release of this package.

Additional considerations when upgrading to this package:

• This package utilizes the latest v0.7.0 release of the dbt-labls/dbt_utils package. If your project also utilizes a version of the fishtown-analytics/dbt_utils package then you will receive a duplicate package error. If this is the case you will need to consider upgrading your other packages to be compatible with this update or use a previous release of this package.
• Similar to the above point, all previous Fivetran dbt packages utilize the fishtown-analytics/dbt_utils package and you will need to upgrade all Fivetran dbt packages to the latest dbt 0.20.0 compatibility releases in order for your packages to run without package conflicts.

Breaking Changes: Disable Campaign & Program Tables

🚨 This update introduces a breaking change. By default, the campaign & program staging models are now disabled.

Fivetran recommends not syncing these tables due to slowness they may cause. If you sync these tables, you will need to enable these models in your dbt_project.yml file as outlined here.

Breaking Changes: Custom Build Schemas + Updates

🚨 This update introduces a breaking change in the form of custom schemas, which output models into a respective <target.schema>+marketo schema. Refer to the ReadMe for more details. 🚨

This release also updates dbt version requirements to include 0.19.*