dbt-core Python dependency audit

[aranke]

A place to discuss dependencies dbt-core should include or exclude in the PyPI distribution.
See a current list of included dependencies here.

Ground rules:

1. Please create a comment for each type of dependency (e.g., CLI library) dbt should include or exclude instead of a specific solution (e.g., Click).
2. Please leave questions/comments/concerns about a dependency type or alternatives as a reply to the comment.
3. Developer dependencies (e.g., pytest flake8, mypy, black) that are not included in the PyPI distribution are out of scope for this discussion.

Looking forward to a great discussion!

[aranke]

CLI Formatting

We're currently using colorama.

[aranke]

I'd like us to move to Rich for the following reasons:

1. Much more popular (42k ⭐ vs. 3k ⭐)
2. Does more formatting than simply coloring output, including:
   1. Tables
   2. Progress bars and spinners
   3. Syntax highlighting and rich error trace backs
3. Integrates well with our new CLI library click via rich-click.

[jtcohen6]

Serialization & Validation

For certain validation steps, we use hologram, a package that we maintain (not very actively). It makes heavy use of jsonschema.

For serialization (and some validation), we use a mix of mashumaro and betterproto (recent addition).

[callum-mcdata]

Crazy idea:

• Maybe we consider pydantic? It appears that pydantic does a combination of both (validation & serialization). The V2 release is currently slated for end of Q1 and contains significant rewrites in Rust to increase performance. But perhaps most interestingly, it autoproduces your jsonschema for you. Gimme Gimme

[iknox-fa]

The performance of Pydantic will be a forcing factor here for sure. We only use mashumaro because it's so much more performant than other options. IIRC, Pydantic was explored in the past and was found to be orders of magnitude too slow for our use case. I know none of us like hologram, it's super buggy and doesn't always create valid jsonschema (!). If we had a solution to fix both of those it would be a big win.

[iknox-fa]

It turns out I didn't recall correctly and Pydantic wasn't considered in the earlier work involving parsing performance, however, the benchmarks done by mashumaro suggests that legacy versions would be a significant slowdown.

Internal slack discussion (for reference only, pertinent details are summarized here)

[aranke]

It's not as mature as the other options here, but I'll throw apischema into the mix because:

1. It claims to be faster than similar libraries like mashumaro.
2. I like the design decision to favor composition over inheritance.

[aranke]

Another one that looks promising: https://jcristharif.com/msgspec/

[jtcohen6]

Dates & times

We currently use pytz for:

• providing users with timezone logic in the base Jinja context
• calculating source freshness
• anonymous usage tracking

[jtcohen6]

Other projects have moved to using zoneinfo, which is part of the standard library for Python 3.9+

e.g. python-babel/babel#716

Python 3.8's end of life is 14 October 2024, so we'd need some sort of backward compatibility until then.