Broken DBT 18 update on Brew ... DBT doesn't support Python 3.9 #2865

lucaswiley · 2020-11-03T19:23:54Z

Describe the bug

This issue originally started when I merged a change into production to begin using the Shopify DBT package. The merge broke because our prod machine was still on dbt 17. So when I went to upgrade I hit the below issues.

Currently, I get the below stack trace when running dbt --version while trying to upgrade to dbt 18.1. I imagine this has something to do with my brew/python setup environment but am not quite sure exactly how to debug after about a day of brute forcing various solutions. Thanks for your help.

Steps To Reproduce

Install brew
Install Git and DBT via Brew
Run dbt --version

Expected behavior

dbt --version to return the updated version

System information

Which database are you using dbt with?

The output of dbt --version:

Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/bin/dbt", line 33, in <module>
    sys.exit(load_entry_point('dbt-core==0.18.1', 'console_scripts', 'dbt')())
  File "/home/linuxbrew/.linuxbrew/bin/dbt", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/main.py", line 11, in <module>
    import dbt.version
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/version.py", line 100, in <module>
    installed = get_installed_version()
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/version.py", line 29, in get_installed_version
    return dbt.semver.VersionSpecifier.from_version_string(__version__)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/dbt/semver.py", line 100, in from_version_string
    return cls.from_dict(matched)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 618, in from_dict
    cls.validate(data)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 957, in validate
    schema = _validate_schema(cls)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 183, in _validate_schema
    schema = schema_cls.json_schema()
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 942, in json_schema
    return cls._json_schema_recursive(
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 913, in _json_schema_recursive
    schema = cls._schema_defs_from_cache(definitions)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 900, in _schema_defs_from_cache
    schema = cls._collect_json_schema(definitions)
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 863, in _collect_json_schema
    properties[target_field], is_required = cls._get_field_schema(
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 809, in _get_field_schema
    field_schema, required = cls._get_schema_for_type(
  File "/home/linuxbrew/.linuxbrew/Cellar/dbt/0.18.1_1/libexec/lib/python3.9/site-packages/hologram/__init__.py", line 783, in _get_schema_for_type
    raise ValidationError(f"Unable to create schema for '{type_name}'")
hologram.ValidationError: Unable to create schema for 'Optional'

The operating system you're using:
Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1028-aws x86_64)

The output of python --version:
Python 3.9.0 (I do have multiple installs of python which may be influencing the issue)

Additional context

$ python --version
Python 3.9.0

$ brew --version
Homebrew 2.5.8
Homebrew/linuxbrew-core (git revision 071501; last commit 2020-11-03)

My best assumption is the issue lies here with the billion python installs, but not sure.

$ whereis dbt
dbt: /home/linuxbrew/.linuxbrew/bin/dbt

$ whereis python
python: /usr/bin/python3.6 /usr/bin/python3.6-config /usr/bin/python2.7 /usr/bin/python2.7-config /usr/bin/python /usr/bin/python3.6m-config /usr/bin/python3.6m /usr/lib/python3.6 /usr/lib/python3.8 /usr/lib/python2.7 /usr/lib/python3.7 /etc/python3.6 /etc/python2.7 /etc/python /usr/local/lib/python3.6 /usr/local/lib/python2.7 /usr/include/python3.6 /usr/include/python2.7 /usr/include/python3.6m /usr/share/python /home/linuxbrew/.linuxbrew/bin/python3.9 /home/linuxbrew/.linuxbrew/bin/python3.9-config /home/airflow/onedrop-scripts/transform/dbt/dbt-env/bin/python /usr/share/man/man1/python.1.gz

$ whereis brew
brew: /home/linuxbrew/.linuxbrew/bin/brew

$ which dbt
/home/linuxbrew/.linuxbrew/bin/dbt

$ which python
/usr/bin/python

$ which brew
/home/linuxbrew/.linuxbrew/bin/brew

Thank you so much

The text was updated successfully, but these errors were encountered:

gshank · 2020-11-03T23:41:32Z

It's possible that hologram version doesn't support Python 3.9. There were some changes in a recent hologram commit to support 3.9, but it hasn't been packaged yet and dbt support for 3.9 hasn't been done yet.

lucaswiley · 2020-11-05T00:19:00Z

Thanks, you're right. Resolved via instructions in this thread to use pip3 instead of brew: https://getdbt.slack.com/archives/C2JRRQDTL/p1594318582280200?thread_ts=1594253045.233300&cid=C2JRRQDTL

mycaule · 2020-11-25T07:26:33Z

Experimenting the same bug inside a GitHub Action (python 3.9).
@lucaswiley can you provide instructions on how to solve here (without having to log in to Slack) ? Thank you.

      - name: Setup Python
        uses: actions/[email protected]

      - name: Setup dbt
      - run: |
          pip install dbt
          dbt --version

lucaswiley · 2020-11-25T19:29:52Z

There are broken dependencies on the DBT brew installation for Python 3.9. Instead use pip3:

brew remove dbt
pip3 install virtualenv virtualenvwrapper
virtualenv foo
pip3 install dbt

mrafayaleem · 2020-11-30T15:48:32Z

I am using Conda and this is what I did to resolve this issue.

Downgrade python from 3.9 to 3.8:

conda install python=3.8

Do a force reinstall of dbt:

pip3 install --upgrade --force-reinstall dbt

Run dbt --version:

(dbt-poc) ➜  dbt-poc git:(master) dbt --version
installed version: 0.18.1
   latest version: 0.18.1

Up to date!

Plugins:
  - bigquery: 0.18.1
  - snowflake: 0.18.1
  - redshift: 0.18.1
  - postgres: 0.18.1

mycaule · 2020-11-30T23:02:12Z

This bug hasn't been solved, please reopen the ticket.

Here is the GitHub action workflow file running Python < 3.9

    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [ '3.6', '3.8' ]

    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Setup Python
        uses: actions/[email protected]

      - name: Setup dbt
        run: |
          python3 -m venv dbt-env
          source dbt-env/bin/activate
          pip3 install dbt
          dbt --version

jtcohen6 · 2020-11-30T23:24:33Z

The latest version of dbt (v0.18.1) does not support python 3.9. We plan to add py39 support (#2822) for the next release of dbt (v0.19.0). If it proves straightforward, I'd also like to release a patch version (v0.18.2) that is compatible with python 3.9 as well.

In the meantime, I am going to close this issue as a duplicate of #2822.

Startouf · 2021-05-18T22:49:49Z

Hello, I still have this issue with sqlfluff, dbt 0.19 and python 3.9

pip3 --version
pip 21.1.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
python3 --version
Python 3.9.4
dbt --version
installed version: 0.19.1
sqlfluff --version
sqlfluff, version 0.5.6

When running sqlfluff lint with templater = dbt. WHen I remove template = dbt it runs, but jinja seems to be broken. Is that to be expected ?

jtcohen6 · 2021-05-19T19:30:49Z

Hey @Startouf, dbt-core==0.19.1 is compatible with py39, but one of the plugins included by pip install dbt (dbt-snowflake==0.19.1) is not compatible with py39. The next minor version (v0.20) will be.

I'm not positive if that clarifies the jinja issue you're seeing, but suffice to say, it is expected that pip install dbt==0.19.1 + py39 don't yet work together. If you know which adapter you're using, however, and it isn't Snowflake, you can install that in a py39 environment just fine, e.g. pip install dbt-postgres.

mike-seekwell · 2021-05-26T15:29:20Z

confirmed this works in a clean py39 env

pip install dbt-postgres==0.19.1
dbt --version
installed version: 0.19.1
   latest version: 0.19.1

Up to date!

Plugins:
  - postgres: 0.19.1

jeroenvlek · 2021-06-23T08:39:22Z

Still encountering this issue for 0.19.1 with a fresh dbt install on Arch Linux and python 3.9. Creating a virtualenv with 3.7 fixed it for me.

levietha92 · 2021-08-26T05:13:08Z

pip3 install --upgrade --force-reinstall dbt

thanks! this works for me 👍🏽 was trying to install dbt 0.18.0

lucaswiley added bug Something isn't working triage labels Nov 3, 2020

lucaswiley changed the title ~~Broken DBT 18 update: Ubuntu, Brew~~ Broken DBT 18 update: Ubuntu, Brew ... hologram.ValidationError: Unable to create schema for 'Optional' Nov 3, 2020

lucaswiley closed this as completed Nov 5, 2020

lucaswiley changed the title ~~Broken DBT 18 update: Ubuntu, Brew ... hologram.ValidationError: Unable to create schema for 'Optional'~~ Broken DBT 18 update on Brew ... DBT doesn't support Python 3.9 Nov 5, 2020

jtcohen6 removed the triage label Nov 5, 2020

dmateusp mentioned this issue Nov 15, 2020

371 Optionally use DBT compiler as the TemplaterInterface sqlfluff/sqlfluff#508

Merged

lucaswiley reopened this Nov 30, 2020

jtcohen6 closed this as completed Nov 30, 2020

jtcohen6 added the duplicate This issue or pull request already exists label Nov 30, 2020

MartinNowak mentioned this issue May 11, 2021

test dbt with Python 3.9 #2822

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Broken DBT 18 update on Brew ... DBT doesn't support Python 3.9 #2865

Broken DBT 18 update on Brew ... DBT doesn't support Python 3.9 #2865

lucaswiley commented Nov 3, 2020 •

edited

Loading

gshank commented Nov 3, 2020

lucaswiley commented Nov 5, 2020

mycaule commented Nov 25, 2020

lucaswiley commented Nov 25, 2020

mrafayaleem commented Nov 30, 2020

mycaule commented Nov 30, 2020 •

edited

Loading

jtcohen6 commented Nov 30, 2020

Startouf commented May 18, 2021

jtcohen6 commented May 19, 2021

mike-seekwell commented May 26, 2021 •

edited

Loading

jeroenvlek commented Jun 23, 2021

levietha92 commented Aug 26, 2021

Broken DBT 18 update on Brew ... DBT doesn't support Python 3.9 #2865

Broken DBT 18 update on Brew ... DBT doesn't support Python 3.9 #2865

Comments

lucaswiley commented Nov 3, 2020 • edited Loading

Describe the bug

Steps To Reproduce

Expected behavior

System information

Additional context

gshank commented Nov 3, 2020

lucaswiley commented Nov 5, 2020

mycaule commented Nov 25, 2020

lucaswiley commented Nov 25, 2020

mrafayaleem commented Nov 30, 2020

mycaule commented Nov 30, 2020 • edited Loading

jtcohen6 commented Nov 30, 2020

Startouf commented May 18, 2021

jtcohen6 commented May 19, 2021

mike-seekwell commented May 26, 2021 • edited Loading

jeroenvlek commented Jun 23, 2021

levietha92 commented Aug 26, 2021

lucaswiley commented Nov 3, 2020 •

edited

Loading

mycaule commented Nov 30, 2020 •

edited

Loading

mike-seekwell commented May 26, 2021 •

edited

Loading