dev env clean up and improvements #3204

kwigley · 2021-03-25T20:53:05Z

resolves #3194

Description

update development dependencies (default to using the latest and greatest)
update code and installed a new pytest plugin named pytest-dotenv to load test.env secrets in testing environment in order to avoid running integration tests in a docker container locally.
remove scripts/dtr.py because it is an abstraction we shouldn't need anymore, default to using pytest to run individual tests, tox to run full test suites, and Makefile for commonly used commands
update tox.ini with better defaults and test envs, we are really close to only needing python and tox installed on your machine to be able to do everything related to development with dbt :)
update Makefile to run target on local machine by default, not in a docker container. Still provided option to run in a docker container using USE_DOCKER=true in case folks don't have a certain python version installed.
update CONTRIBUTING.md to reflect changes
CI runs a lot fast

Future ideas

use tox to create development virtualenvs with everything necessary to develop (example tox --devenv env -e py38 is the same thing as python3.8 -m venv env && source env/bin/activate && pip install -r dev-requirements.txt -r editable-requirements.txt)
create a VSCode dev container for folks who use VSCode
remove initial step to setup postgres database for integration tests
- explore using something like https:/avast/pytest-docker

Checklist

I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

test/rpc/conftest.py

gshank

This is great! A lot of good cleanup. Found a couple of typos and was wondering if we could take this opportunity to change line length checks?

tox.ini

CONTRIBUTING.md

nathaniel-may

I'm actually unfamiliar with the vast majority of changes you're proposing here so I'm not hitting approve, but I looked through it all and didn't see any red flags.

we are really close to only needing python and tox installed on your machine to be able to do everything related to development with dbt
I am so excited for this future.

Very excited about these changes. Thanks for all the hard work!

nathaniel-may · 2021-03-26T14:58:24Z

.circleci/config.yml

- name: Run tests
- command: tox -e integration-postgres-py39
+ name: Postgres integration tests
+ command: tox -p -e py36-postgres,py38-postgres -- -v -n4


Oh nice. This makes things much simpler. Good idea.

kwigley · 2021-03-26T16:38:54Z

core/setup.py

@@ -64,7 +64,7 @@ def read(fname):
 'sqlparse>=0.2.3,<0.4',
 'networkx>=2.3,<3',
 'minimal-snowplow-tracker==0.0.2',
- 'colorama>=0.3.9,<0.4.4',
+ 'colorama>=0.3.9,<0.4.5',


We had to pin this before because the package didn't provide the source distribution on PyPI, but they have since added it! 👍 https://pypi.org/project/colorama/0.4.4/#files

kwigley · 2021-03-26T16:45:37Z

test/integration/015_cli_invocation_tests/test_cli_invocation.py

@@ -149,15 +149,15 @@ def test_postgres_dbt_commands_with_cwd_as_project_dir(self):

 @use_profile('postgres')
 def test_postgres_dbt_commands_with_randomdir_as_project_dir(self):
- workdir = os.getcwd()
+ workdir = self.test_root_dir


Without this change, integration tests failed on MacOS because of how the os handles temp dir/file. Functionally, os.getcwd() and self.test_root_dir are the same value. os.getcwd() returns a fully resolved filepath while self.test_root_dir is a string value pointing to a symlinked path that is equivalent to os.getcwd().

jtcohen6

This is amazing! 🎤 → ⬇️

I was able to run locally with pytest, and it was a breeze.

jtcohen6 · 2021-03-26T17:48:42Z

CONTRIBUTING.md

@@ -128,100 +127,105 @@ Docker and docker-compose are both used in testing. Specific instructions for yo

 For testing, and later in the examples in this document, you may want to have `psql` available so you can poke around in the database and see what happened. We recommend that you use [homebrew](https://brew.sh/) for that on macOS, and your package manager on Linux. You can install any version of the postgres client that you'd like. On macOS, with homebrew setup, you can run:



Do you want to change/edit/remove the section about docker above, given that it's no longer a requirement for folks to test locally? Do we still want to make it available as an option?

Docker is still required to create and run a testing postgres database, the scripts to setup the database for testing (test/setup_db.sh) are specific to the database created by docker-compose. So it it still a requirement to setup a postgres database, but no longer needed to invoke tests locally. I'll try to make this more clear!

…ev workflow cleanup

cla-bot bot added the cla:yes label Mar 25, 2021

kwigley requested review from gshank, nathaniel-may and jtcohen6 March 25, 2021 20:53

kwigley self-assigned this Mar 25, 2021

gshank reviewed Mar 25, 2021

View reviewed changes

test/rpc/conftest.py Outdated Show resolved Hide resolved

gshank requested changes Mar 25, 2021

View reviewed changes

tox.ini Outdated Show resolved Hide resolved

CONTRIBUTING.md Outdated Show resolved Hide resolved

kwigley force-pushed the updates/tox branch from 0b25dee to 84164e3 Compare March 25, 2021 21:47

This was referenced Mar 26, 2021

Bump virtualenv from 20.0.3 to 20.4.3 #3173

Closed

Bump bumpversion from 0.5.3 to 0.6.0 #3084

Closed

Bump tox from 3.14.4 to 3.23.0 #3144

Closed

nathaniel-may reviewed Mar 26, 2021

View reviewed changes

kwigley requested a review from gshank March 26, 2021 16:14

kwigley force-pushed the updates/tox branch from 84164e3 to 1ca69ef Compare March 26, 2021 16:32

kwigley commented Mar 26, 2021

View reviewed changes

gshank approved these changes Mar 26, 2021

View reviewed changes

jtcohen6 approved these changes Mar 26, 2021

View reviewed changes

kwigley force-pushed the updates/tox branch 2 times, most recently from 9176a35 to ba03988 Compare March 28, 2021 19:26

Kyle Wigley added 9 commits March 29, 2021 09:11

update tox, update makefile, run tests natively by default, general d…

2c67e3f

…ev workflow cleanup

use new docker image for tests

26e5010

CI tweaks

f5880cb

fix last CI step

2783fe2

update changelog

777e7b3

fix typos, set max-line-length (force azure pipeline tests)

39fd53d

debug windows tests

943b090

fix windows rpc tests

80031d1

fix circleci tests (forcing azure pipeline tests)

c04d1e9

kwigley force-pushed the updates/tox branch from ba03988 to c04d1e9 Compare March 29, 2021 13:19

kwigley merged commit ce30dfa into develop Mar 29, 2021

kwigley deleted the updates/tox branch March 29, 2021 13:47

This was referenced Mar 29, 2021

Update pytest-xdist requirement from <2,>=1.28.0 to >=1.28.0,<3 #3208

Closed

Bump pytest from 5.4.3 to 6.2.2 #3087

Closed

Update colorama requirement from <0.4.4,>=0.3.9 to >=0.3.9,<0.4.5 in /core #3081

Closed

JCZuurmond mentioned this pull request Apr 28, 2021

Use tox for all integration tests dbt-msft/dbt-sqlserver#131

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dev env clean up and improvements #3204

dev env clean up and improvements #3204

kwigley commented Mar 25, 2021 •

edited

Loading

gshank left a comment

nathaniel-may left a comment

nathaniel-may Mar 26, 2021

kwigley Mar 26, 2021

kwigley Mar 26, 2021

jtcohen6 left a comment

jtcohen6 Mar 26, 2021

kwigley Mar 29, 2021 •

edited

Loading

		@@ -128,100 +127,105 @@ Docker and docker-compose are both used in testing. Specific instructions for yo

		For testing, and later in the examples in this document, you may want to have `psql` available so you can poke around in the database and see what happened. We recommend that you use [homebrew](https://brew.sh/) for that on macOS, and your package manager on Linux. You can install any version of the postgres client that you'd like. On macOS, with homebrew setup, you can run:

dev env clean up and improvements #3204

dev env clean up and improvements #3204

Conversation

kwigley commented Mar 25, 2021 • edited Loading

Description

Future ideas

Checklist

gshank left a comment

Choose a reason for hiding this comment

nathaniel-may left a comment

Choose a reason for hiding this comment

nathaniel-may Mar 26, 2021

Choose a reason for hiding this comment

kwigley Mar 26, 2021

Choose a reason for hiding this comment

kwigley Mar 26, 2021

Choose a reason for hiding this comment

jtcohen6 left a comment

Choose a reason for hiding this comment

jtcohen6 Mar 26, 2021

Choose a reason for hiding this comment

kwigley Mar 29, 2021 • edited Loading

Choose a reason for hiding this comment

kwigley commented Mar 25, 2021 •

edited

Loading

kwigley Mar 29, 2021 •

edited

Loading