Issue 275: Support dbt package dependencies in Git subdirectories #3267

dmateusp · 2021-04-15T11:11:58Z

resolves #275

Description

Add an optional subdirectory key for Git packages.
Use sparse-checkout to pull only that directory.
Install dbt package from that subdirectory.
Make sure the git version of the user is >= 2.25.0, the version that introduces --sparse.

Checklist

I have signed the CLA (Or at least I remember signing it a while back)
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

dmateusp · 2021-04-15T11:19:25Z

test/rpc/test_deps.py

+ # TODO: Change me to something that fishtown-analytics manages!
+ # here I just moved the dbt_utils code into a subdirectory
+ {
+ 'git': 'https:/dmateusp/dbt-utils.git',


This needs to be changed before merging :)

dmateusp · 2021-04-15T13:41:59Z

Dockerfile.test

@@ -3,6 +3,9 @@ FROM ubuntu:18.04
 ENV DEBIAN_FRONTEND noninteractive

 RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+ software-properties-common \
+ && add-apt-repository ppa:git-core/ppa -y \


I had to update the git version in the test container, it was 2.17, needs to be >= 2.25

dmateusp · 2021-04-15T13:43:02Z

.circleci/config.yml

- - image: fishtownanalytics/test-container:11
+ # TODO: Ask for someone at fishtown to publish
+ # a new version of the test container
+ - image: dmateusp/dbt-test-container


Will need to push a new test-container version with the updated git version

dmateusp · 2021-04-15T13:43:07Z

.circleci/config.yml

- - image: fishtownanalytics/test-container:11
+ # TODO: Ask for someone at fishtown to publish
+ # a new version of the test container 
+ - image: dmateusp/dbt-test-container


Will need to push a new test-container version with the updated git version

jtcohen6

This is looking great @dmateusp! Once the TODOs are sorted, I think this will be good to go.

@kwigley Whenever you get a chance, could we publish a new fishtownanalytics/test-container with git>=2.25?

jtcohen6 · 2021-04-17T16:51:10Z

test/rpc/test_deps.py

+ # TODO: Change me to something that fishtown-analytics manages!
+ # here I just moved the dbt_utils code into a subdirectory
+ packages = [{
+ 'git': 'https:/dmateusp/dbt-utils.git',
+ 'revision': 'dmateusp/0.5.0/move_dbt_utils_to_subdir',
+ 'subdirectory': 'dbt_projects/dbt_utils'
+ }]


Nice! Ok, here are some options:

Use a long-lived subdirectory of dbt-utils:

packages = [{ 'git': 'https:/dmateusp/dbt-utils.git', 'revision': '0.5.0', 'subdirectory': 'integration_tests' }]

But because integration_tests has a local dependency on ../, we'd need to add a dummy dbt_project.yml file to the directory right above this one. (Those relative paths become relative to the root project location.)

Use a less-well established "package" that has a lot of project subdirectories (and which would really benefit from the addition of this feature):

packages = [{ 'git': 'https:/fishtown-analytics/dbt-labs-experimental-features', 'subdirectory': 'materialized-views' }]

We could add a branch to dbt-utils, similar to move_dbt_utils_to_subdir... and hope it's never accidentally deleted.

Sorry I didn't get if these options were for me or @kwigley ^^, I'd use dbt-labs-experimental-features on a specific commit sha, then we wouldn't worry about anything breaking the tests :)

dmateusp · 2021-04-19T13:43:16Z

test/rpc/test_deps.py

@@ -72,42 +72,80 @@ def deps_with_packages(packages, bad_packages, project_dir, profiles_dir, schema
 querier.is_result(querier.async_wait(tok1))


+@pytest.mark.parametrize(


@jtcohen6 I parametrized the test instead of the copy paste fyi

(I did this when I merged the changes from the PR that adds support for commits here)

dmateusp · 2021-04-22T08:12:48Z

@kwigley let me know if I can help with the TODOs, I'm on the dbt Slack with the same handle

kwigley · 2021-04-27T21:09:31Z

@kwigley let me know if I can help with the TODOs, I'm on the dbt Slack with the same handle

thanks for bearing with me! I just pushed a new image tagged fishtownanalytics/test-container:12 using the Dockerfile from this branch, you should be able to update CI with that new image 👍

kwigley · 2021-04-27T21:44:51Z

test/rpc/test_deps.py

+ # TODO: Change me to something that fishtown-analytics manages!
+ # here I just moved the dbt_utils code into a subdirectory
+ [{
+ 'git': 'https:/dmateusp/dbt-utils.git',
+ 'revision': 'dmateusp/0.5.0/move_dbt_utils_to_subdir',
+ 'subdirectory': 'dbt_projects/dbt_utils',
+ }],
+ [{
+ 'git': 'https:/fishtown-analytics/dbt-utils.git',
+ 'revision': '0.5.0',
+ 'subdirectory': 'path/to/nonexistent/dir',
+ }],
+ ),


Suggested change

# TODO: Change me to something that fishtown-analytics manages!

# here I just moved the dbt_utils code into a subdirectory

[{

'git': 'https:/dmateusp/dbt-utils.git',

'revision': 'dmateusp/0.5.0/move_dbt_utils_to_subdir',

'subdirectory': 'dbt_projects/dbt_utils',

}],

[{

'git': 'https:/fishtown-analytics/dbt-utils.git',

'revision': '0.5.0',

'subdirectory': 'path/to/nonexistent/dir',

}],

),

[{

'git': 'https:/fishtown-analytics/dbt-labs-experimental-features.git',

'revision': '0.0.1',

'subdirectory': 'materialized-views',

}],

[{

'git': 'https:/fishtown-analytics/dbt-labs-experimental-features.git',

'revision': '0.0.1',

'subdirectory': 'path/to/nonexistent/dir',

}],

),

this should do the trick!

Done! thanks!

…rsion of git)

jtcohen6

@dmateusp This is awesome! Thank you so much for the contribution, and for the wonderful commit messages along the way :)

@kwigley

) * 🔨 Extend git package contract and signatures to pass `subdirectory` * Add sparse checkout logic * ✅ Add test * 🧹 Lint * ✏️ Update CHANGELOG * 🐛 Make os.path.join safe * Use a test-container with an updated `git` version * 🔨 Fix integration tests * 📖 Update CHANGELOG contributors to include this PR * 🧪 Parameterize the test * Use new test-container published by @kwigley (contains more recent version of git) * Use repositories managed by fishtown * 🧘‍♂️ Merge the CHANGELOG * 🤦‍♂️ Remove repetition of my contribution on the CHANGELOG Co-authored-by: Jeremy Cohen <[email protected]> automatic commit by git-black, original commits: 5fb36e3

cla-bot bot added the cla:yes label Apr 15, 2021

dmateusp marked this pull request as ready for review April 15, 2021 11:14

dmateusp commented Apr 15, 2021

View reviewed changes

dmateusp changed the title ~~Support dbt package dependencies in Git subdirectories~~ Issue 275: Support dbt package dependencies in Git subdirectories Apr 15, 2021

dmateusp commented Apr 15, 2021

View reviewed changes

This was referenced Apr 15, 2021

Support commit hashes in dbt deps coming from a Git source #3268

Closed

support dbt packages that are not located at the root directory of a repo #275

Closed

jtcohen6 reviewed Apr 17, 2021

View reviewed changes

dmateusp force-pushed the dmateusp/275/support_github_subdirs branch from 76b9bf4 to dc3524f Compare April 19, 2021 13:40

dmateusp commented Apr 19, 2021

View reviewed changes

kwigley reviewed Apr 27, 2021

View reviewed changes

Daniel Mateus Pires added 13 commits April 28, 2021 10:30

🔨 Extend git package contract and signatures to pass subdirectory

4cf9674

Add sparse checkout logic

032d2b2

✅ Add test

82527fe

🧹 Lint

db1f5e9

✏️ Update CHANGELOG

b808648

🐛 Make os.path.join safe

1987374

Use a test-container with an updated git version

1767b53

🔨 Fix integration tests

7be1a48

📖 Update CHANGELOG contributors to include this PR

db40aaf

🧪 Parameterize the test

f50c5f1

Use new test-container published by @kwigley (contains more recent ve…

87250bb

…rsion of git)

Use repositories managed by fishtown

a183888

🧘‍♂️ Merge the CHANGELOG

e1234e0

dmateusp force-pushed the dmateusp/275/support_github_subdirs branch from 5186407 to e1234e0 Compare April 28, 2021 09:32

🤦‍♂️ Remove repetition of my contribution on the CHANGELOG

405d524

dmateusp requested review from kwigley and jtcohen6 April 28, 2021 10:17

Merge branch 'develop' into dmateusp/275/support_github_subdirs

26c3b95

jtcohen6 approved these changes Apr 28, 2021

View reviewed changes

jtcohen6 merged commit 5fb36e3 into dbt-labs:develop Apr 28, 2021

kwigley mentioned this pull request May 3, 2021

Missing package named packaging #3312

Closed

dmateusp mentioned this pull request May 11, 2021

Need a more recent version of git, and ssh-client installed in the dbt Docker image #3337

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue 275: Support dbt package dependencies in Git subdirectories #3267

Issue 275: Support dbt package dependencies in Git subdirectories #3267

dmateusp commented Apr 15, 2021 •

edited

Loading

dmateusp Apr 15, 2021

dmateusp Apr 15, 2021

dmateusp Apr 15, 2021

dmateusp Apr 15, 2021

jtcohen6 left a comment

jtcohen6 Apr 17, 2021

dmateusp Apr 22, 2021

dmateusp Apr 19, 2021

dmateusp commented Apr 22, 2021

kwigley commented Apr 27, 2021

kwigley Apr 27, 2021

dmateusp Apr 28, 2021

jtcohen6 left a comment •

edited

Loading

		@@ -72,42 +72,80 @@ def deps_with_packages(packages, bad_packages, project_dir, profiles_dir, schema
		querier.is_result(querier.async_wait(tok1))


		@pytest.mark.parametrize(

Issue 275: Support dbt package dependencies in Git subdirectories #3267

Issue 275: Support dbt package dependencies in Git subdirectories #3267

Conversation

dmateusp commented Apr 15, 2021 • edited Loading

Description

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtcohen6 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmateusp commented Apr 22, 2021

kwigley commented Apr 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtcohen6 left a comment • edited Loading

Choose a reason for hiding this comment

dmateusp commented Apr 15, 2021 •

edited

Loading

jtcohen6 left a comment •

edited

Loading