support dbt packages that are not located at the root directory of a repo

[drewbanin]

via dbt slack:

question on dependency management. we initially have 2 services that we want to ‘own’ the initial model (transform) in our DW DB. specifically, we want them to run automated tests (in circleci) so that developers are prevented from making breaking changes to the source OLTP schema.

[2:54]
i believe ergonomically that means our service-a and service-b github repos have a ./dbt subfolder with some basic models (edited)

[2:55]
which the service can run/test as part of the normal release for the service (edited)

[2:55]
and then a shared bizint-models repo that consumes those base models for more complex analyses

[2:56]
however, it doesn’t look like the existing dependency management in dbt supports pulling from a small subdirectory in a (fairly large) service repo

[jthandy]

@cmcarthur take a look--this might be relevant to the stuff you're working on right now.

[cmcarthur]

thanks @jthandy . there will indeed be an option to point to a local directory in a near future release

[drewbanin]

this is doable today with local packages - closing this.

[dmateusp]

I understand the specific use case mentioned in the issue is handled by the local installation option.

Would you consider still adding the feature of supporting sub directories? In pip you can specify &subdirectory so we can just run one pip install rather than doing a separate git clone, I'd like to be able to do the same in dbt.

[jtcohen6]

Would you consider still adding the feature of supporting sub directories?

@dmateusp I'm open to this! What do you have in mind?

Given that dbt deps is totally dependent on git for packages hosted on git providers or hub.getdbt.com, we'd need some git-friendly way to do this. I see that sparse checkout now exists as an experimental feature...

[dmateusp]

@jtcohen6 I think the sparse checkout could be a good solution! I looked into how pip supports it, and it seems that they clone the whole repo https:/pypa/pip/blob/0dd3a2b84997e6d0cebcb7cf3e090e03bc2eedd9/src/pip/_internal/vcs/git.py#L358

[dmateusp]

Should be ready for review: #3267