Fix dbt deps issues (#778 #994 #895)

[beckjake]

Fix some issues around dbt deps.

I think this fixes #778, fixes #994, fixes #895 - some of those are sort of judgement calls

Behavior after this PR

• When git is not available, dbt is more helpful now

> dbt deps
Encountered an error:
Could not find command, ensure it is in the user's PATH: "git"

• When a repository is incorrect, shows the command that failed (it'd be nice to remove the username/password prompt!):

> dbt deps
Username for 'https:': beckjake
Password for 'https://[email protected]':
Encountered an error:
Got a non-zero returncode running: ['git', 'clone', '--depth', '1', 'https:/fishtown-analytics/dbt-utilz.git', '23da7ac89d7d5c4086737ad8dc02feff']

• When a revision is invalid, shows the ref that could not be found and the package URL:

> dbt deps
Encountered an error:
Error checking out spec='0.9.17' for repo https:/fishtown-analytics/dbt-utils.git
fatal: Couldn't find remote ref 0.9.17

• Uses shutil.move instead of os.rename to move things out of the download directory, to handle the cross-drive issue

• By default, downloads go to a temp directory dbt-downloads-xxxxx where xxxxx is some random stuff, created by tempfile.mkdtemp that is removed via shutil.rmtree after a successful run. This should resolve the concurrent users issue. Users can override this behavior to download to a fixed directory using DBT_DOWNLOADS_DIR, in which case dbt will not shutil.rmtree the directory.

• dbt no longer ignores return codes and instead raises exceptions that have stderr/stdout

• More explicitly log what failed and where on permissions-related execution failures, this should be especially more helpful on windows

[cmcarthur]

is this built by Dockerfile in this repo?

[beckjake]

Yeah. I use this gross and lame shell script to do it:

#!/bin/bash
set -xeuo pipefail
commit_message="$1"
shift
if [[ -z ${commit_message} ]]; then
    echo 'no commit message'
    exit 1
fi
docker build --tag 'test-container:latest' .
container_id="$(docker run -it test-container:latest hostname | tr -d '[[:space:]]')"
docker commit -a 'Jacob Beck <[email protected]>' -m "${commit_message}" ${container_id} test-container
docker tag test-container:latest fishtownjacob/test-container:latest
docker push fishtownjacob/test-container:latest

[drewbanin]

This looks really great @beckjake!

Can you think of a good way to show a more obvious error message if git is not in the user's path? My guess is that if someone is just getting started with dbt and they don't have git installed on windows, this message might be a little confusing:

Could not find command, ensure it is in the user's PATH: "git"

Ideally, I think we'd show:

Could not find command, ensure it is in the user's PATH: "git"
Make sure git is installed on your machine. More information: https://docs.getdbt.com/docs/package-management

^ We can make a section about the requirements for dbt deps (including more info about how to install git) and link to that specific section.

I'm a little confused that you're seeing the Username/Password prompt here:

> dbt deps
Username for 'https:': beckjake
Password for 'https://[email protected]':
...

I couldn't reproduce that locally -- do you think it's a git version thing? Or something else?

I'll give this some spins on a Windows machine, but this is looking pretty good to me!

[beckjake]

for the missing git error, I can easily change it to this:

Make sure git is installed on your machine. More information: https://docs.getdbt.com/docs/package-management
Encountered an error:
Could not find command, ensure it is in the user's PATH: "git"

Which is what you suggested but with order reversed. Is that ok?

I think the password prompt is a global git settings thing, or maybe version-dependent. it happens to me a lot when cloning a private repository from github via https urls. I've always taken it as a sign that my ssh key is somehow wrong.

[atharvai]

Drew asked me to test this out on Windows, here are the results: https://gist.github.com/atharvai/a302f1777a1f5b0202dffca937cb8de3

Summary: If the DBT_DOWNLOADS_DIR or default temp dir are on same drive as dbt project, then dbt deps is successful. Otherwise it fails on Access Denied on os.remove

[drewbanin]

Thanks @atharvai! This is super helpful :)

[beckjake]

With that most recent push I believe it is now fixed. I tested on Windows with an old drive I found, and cross drive moves from both temp directories and explicit DBT_DOWNLOADS_DIR settings are working as they should.

[atharvai]

I tested the latest commit and can confirm all is good!
One suggestion is to log if DBT_DOWNLOADS_DIR is set or not and value if set. It would help debugging.

Alternatively, set this log line to print full path

2018-11-14 18:06:46,187 (MainThread): Executing "git clone --depth 1 [email protected]:fishtown-analytics/dbt-utils.git 0a6a5f508a311be0d4ba05de66b74124"
...
2018-11-14 18:06:47,742 (MainThread): STDERR: "Cloning into '0a6a5f508a311be0d4ba05de66b74124'...

[drewbanin]

@beckjake @atharvai awesome!! Really glad we got this working. Thanks for your insight @atharvai :)

This LGTM when the tests all pass!