tasks that define their own arguments and types

[beckjake]

Describe the feature

dbt should define task-specific argument parsing behavior in the tasks that use those arguments, instead of main.py. The output of argument parsing should be a typed object so tasks can be sure of what to rely on.

Currently, dbt-core defines tasks in the task/ folder and ties argument parsing to tasks in main.py. The argument parsing code in main.py all has explicit references to tasks and task names (and their RPC names, if they exist). It's difficult to know for sure out what args are safe to access in a given task or what they are. We are fortunate that argparse results in generally-reasonable and intuitive behavior, and that we've mostly kept the arguments available to various commands sane. But wouldn't it be nice if flake8/mypy just did that for us?

So I propose a couple related changes:

• tie some sort of argparse-setting method to classes themselves
• make the args object that ends up on configs be a typed dataclass
• have main.py look at the tasks and use those to generate the argument parser

I think a cool and kind of fun thing that's probably not too hard would be to write "hologram for argparse", but that's not a requirement for this by any means.

Motivation

There are two big motivations here:
One is that it would be nice for RPC and CLI tasks to converge more. RPC tasks already define their arguments as typed objects. It doesn't make much sense to push those back into an argparse.Namespace, but going the other way does make a lot of sense because type systems are a nice way to find bugs before the people running your program do.

The second is that in the future, it would be great to provide some sort of pluggable tasks. One way of doing that is to make tasks a PEP 420 namespace package, like adapters and include already are, and then make tasks discoverable like adapters. That would be very nice for development. We could write some sort of dbt-dev-tools plugin that supplies useful tasks that are useful for developing dbt ("connected adapter in a repl", "render me this jinja file in this context", "jinja repl"). I can also imagine prototyping a dbt run+test that way, as an external package.

Describe alternatives you've considered

I could set up complicated and brittle local framework with git tag and git cherry-pick and a local-only branch that has debugging tasks. I kind of have that now, but I don't use it much because maintaining it when main.py changes is not really fun and getting it set up/torn down is a pain every time I use it.

Who will this benefit?

This is pretty developer-centric!

[drewbanin]

I think there are a lot of people out there that would like the ability to register their own dbt tasks too :D

I'm into it!

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[jtcohen6]

I'd still like to do this, or something like it. When we get there, we'll open a new issue :)