[CT-2829] [Bug] dbt core doesn't always provide JSON logs in jobs when requested by Cloud

[sophiad96]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

An Enterprise customer had a job erroring out due to a missing quotation at the end of a vars statement:

dbt run -m stg_action_defs_from_couchbase --vars ‘{"ctm_odate": "2023-07-11", "ctm_jobname": "JBCPER105221-CMBSJ", "v_flow_nm":"'HEQ_ELIG_AGG'"}

Instead of returning a clear error message to indicate that the missing single quotation at the end is the problem, the job errored out and dbt didn’t display any obvious error logs.

Expected Behavior

The error message found in Datadog should be displayed in the job logs in the UI.

Steps To Reproduce

1. Create a job and include a run command such as dbt run --vars '{"key": "value"}
2. Run the job.

Relevant log output

No response

Environment

- OS:
- Python:
- dbt: 1.4

Which database adapter are you using with dbt?

No response

Additional Context

https://dbtlabs.atlassian.net/browse/TRIAGE-549

[dbeatty10]

Thank you for reporting this @sophiad96 !

Reprex

It is pretty easy to reproduce this on dbt-core 1.4:

dbt ls --vars ‘{"foo":"'bar'"}

❌ dbt-core 1.4 gives a long stack trace like this:

Traceback (most recent call last):
  File "/my_path/bin/dbt", line 8, in <module>
    sys.exit(cli())
  File "/my_path/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/my_path/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/my_path/lib/python3.9/site-packages/click/core.py", line 1655, in invoke
    sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
  File "/my_path/lib/python3.9/site-packages/click/core.py", line 920, in make_context
    self.parse_args(ctx, args)
  File "/my_path/lib/python3.9/site-packages/click/core.py", line 1378, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/my_path/lib/python3.9/site-packages/click/core.py", line 2360, in handle_parse_result
    value = self.process_value(ctx, value)
  File "/my_path/lib/python3.9/site-packages/click/core.py", line 2316, in process_value
    value = self.type_cast_value(ctx, value)
  File "/my_path/lib/python3.9/site-packages/click/core.py", line 2304, in type_cast_value
    return convert(value)
  File "/my_path/lib/python3.9/site-packages/click/types.py", line 82, in __call__
    return self.convert(value, param, ctx)
  File "/my_path/lib/python3.9/site-packages/dbt/cli/option_types.py", line 19, in convert
    return parse_cli_vars(value)
  File "/my_path/lib/python3.9/site-packages/dbt/config/utils.py", line 11, in parse_cli_vars
    return parse_cli_yaml_string(var_string, "vars")
  File "/my_path/lib/python3.9/site-packages/dbt/config/utils.py", line 21, in parse_cli_yaml_string
    raise OptionNotYamlDictError(var_type, cli_option_name)
dbt.exceptions.OptionNotYamlDictError: Compilation Error
  The --vars argument must be a YAML dictionary, but was of type 'str'

✅ Using dbt-core >=1.5.2, there is no stack trace (which I think means this problem is avoided):

$ dbt ls --vars ‘{"foo":"'bar'"}

22:40:58  The YAML provided in the --vars argument is not valid.
Usage: dbt ls [OPTIONS]
Try 'dbt ls -h' for help.

Error: Invalid value for '--vars': String '‘{foo:'bar'}' is not valid YAML

Next steps

@jtcohen6 #7735 solves this and was already backported to 1.5 in #7878 -- do you want to backport it any further back?

[jtcohen6]

Thanks @dbeatty10! Important context here is that these logs are being produced with --log-format json, so I'd guess the operative error here was while trying to serialize the stacktrace to JSON.

I think this will be trickier to backport to 1.4.latest, given the changes we made to the CLI / parameters between v1.4 and v1.5, though not impossible.

[jtcohen6]

Specifically, the issue here is that CLI validation (both argparse for v1.4 + click for v1.5+) returns text to stdout, not JSON. So trying to parse from stdout as JSON would lead to an error.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.