Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parenthesis removal in VDS creation #167

Open
Alain-Barrette opened this issue Mar 23, 2023 · 9 comments
Open

Parenthesis removal in VDS creation #167

Alain-Barrette opened this issue Mar 23, 2023 · 9 comments
Labels

Comments

@Alain-Barrette
Copy link

Describe the enhancement requested

We put dbt in place for about 1,100 objects over the last month. We noticed some behavior change in the query that are executed by our users.

We traced that to the presence of encapsulating parenthesis in the object deployed thru dbt.

Going from select column from table to ( select column from table ) is causing issue when the top vds are querying.
The top vds used to do

select column from tableA, tableB join on PK are now doing

select column from tableA, ( select column from tableB ) where PK join

This cause massive performance issue.

Is there any way we can remove theses parenthesis ?
image

Justification for this enhancement

Query performance improvement

@ArgusLi
Copy link
Contributor

ArgusLi commented Mar 23, 2023

Hi Alain, that is very strange behaviour. We will look into this as soon as we can. Thanks for raising this issue!

@ArgusLi
Copy link
Contributor

ArgusLi commented Mar 23, 2023

Could you please:

  1. Let us know what version of Dremio is being used
  2. Let us know what version of the adapter is being used
  3. Provide the model (or a snippet of the model) that leads to this behaviour?

Also, could you clarify what you mean by "we noticed some behaviour change in the query that are executed by our users"? Do you mean that this was not happening before and it is now happening?

@Alain-Barrette
Copy link
Author

  1. 23.1.0-202211250136090978-a79618c7

  2. fix for issue 134, base release was 1.3.2
    dbt-core>=1.4.1
    dbt-dremio>=1.4.1

  3. prepping model for upload

@Alain-Barrette
Copy link
Author

dbt-dags-issues-167.zip

@Alain-Barrette
Copy link
Author

Any update on this ?

@jlarue26
Copy link
Contributor

jlarue26 commented Apr 3, 2023

@Alain-Barrette we just finished getting dbt-dremio 1.4.5 out the door. Can you please see if you can reproduce this after upgrading?

However, dbt-dremio 1.4.5 won't automatically upgrade dbt-core. So please run pip install dbt-core==1.4.5 before pip install dbt-dremio

We believe that this regression is in dbt-core itself, but still looking into it.

@Alain-Barrette
Copy link
Author

reproduced after upgrade to 1.4.5

@ravjotbrar
Copy link
Contributor

ravjotbrar commented Apr 4, 2023

@Alain-Barrette I made a small change in my test branch, which removes parentheses from views. Do you mind testing if this resolves your problem by cloning this repo, switching to test-branch and then installing your local repo (similar to how your team did for #134)?

@Alain-Barrette
Copy link
Author

Fix from test branch is working. We are deploying to our dev team member until the fix made it to an official release.
Thanks a lot everyone.

ravjotbrar added a commit that referenced this issue Jul 14, 2023
### Summary

Remove parentheses from view materialization to optimize VDSs in Dremio.

### Description

Before:
```
(
   select * from table
)
```
After:
```
select * from table
```

### Test Results

All functional tests pass. 

### Changelog

-   [ ] Added a summary of what this PR accomplishes to CHANGELOG.md

### Related Issue
#167
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

4 participants