fix(python, rust): allow in pushdowns in early_filter

[ion-elgreco]

Description

is_in filters can be used now to do partition pruning, I also moved all the filter logic into it's own module.

Related issues

• closes Large Memory Spike on Merge #2802
• closes IN (...) clauses appear to be ignored in merge commands with S3 - extra partitions scanned #2726

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[rtyler]

@ion-elgreco I pulled this into my CI infrastructure, and also have some bigger tables at $WORKPLACE that @roeap and I were doing some performance analysis with. I will take a look and report back with a review.

[JonasDev1]

In the mod.rs are also tests related to the filters, maybe you can move them also to the filter.rs

[ion-elgreco]

In the mod.rs are also tests related to the filters, maybe you can move them also to the filter.rs

Ah yes, missed those :)

[JonasDev1]

A test with a condition like target.col in (source.col1, source.col2) would be interesting

[ion-elgreco]

A test with a condition like target.col in (source.col1, source.col2) would be interesting

Hmm I will add some more tests, by the way I wonder if we want placeholder exprs if a user already explicitly passed a predicate for that partition column.

If I do s.partition and t.partition and t.partition in (3,4), I will get multiple placeholders saying t.partition=3 or t.partition=4 and t.partition in (3,4), result will be the same but I wonder if this would make it less efficient in terms of query runtime

[ion-elgreco]

I've put it in draft until I add some more tests in Rust

[JonasDev1]

A test with a condition like target.col in (source.col1, source.col2) would be interesting

Hmm I will add some more tests, by the way I wonder if we want placeholder exprs if a user already explicitly passed a predicate for that partition column.

If I do s.partition and t.partition and t.partition in (3,4), I will get multiple placeholders saying t.partition=3 or t.partition=4 and t.partition in (3,4), result will be the same but I wonder if this would make it less efficient in terms of query runtime

The placeholder expr could be more restrictive than the explicit expr. I think performance should not be affected

[omkar-foss]

A test with a condition like target.col in (source.col1, source.col2) would be interesting

Hi! I've added a Rust test that covers above mentiond condition here in this PR.

Also added a few more Rust unit tests for try_construct_early_filter().

[rtyler]

I have done some manual testing with this pull request and I can say that at least it did not adversely impact performance 😄 . I will rebase this branch, merge it, and then we can go from there