Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(python, rust): cdc in writer not creating inserts #2751

Merged
merged 2 commits into from
Aug 12, 2024
Merged

fix(python, rust): cdc in writer not creating inserts #2751

merged 2 commits into from
Aug 12, 2024

Conversation

ion-elgreco
Copy link
Collaborator

Description

Improved logic

  • If it's partition scan, we don't write CDC actions and data. CDF reader will use remove actions in this case now to know what's deleted or not
  • When the predicate doesn't hit a partition, we also write CDC data for the inserts

Related Issue(s)

@github-actions github-actions bot added binding/python Issues for the Python package binding/rust Issues for the Rust crate labels Aug 11, 2024
@github-actions GitHub Actions
Copy link

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

@ion-elgreco ion-elgreco changed the title fix(python, rust): CDC in writer not creating inserts fix(python, rust): cdc in writer not creating inserts Aug 11, 2024
@ion-elgreco ion-elgreco force-pushed the fix/writer_insert_changes branch 2 times, most recently from 90649d2 to c77a704 Compare August 11, 2024 15:34
@rtyler rtyler enabled auto-merge August 12, 2024 15:37
rtyler
rtyler previously approved these changes Aug 12, 2024
View reviewed changes
@rtyler rtyler added this pull request to the merge queue Aug 12, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch Aug 12, 2024
@ion-elgreco ion-elgreco added this pull request to the merge queue Aug 12, 2024
Merged via the queue into delta-io:main with commit 85ffddf Aug 12, 2024
21 checks passed
@ion-elgreco ion-elgreco deleted the fix/writer_insert_changes branch August 12, 2024 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hntd187 hntd187 hntd187 left review comments

@rtyler rtyler rtyler approved these changes

@wjones127 wjones127 Awaiting requested review from wjones127 wjones127 is a code owner

@fvaleye fvaleye Awaiting requested review from fvaleye fvaleye is a code owner

@roeap roeap Awaiting requested review from roeap roeap is a code owner

Assignees
No one assigned
Labels
binding/python Issues for the Python package binding/rust Issues for the Rust crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Write also insert change types in writer CDC
3 participants
@ion-elgreco @rtyler @hntd187