Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(python): expose MERGE operation #1685

Merged
merged 42 commits into from
Oct 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
80d328b
save work to continue on other pc
ion-elgreco Sep 23, 2023
f62abab
Setup merge skeleton on python side
ion-elgreco Sep 23, 2023
fd07d5f
use arrow
ion-elgreco Sep 26, 2023
57932f4
Merge branch 'main' into feat/expose_merge_python
ion-elgreco Sep 26, 2023
4e4e432
save to continue home pc
ion-elgreco Sep 29, 2023
ee7860f
save
ion-elgreco Sep 29, 2023
774ff55
Merge branch 'main' into feat/expose_merge_python
ion-elgreco Sep 29, 2023
7176771
Allow metrics to be serialized
ion-elgreco Sep 30, 2023
ec7c2b2
make datafusion_utils mod public
ion-elgreco Sep 30, 2023
2ded189
Comment the deny missing docs for now : )
ion-elgreco Sep 30, 2023
ecef07b
Add merge_execute
ion-elgreco Sep 30, 2023
239af35
Add logic to create (col, expr) hashmap for updates/set
ion-elgreco Oct 1, 2023
4a927a9
Remove println
ion-elgreco Oct 1, 2023
bcb8d4a
Use Typing type hints
ion-elgreco Oct 2, 2023
1c866b1
Fix rust lints
ion-elgreco Oct 2, 2023
2522d3e
Add merge when_matched_delete test
ion-elgreco Oct 2, 2023
08497d3
Move property to merge
ion-elgreco Oct 2, 2023
611d93a
Add all test cases
ion-elgreco Oct 2, 2023
6a05320
Fix lint and type hint
ion-elgreco Oct 3, 2023
7770b4e
Add type hints
ion-elgreco Oct 3, 2023
2e3c388
Merge branch 'main' into feat/expose_merge_python
ion-elgreco Oct 3, 2023
28734be
add into
ion-elgreco Oct 3, 2023
1fd7909
simplify code
ion-elgreco Oct 3, 2023
3c4493c
Move fixture
ion-elgreco Oct 3, 2023
15af04e
format
ion-elgreco Oct 3, 2023
ec82c91
Merge branch 'main' into feat/expose_merge_python
ion-elgreco Oct 6, 2023
98a83d6
Use target_alias, fix tests
ion-elgreco Oct 7, 2023
de56cec
Add passing tests
ion-elgreco Oct 7, 2023
70dc65a
formatting
ion-elgreco Oct 7, 2023
b0de38e
Make consistent with update
ion-elgreco Oct 7, 2023
a042273
Merge branch 'main' into feat/expose_merge_python
ion-elgreco Oct 12, 2023
0d9b8e6
Merge branch 'main' into feat/expose_merge_python
ion-elgreco Oct 12, 2023
cc7fd15
add target_alias
ion-elgreco Oct 12, 2023
547bcaf
Add update_all and insert_all method
ion-elgreco Oct 12, 2023
5a2e66f
formatting + update docs
ion-elgreco Oct 12, 2023
d045ec1
Adjust test cases to include less or different columns compared to ta…
ion-elgreco Oct 14, 2023
06480ef
Add when_not_matched_by_source_delete_wo_predicate test
ion-elgreco Oct 14, 2023
0c1af87
use recordbatchrearder and fix bug
ion-elgreco Oct 14, 2023
e83d327
Merge branch 'main' into feat/expose_merge_python
ion-elgreco Oct 14, 2023
a35f67a
resolve lint errors
ion-elgreco Oct 14, 2023
c7709cb
clean up code
ion-elgreco Oct 15, 2023
ad41a7d
remove debug print
ion-elgreco Oct 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions python/deltalake/_internal.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,25 @@ class RawDeltaTable:
writer_properties: Optional[Dict[str, int]],
safe_cast: bool = False,
) -> str: ...
def merge_execute(
self,
source: pa.RecordBatchReader,
predicate: str,
source_alias: Optional[str],
target_alias: Optional[str],
writer_properties: Optional[Dict[str, int | None]],
safe_cast: bool,
matched_update_updates: Optional[Dict[str, str]],
matched_update_predicate: Optional[str],
matched_delete_predicate: Optional[str],
matched_delete_all: Optional[bool],
not_matched_insert_updates: Optional[Dict[str, str]],
not_matched_insert_predicate: Optional[str],
not_matched_by_source_update_updates: Optional[Dict[str, str]],
not_matched_by_source_update_predicate: Optional[str],
not_matched_by_source_delete_predicate: Optional[str],
not_matched_by_source_delete_all: Optional[bool],
) -> str: ...
def get_active_partitions(
self, partitions_filters: Optional[FilterType] = None
) -> Any: ...
Expand Down
Loading
Loading