Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(python): add pyarrow to delta compatible schema conversion in writer/merge #1820

Merged
merged 23 commits into from
Nov 24, 2023
Merged

feat(python): add pyarrow to delta compatible schema conversion in writer/merge #1820

merged 23 commits into from
Nov 24, 2023

Conversation

ion-elgreco
Copy link
Collaborator

@ion-elgreco ion-elgreco commented Nov 7, 2023 •
edited
Loading

Description

This ports some functionality that @stinodego and I had worked on in Polars. Where we converted a pyarrow schema to a compatible delta schema. It converts the following:

  • uint -> int
  • timestamp(any timeunit) -> timestamp(us)

I adjusted the functionality to do schema conversion from large to normal when necessary, which is still needed in MERGE as workaround #1753.

Additional things I've added:

  • Schema conversion for every input in write_deltalake/merge
  • Add Pandas dataframe conversion
  • Add Pandas dataframe as input in merge

Related Issue(s)

@github-actions github-actions bot added the binding/python Issues for the Python package label Nov 7, 2023
@ion-elgreco ion-elgreco changed the title feat: add pyarrow to delta compatible schema conversion in writer and feat: add pyarrow to delta compatible schema conversion in writer/merge Nov 7, 2023
@ion-elgreco ion-elgreco changed the title feat: add pyarrow to delta compatible schema conversion in writer/merge feat(python): add pyarrow to delta compatible schema conversion in writer/merge Nov 7, 2023
@ion-elgreco ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from 3b9eebc to 2302148 Compare November 7, 2023 21:59
@ion-elgreco ion-elgreco marked this pull request as ready for review November 7, 2023 21:59
@ion-elgreco
Copy link
Collaborator Author

With 843c1d8 it's robust against schema's that have mixed Large and normal types. The schema will either be down casted to normal types or everything to large types, depending on user input.

@ion-elgreco ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from 1e7e9b1 to 3a1a120 Compare November 11, 2023 09:30
@ion-elgreco
Copy link
Collaborator Author

@wjones127 do you know how we can cast schemas onto a pyarrow dataset or recordbatchreader without materializing? replace_schema on Dataset doesn't work apparently.

Also I am not sure if it's even wrong to not materialize since we have the validate_batch to check the invariants at the end

@ion-elgreco ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from e370ecb to 8e076eb Compare November 11, 2023 23:15
@ion-elgreco ion-elgreco mentioned this pull request Nov 12, 2023
Merged
@ion-elgreco ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from ea25d41 to 850a05a Compare November 12, 2023 09:45
This was referenced Nov 15, 2023
Closed
Closed
wjones127
wjones127 previously requested changes Nov 17, 2023
View reviewed changes
Copy link
Collaborator

@wjones127 wjones127 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looking good. Just a few minor changes and it's ready to ship :)

python/deltalake/schema.py Outdated Show resolved Hide resolved
python/deltalake/schema.py Outdated Show resolved Hide resolved
python/tests/test_schema.py Show resolved Hide resolved
python/tests/test_writer.py Outdated Show resolved Hide resolved
python/tests/test_writer.py Outdated Show resolved Hide resolved
python/tests/test_writer.py Outdated Show resolved Hide resolved
@ion-elgreco ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from 0e29368 to 08cc31e Compare November 17, 2023 07:48
@ion-elgreco
Copy link
Collaborator Author

@wjones127 I've applied all the requested changes

roeap
roeap reviewed Nov 18, 2023
View reviewed changes
python/deltalake/writer.py Outdated
Copy link
Collaborator

@roeap roeap Nov 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this file live at the same level as the regular license and be included via the pyproject.toml?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's what I would propose:

  1. Move the license to python/licenses/polars_license.txt and reference that path in code comment.
  2. Create a file python/licenses/readme.md and include a list of which license apply to which code.
  3. Add MIT license to pyproject.toml

I think it's possible we will borrow other polars code in the futures, so nice to name the file polars_license.txt.

Does that sound good @ion-elgreco ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just realized I commented on a completely unrelated file - luckily the question seems to have been understood regardless 😆.

Personally, I like @wjones127's suggestion.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wjones127 can you take a look at point 3, I wasn't sure on how I could add another license file in the toml for it to work with maturin.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we ready to merge this? 😀

@ion-elgreco ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch 3 times, most recently from 83cbe65 to 3a31fc7 Compare November 20, 2023 06:42
@ion-elgreco ion-elgreco mentioned this pull request Nov 20, 2023
Merged
@ion-elgreco ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from 3a31fc7 to 06931fa Compare November 20, 2023 22:18
roeap
roeap approved these changes Nov 24, 2023
View reviewed changes
Copy link
Collaborator

@roeap roeap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

ion-elgreco and others added 23 commits November 24, 2023 20:18
@ion-elgreco
port schema conversion from polars and update writer and merge
424a92f
@ion-elgreco
change folder name
fcb8534
@ion-elgreco
only overwrite schema variable if schema is none
dee4864
@ion-elgreco
Also upcast from normal to large when large = True
d56a9d2
@ion-elgreco
use struct direct indexing for backwards pyarrow compat
4c0a549
@ion-elgreco
Fix and improve pyarrow conversions
863d3e4
@ion-elgreco
add many tests for pyarrow to delta schema conversions
e5fca75
@ion-elgreco
Add test for more write inputs with large types
c2ac4fa
@ion-elgreco
don't return schema and use generator
3cb085c
@ion-elgreco
move imports to the top
95a5ff2
@ion-elgreco
use generator typehint
687bd24
@ion-elgreco @wjones127
Update python/tests/test_writer.py
a11c432 
Co-authored-by: Will Jones <[email protected]>
@ion-elgreco @wjones127
Update python/tests/test_writer.py
d6a0c4b 
Co-authored-by: Will Jones <[email protected]>
@ion-elgreco @wjones127
Update python/deltalake/schema.py
33cb863 
Co-authored-by: Will Jones <[email protected]>
@ion-elgreco @wjones127
Update python/deltalake/schema.py
09b4a19 
Co-authored-by: Will Jones <[email protected]>
@ion-elgreco
add test to check if nullability is maintained
b351739
@ion-elgreco @wjones127
Update python/tests/test_writer.py
3deb15b 
Co-authored-by: Will Jones <[email protected]>
@ion-elgreco
use parametrized constructor to reduce tests
a86c8f3
@ion-elgreco
restructure licenses
cba6105
@ion-elgreco
Update schema.py
9468dae
@ion-elgreco
Update table.py
3478369
@ion-elgreco
Update schema.py
50c4255
@ion-elgreco
Update table.py
e2beb38
@ion-elgreco ion-elgreco force-pushed the feat/add_schema_conversion_helpers branch from 3356048 to e2beb38 Compare November 24, 2023 19:18
@ion-elgreco
Copy link
Collaborator Author

@roeap had to force push :) can you approve again?

@ion-elgreco ion-elgreco dismissed wjones127’s stale review November 24, 2023 20:23

Changes are resolved

@ion-elgreco ion-elgreco merged commit fa6c513 into delta-io:main Nov 24, 2023
24 checks passed
ion-elgreco added a commit to ion-elgreco/delta-rs that referenced this pull request Nov 25, 2023
@ion-elgreco @wjones127
feat(python): add pyarrow to delta compatible schema conversion in wr…
3ed7df0 
…iter/merge (delta-io#1820)

This ports some functionality that @stinodego and I had worked on in
Polars. Where we converted a pyarrow schema to a compatible delta
schema. It converts the following:

- uint -> int
- timestamp(any timeunit) -> timestamp(us)

I adjusted the functionality to do schema conversion from large to
normal when necessary, which is still needed in MERGE as workaround
delta-io#1753.

Additional things I've added:

- Schema conversion for every input in write_deltalake/merge
- Add Pandas dataframe conversion
- Add Pandas dataframe as input in merge

- closes delta-io#686
- closes delta-io#1467

---------

Co-authored-by: Will Jones <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@roeap roeap roeap approved these changes

@fvaleye fvaleye Awaiting requested review from fvaleye fvaleye is a code owner

@wjones127 wjones127 Awaiting requested review from wjones127 wjones127 is a code owner

Assignees
No one assigned
Labels
binding/python Issues for the Python package
Projects
None yet
Milestone
python v0.14
3 participants
@ion-elgreco @wjones127 @roeap