Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-8203] Make record merge mode the primary merging config #11943

Open
wants to merge 105 commits into
base: master
Choose a base branch
from
Open

[HUDI-8203] Make record merge mode the primary merging config #11943

wants to merge 105 commits into from
+1,028 −880

Commits on Sep 13, 2024

  1. first attempt at making merge mode the primary config for merging/pay… 

    …load
    Jonathan Vexler committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    a010515 View commit details
    Browse the repository at this point in the history

  2. remove unused methods in meta client

    Jonathan Vexler committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    d3045da View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. Merge branch 'master' into create_new_merger_configs

    Jonathan Vexler committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    371fd96 View commit details
    Browse the repository at this point in the history

  2. get the quickstart to pass

    Jonathan Vexler committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    1a64f42 View commit details
    Browse the repository at this point in the history

  3. get rid of defaults for payload in hoodie option config

    Jonathan Vexler committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    373f67f View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. don't infer based on engine type now

    Jonathan Vexler committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    8745ea0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into create_new_merger_configs

    Jonathan Vexler committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    f77cb87 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. fix some things and fix the issue with MIT

    Jonathan Vexler committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    60755d4 View commit details
    Browse the repository at this point in the history

  2. fix checkstyle

    Jonathan Vexler committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    aac457a View commit details
    Browse the repository at this point in the history

  3. support avro read strategy

    Jonathan Vexler committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    63c9a00 View commit details
    Browse the repository at this point in the history

  4. checkstyle

    Jonathan Vexler committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    c82c27f View commit details
    Browse the repository at this point in the history

  5. allow payload during read

    Jonathan Vexler committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    5d466c3 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into create_new_merger_configs

    Jonathan Vexler committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    416835f View commit details
    Browse the repository at this point in the history

  7. fix merge issue

    Jonathan Vexler committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    9795fe8 View commit details
    Browse the repository at this point in the history

  8. validate that we don't try to do payload merging with event time or o… 

    …verwrite
    Jonathan Vexler committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    ab6d567 View commit details
    Browse the repository at this point in the history

  9. fix bugs and tests

    Jonathan Vexler committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    ae29647 View commit details
    Browse the repository at this point in the history

  10. fix failing tests

    Jonathan Vexler committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    12e8de2 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. fix test

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ce04bd5 View commit details
    Browse the repository at this point in the history

  2. fix testMergerStrategySet

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    01f2024 View commit details
    Browse the repository at this point in the history

  3. fix utf8 comparable issue in default record payload

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ffc94b8 View commit details
    Browse the repository at this point in the history

  4. fix getting payload in bootstrap

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    c789678 View commit details
    Browse the repository at this point in the history

  5. fix testCreateHoodieConfigWithAsyncClustering

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    88799a2 View commit details
    Browse the repository at this point in the history

  6. fix log format capitalization

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    638d350 View commit details
    Browse the repository at this point in the history

  7. deprecate payload config

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    3e486d2 View commit details
    Browse the repository at this point in the history

  8. fix checkstyle and make some more changes like not calling get defaul… 

    …t for the payload. And refactor create avro record to move it to the base buffer instead of the reader context
    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    1db1fec View commit details
    Browse the repository at this point in the history

  9. infer merge mode if they use default or overwrite payload

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    8d0d358 View commit details
    Browse the repository at this point in the history

  10. fix TestTableSchemaEvolution.testMORTable

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    4d04980 View commit details
    Browse the repository at this point in the history

  11. fix bad parens

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    3ee84d8 View commit details
    Browse the repository at this point in the history

  12. fix log block casing issue

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    da849df View commit details
    Browse the repository at this point in the history

  13. fix checkstyle

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    3480af2 View commit details
    Browse the repository at this point in the history

  14. fix more places where the table is not updated

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    1fce20c View commit details
    Browse the repository at this point in the history

  15. fix checkstyle

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    2371e19 View commit details
    Browse the repository at this point in the history

  16. need to set strategy when payload is set

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    8af0e40 View commit details
    Browse the repository at this point in the history

  17. fix some errors I made

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    8dfedc0 View commit details
    Browse the repository at this point in the history

  18. fix instantiate server client

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    5ace882 View commit details
    Browse the repository at this point in the history

  19. problem seems to be with the old read path, not with fg reader

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    080f024 View commit details
    Browse the repository at this point in the history

  20. fix checkstyle

    Jonathan Vexler committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    f10ab50 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. Merge branch 'master' into create_new_merger_configs

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    5c9783b View commit details
    Browse the repository at this point in the history

  2. TestHoodieMergeHandleWithSparkMerger fix because delete records were … 

    …using the wrong data type for ordering val
    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    5f59bd2 View commit details
    Browse the repository at this point in the history

  3. fix configs for more tests

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    f9aee6e View commit details
    Browse the repository at this point in the history

  4. fix bootstrap tests

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    aeea89b View commit details
    Browse the repository at this point in the history

  5. bootstrap executor remove payload defaulting

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    8a737c0 View commit details
    Browse the repository at this point in the history

  6. fix checkstyle

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    6e08ef5 View commit details
    Browse the repository at this point in the history

  7. fix some more test cases

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    6e43f93 View commit details
    Browse the repository at this point in the history

  8. disable test pruned filtered

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    2ab97ce View commit details
    Browse the repository at this point in the history

  9. fix deltastreamer infer merge modes and some other tests

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    eaaf998 View commit details
    Browse the repository at this point in the history

  10. remove comment in tableconfig

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    e6ee76a View commit details
    Browse the repository at this point in the history

  11. overwrite the tableconfig payload as well for mit

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    100e686 View commit details
    Browse the repository at this point in the history

  12. get rid of stupid get record type method

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    3b89dcb View commit details
    Browse the repository at this point in the history

  13. fix Test Call repair_overwrite_hoodie_props Procedure

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    22ebef6 View commit details
    Browse the repository at this point in the history

  14. fix more test issues

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    6c2abc9 View commit details
    Browse the repository at this point in the history

  15. fix checkstyle

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    f6a0da2 View commit details
    Browse the repository at this point in the history

  16. fix failing test

    Jonathan Vexler committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    80a2e16 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. fix pruned filtered test

    Jonathan Vexler committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    f61553c View commit details
    Browse the repository at this point in the history

  2. add log for bootstrap IT

    Jonathan Vexler committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    b07d02b View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. fix minargs for the cli command

    Jonathan Vexler committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    597677f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into create_new_merger_configs

    Jonathan Vexler committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    7aac16b View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. use get value for record merge mode

    Jonathan Vexler committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    080de60 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into create_new_merger_configs

    Jonathan Vexler committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    6fc3da5 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Merge branch 'apache:master' into create_new_merger_configs

    @jonvex
    jonvex authored Oct 4, 2024
    Configuration menu
    Copy the full SHA
    1751520 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Merge branch 'apache:master' into create_new_merger_configs

    @jonvex
    jonvex authored Oct 7, 2024
    Configuration menu
    Copy the full SHA
    da99393 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. add spark write support for avro log block

    Jonathan Vexler committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    232d0d0 View commit details
    Browse the repository at this point in the history

  2. add merger impls to bootstrap and deltastreamer

    Jonathan Vexler committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    9630cbe View commit details
    Browse the repository at this point in the history

  3. add string casting to allow for case where ordering field is missing

    Jonathan Vexler committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    a345c4c View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2024

  1. Merge branch 'master' into create_new_merger_configs

    Jonathan Vexler committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    4af7de5 View commit details
    Browse the repository at this point in the history

  2. merge mode in the fg reader should be consistent

    Jonathan Vexler committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    36bd377 View commit details
    Browse the repository at this point in the history

  3. set merger configs for bootstrap

    Jonathan Vexler committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    0e0775c View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. fix avro payload merger strategy var name

    Jonathan Vexler committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    6d581c5 View commit details
    Browse the repository at this point in the history

  2. add comments to the merger strategies

    Jonathan Vexler committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    7723029 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Add lokesh changes oct 16th 

    Co-authored-by: Lokesh Jain <[email protected]>
    @jonvex @lokeshj1703
    jonvex and lokeshj1703 authored Oct 16, 2024
    Configuration menu
    Copy the full SHA
    90db422 View commit details
    Browse the repository at this point in the history

  2. fix checkstyle

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    3ff9e74 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into create_new_merger_configs

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    699d455 View commit details
    Browse the repository at this point in the history

  4. get rid of avro block write

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    70852c5 View commit details
    Browse the repository at this point in the history

  5. get rid of javascalautils

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    c4dc95f View commit details
    Browse the repository at this point in the history

  6. throw real error message

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    3726f54 View commit details
    Browse the repository at this point in the history

  7. fix build errors

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    8306c68 View commit details
    Browse the repository at this point in the history

  8. always set values for payload and strategy

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    95f8969 View commit details
    Browse the repository at this point in the history

  9. fix up some more things

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    6e96999 View commit details
    Browse the repository at this point in the history

  10. fix build errors that didn't appear locally

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    b1bd0ca View commit details
    Browse the repository at this point in the history

  11. remove log block changes

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    ab40874 View commit details
    Browse the repository at this point in the history

  12. fix checkstyle

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    ac45359 View commit details
    Browse the repository at this point in the history

  13. remove java scala option converters

    Jonathan Vexler committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    cef2687 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. handle some sql config management

    Jonathan Vexler committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    9f3adb4 View commit details
    Browse the repository at this point in the history

  2. fix custom merger case

    Jonathan Vexler committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    709a37a View commit details
    Browse the repository at this point in the history

  3. fix more tests

    Jonathan Vexler committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    143dfa7 View commit details
    Browse the repository at this point in the history

  4. fix style

    Jonathan Vexler committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    bdab7de View commit details
    Browse the repository at this point in the history

  5. use config instead of get string

    Jonathan Vexler committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    b0506a2 View commit details
    Browse the repository at this point in the history

  6. spark sql config issue fix + fix some other tests

    Jonathan Vexler committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    8fab0f0 View commit details
    Browse the repository at this point in the history

  7. fix npe issue when creating writer

    Jonathan Vexler committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    57ad75e View commit details
    Browse the repository at this point in the history

  8. fix some more tests

    Jonathan Vexler committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    25897cb View commit details
    Browse the repository at this point in the history

  9. fix testMergeOnReadSnapshotRelationWithDeltaLogsFallback()(TestParque… 

    …tColumnProjection) and Test Call repair_overwrite_hoodie_props Procedure
    Jonathan Vexler committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    9119372 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. fix failing tests

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    addd352 View commit details
    Browse the repository at this point in the history

  2. fix azure tests

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    4cdbd16 View commit details
    Browse the repository at this point in the history

  3. add more validation to infer merge mode

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    f619f0b View commit details
    Browse the repository at this point in the history

  4. address review comments and fix tests

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    f55704f View commit details
    Browse the repository at this point in the history

  5. fix checkstyle

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    ad0682a View commit details
    Browse the repository at this point in the history

  6. remove duplicate test case

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    4d11955 View commit details
    Browse the repository at this point in the history

  7. fix style

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    b851357 View commit details
    Browse the repository at this point in the history

  8. throw exception if no valid merger impl is present for custom merger

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    4303223 View commit details
    Browse the repository at this point in the history

  9. address review comments and fix tests

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    9a56f78 View commit details
    Browse the repository at this point in the history

  10. address more review comments

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    08b73c0 View commit details
    Browse the repository at this point in the history

  11. revert config name change

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    1bb38d2 View commit details
    Browse the repository at this point in the history

  12. fix failing tests

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    413ebc4 View commit details
    Browse the repository at this point in the history

  13. change config name to have id in it, but not table config

    Jonathan Vexler committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    75beb3d View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2024

  1. Merge branch 'master' into create_new_merger_configs

    Jonathan Vexler committed Oct 19, 2024
    Configuration menu
    Copy the full SHA
    3c8df12 View commit details
    Browse the repository at this point in the history