Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet WAL #1878

Merged
merged 39 commits into from
Nov 28, 2022
Merged

Parquet WAL #1878

merged 39 commits into from
Nov 28, 2022

Commits on Oct 31, 2022

  1. vParquet WALBlock 

    Signed-off-by: Joe Elliott <[email protected]>
    @joe-elliott
    joe-elliott committed Oct 31, 2022
    Configuration menu
    Copy the full SHA
    1bc0a59 View commit details
    Browse the repository at this point in the history

  2. tests 

    Signed-off-by: Joe Elliott <[email protected]>
    @joe-elliott
    joe-elliott committed Oct 31, 2022
    Configuration menu
    Copy the full SHA
    8ea93f1 View commit details
    Browse the repository at this point in the history

  3. first pass fetch 

    Signed-off-by: Joe Elliott <[email protected]>
    @joe-elliott
    joe-elliott committed Oct 31, 2022
    Configuration menu
    Copy the full SHA
    192ff79 View commit details
    Browse the repository at this point in the history

  4. todo 

    Signed-off-by: Joe Elliott <[email protected]>
    @joe-elliott
    joe-elliott committed Oct 31, 2022
    Configuration menu
    Copy the full SHA
    c40b302 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2022

  1. Fix flaky wal tests, add benchmark just for write path

    @mdisibio
    mdisibio committed Nov 2, 2022
    Configuration menu
    Copy the full SHA
    72c9ca7 View commit details
    Browse the repository at this point in the history

  2. Reuse buffer/writer for efficiency, fix timestamp replay

    @mdisibio
    mdisibio committed Nov 2, 2022
    Configuration menu
    Copy the full SHA
    fa28664 View commit details
    Browse the repository at this point in the history

  3. Make wal block version configurable, update many tests

    @mdisibio
    mdisibio committed Nov 2, 2022
    Configuration menu
    Copy the full SHA
    49647be View commit details
    Browse the repository at this point in the history

  4. ingester flush wal after cutting traces

    @mdisibio
    mdisibio committed Nov 2, 2022
    Configuration menu
    Copy the full SHA
    b4ba183 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2022

  1. Merge branch 'main' into parquet-wal

    @mdisibio
    mdisibio committed Nov 4, 2022
    Configuration menu
    Copy the full SHA
    3f10f34 View commit details
    Browse the repository at this point in the history

  2. Fix parquet slack time handling by writing meta.json along with data … 

    …files on flush
    @mdisibio
    mdisibio committed Nov 4, 2022
    Configuration menu
    Copy the full SHA
    d26ffa5 View commit details
    Browse the repository at this point in the history

  3. Don't use parquet.GenericBuffer

    @mdisibio
    mdisibio committed Nov 4, 2022
    Configuration menu
    Copy the full SHA
    39b7a6a View commit details
    Browse the repository at this point in the history

  4. Pool trace buffers

    @mdisibio
    mdisibio committed Nov 4, 2022
    Configuration menu
    Copy the full SHA
    a0e2031 View commit details
    Browse the repository at this point in the history

  5. Add more wal benchmarks

    @mdisibio
    mdisibio committed Nov 4, 2022
    Configuration menu
    Copy the full SHA
    38b26d9 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2022

  1. Store map of trace ids to flushed page

    @mdisibio
    mdisibio committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    5d6b158 View commit details
    Browse the repository at this point in the history

  2. Read fewer columns on wal replay

    @mdisibio
    mdisibio committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    7c67bec View commit details
    Browse the repository at this point in the history

  3. Add completeblock benchmark for all encoding combinations

    @mdisibio
    mdisibio committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    d794ccb View commit details
    Browse the repository at this point in the history

  4. Use internal iterator when completing parquet wal -> parquet complete… 

    … block
    @mdisibio
    mdisibio committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    08c0f4f View commit details
    Browse the repository at this point in the history

  5. Fix tests and lint

    @mdisibio
    mdisibio committed Nov 7, 2022
    Configuration menu
    Copy the full SHA
    a5d98fe View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2022

  1. Rename/cleanup/fix benchmark config

    @mdisibio
    mdisibio committed Nov 8, 2022
    Configuration menu
    Copy the full SHA
    b5c2538 View commit details
    Browse the repository at this point in the history

  2. Wire up searching wal blocks, and don't write flatbuffer data for wal… 

    … blocks unless needed
    @mdisibio
    mdisibio committed Nov 8, 2022
    Configuration menu
    Copy the full SHA
    8803880 View commit details
    Browse the repository at this point in the history

  3. Fix flaky test

    @mdisibio
    mdisibio committed Nov 8, 2022
    Configuration menu
    Copy the full SHA
    5906d6f View commit details
    Browse the repository at this point in the history

  4. Test all combinations of wal->complete versions, fix bug in v2 create… 

    … path that was unused until now
    @mdisibio
    mdisibio committed Nov 8, 2022
    Configuration menu
    Copy the full SHA
    5c3b98d View commit details
    Browse the repository at this point in the history

  5. Use pooling in parquet wal->complete process

    @mdisibio
    mdisibio committed Nov 8, 2022
    Configuration menu
    Copy the full SHA
    52059e5 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2022

  1. Global trace pooling instead of per-iterator

    @mdisibio
    mdisibio committed Nov 9, 2022
    Configuration menu
    Copy the full SHA
    a75d66d View commit details
    Browse the repository at this point in the history

  2. Disable pooling again

    @mdisibio
    mdisibio committed Nov 9, 2022
    Configuration menu
    Copy the full SHA
    e0a6ddf View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into parquet-wal

    @mdisibio
    mdisibio committed Nov 9, 2022
    Configuration menu
    Copy the full SHA
    1532780 View commit details
    Browse the repository at this point in the history

  4. Honor ingestionslack, lint

    @mdisibio
    mdisibio committed Nov 9, 2022
    Configuration menu
    Copy the full SHA
    da5fcb8 View commit details
    Browse the repository at this point in the history

  5. changelog

    @mdisibio
    mdisibio committed Nov 9, 2022
    Configuration menu
    Copy the full SHA
    fe9b5a1 View commit details
    Browse the repository at this point in the history

  6. review feedback

    @mdisibio
    mdisibio committed Nov 9, 2022
    Configuration menu
    Copy the full SHA
    edb2712 View commit details
    Browse the repository at this point in the history

  7. Fix test

    @mdisibio
    mdisibio committed Nov 9, 2022
    Configuration menu
    Copy the full SHA
    6663964 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2022

  1. Ingester flush traces in order by ID, so we can skip buffering in the… 

    … wal
    @mdisibio
    mdisibio committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    04bf1dc View commit details
    Browse the repository at this point in the history

  2. comment

    @mdisibio
    mdisibio committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    29f12e9 View commit details
    Browse the repository at this point in the history

  3. comment

    @mdisibio
    mdisibio committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    f51937a View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2022

  1. Parquet wal maintain index of traces to rownumber to allow out-of-ord… 

    …er writes (although in-order performs better)
    @mdisibio
    mdisibio committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    6784137 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main into parquet-wal

    @mdisibio
    mdisibio committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    675fc64 View commit details
    Browse the repository at this point in the history

  3. update mod

    @mdisibio
    mdisibio committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    a7fac91 View commit details
    Browse the repository at this point in the history

  4. cleanup

    @mdisibio
    mdisibio committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    cc14484 View commit details
    Browse the repository at this point in the history

  5. Add partial replay test for parquet wal block

    @mdisibio
    mdisibio committed Nov 22, 2022
    Configuration menu
    Copy the full SHA
    8b0464b View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2022

  1. lint

    @mdisibio
    mdisibio committed Nov 28, 2022
    Configuration menu
    Copy the full SHA
    e058431 View commit details
    Browse the repository at this point in the history