Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor datasets and tokenizer #624

Merged
merged 11 commits into from
Apr 2, 2024
Merged

Refactor datasets and tokenizer #624

merged 11 commits into from
Apr 2, 2024

Commits on Mar 31, 2024

  1. [RFC] Refactor datasets and tokenizer

    @ebsmothers
    ebsmothers committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    b9e0f5f View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. add option to remove leading whitespace

    @ebsmothers
    ebsmothers committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    117dc32 View commit details
    Browse the repository at this point in the history

  2. address comments, add tokenizer test, better whitespace handling

    @ebsmothers
    ebsmothers committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    73ef79d View commit details
    Browse the repository at this point in the history

  3. better choice of prefix

    @ebsmothers
    ebsmothers committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    a6e24d7 View commit details
    Browse the repository at this point in the history

  4. get tests passing

    @ebsmothers
    ebsmothers committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    6592526 View commit details
    Browse the repository at this point in the history

  5. add missing assertion

    @ebsmothers
    ebsmothers committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    d71f397 View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2024

  1. address comments

    @ebsmothers
    ebsmothers committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    5a66885 View commit details
    Browse the repository at this point in the history

  2. address comments

    @ebsmothers
    ebsmothers committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    dbc646e View commit details
    Browse the repository at this point in the history

  3. missed dead import

    @ebsmothers
    ebsmothers committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    c29e8cb View commit details
    Browse the repository at this point in the history

  4. bug fix and some cleanup

    @ebsmothers
    ebsmothers committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    f9ec55d View commit details
    Browse the repository at this point in the history

  5. address remaining comments

    @ebsmothers
    ebsmothers committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    5603599 View commit details
    Browse the repository at this point in the history