Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add split one token into several (resolves #2838) #3253

Merged
merged 10 commits into from
Feb 14, 2019
Merged

Add split one token into several (resolves #2838) #3253

merged 10 commits into from
Feb 14, 2019

Commits on Feb 8, 2019

  1. Add split one token into several (resolves explosion#2838)

    @grivaz
    grivaz committed Feb 8, 2019
    Configuration menu
    Copy the full SHA
    da36b35 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2019

  1. Improve error message for token splitting

    @honnibal
    honnibal authored Feb 13, 2019
    Configuration menu
    Copy the full SHA
    9864236 View commit details
    Browse the repository at this point in the history

  2. Make retokenizer.split() tests use a Token object 

    Change retokenizer.split() to use a Token object, instead of an index.
    @honnibal
    honnibal authored Feb 13, 2019
    Configuration menu
    Copy the full SHA
    4e36c41 View commit details
    Browse the repository at this point in the history

  3. Pass Token into retokenize.split() 

    Tweak retokenize.split() API so that we pass the `Token` object, not the index.
    @honnibal
    honnibal authored Feb 13, 2019
    Configuration menu
    Copy the full SHA
    822ccd5 View commit details
    Browse the repository at this point in the history

  4. Fix token.idx in retokenize.split()

    @honnibal
    honnibal authored Feb 13, 2019
    Configuration menu
    Copy the full SHA
    be9bbfb View commit details
    Browse the repository at this point in the history

  5. Test that token.idx is correct after split

    @honnibal
    honnibal authored Feb 13, 2019
    Configuration menu
    Copy the full SHA
    afc5b3e View commit details
    Browse the repository at this point in the history

  6. Fix token.idx for split tokens

    @honnibal
    honnibal authored Feb 13, 2019
    Configuration menu
    Copy the full SHA
    7356410 View commit details
    Browse the repository at this point in the history

  7. Fix retokenize.split()

    @honnibal
    honnibal authored Feb 13, 2019
    Configuration menu
    Copy the full SHA
    e7dc0ec View commit details
    Browse the repository at this point in the history

  8. Fix retokenize.split

    @honnibal
    honnibal authored Feb 13, 2019
    Configuration menu
    Copy the full SHA
    906022e View commit details
    Browse the repository at this point in the history

  9. Fix retokenize.split() test

    @honnibal
    honnibal authored Feb 13, 2019
    Configuration menu
    Copy the full SHA
    7841edd View commit details
    Browse the repository at this point in the history