-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce allocations in keyword lexing #1001
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In #723 we implemented contextual keywords but at the cost of always allocating the input substring that we attempt to promote into a contextual keyword. This is a remnant of how we implemented the lexer via macros and implemented the keyword promotion using a similar mechanism.
Problem
Currently, it's impossible to only borrow the string from the same time because we mutably borrow
ParserContext
to advance its innerposition
in the lexer macros.Possible solution
We're only collecting the identifier to see if it was fully recognized by the keyword trie (exactly matching its length), so it might be possible to just collect the length rather than allocating a string and passing it down to the macros.
Alternatively, we could rework the promotion mechanism not to use the keyword trie and combine the identifier+keyword lexer to return whether the identifier is directly promotable (contextual), reserved or a legal identifier.
The text was updated successfully, but these errors were encountered: