Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce allocations in keyword lexing #1001

Open
Xanewok opened this issue Jun 4, 2024 · 0 comments
Open

Reduce allocations in keyword lexing #1001

Xanewok opened this issue Jun 4, 2024 · 0 comments

Comments

@Xanewok
Copy link
Contributor

Xanewok commented Jun 4, 2024

In #723 we implemented contextual keywords but at the cost of always allocating the input substring that we attempt to promote into a contextual keyword. This is a remnant of how we implemented the lexer via macros and implemented the keyword promotion using a similar mechanism.

Problem

Currently, it's impossible to only borrow the string from the same time because we mutably borrow ParserContext to advance its inner position in the lexer macros.

Possible solution

We're only collecting the identifier to see if it was fully recognized by the keyword trie (exactly matching its length), so it might be possible to just collect the length rather than allocating a string and passing it down to the macros.
Alternatively, we could rework the promotion mechanism not to use the keyword trie and combine the identifier+keyword lexer to return whether the identifier is directly promotable (contextual), reserved or a legal identifier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: ⏳ Todo
Development

No branches or pull requests

2 participants