-
-
Notifications
You must be signed in to change notification settings - Fork 410
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Terminals cannot be transformed in tree-less LALR? #818
Comments
@erezsh The problem is that the TERMINAL callbacks get added to |
I don't think there's a way to fix this that makes sense. We can process the terminals after the parser, but the user can do that manually as well, and it's better to be transparent about this. |
I suppose another approach would be to replace the return values with a proxy object, that also maintains the token type. But that sounds a little too "magical". |
After looking at the code, I think it might make sense to handle the Terminal Transformer like we do the rules transformer: Here similar to what happens a few line later. We would have to extend the callback generation to also add the Terminals, but that shouldn't be to hard. |
@MegaIng You could be right, that might work. |
Forgot to close this. This issue is already fixed. |
Describe the bug
I've written a transformer with the line
DECIMAL = float
(from%import common.DECIMAL
in the grammar) and it seemed to work fine post-transforming an Earley-parsed tree – that is, the relevant locations in the transformed tree would be48.0
rather thanToken('DECIMAL', '48.0')
and similar.I've written another transformer, this one for a grammar that I was able to make run in LALR. Therefore, I tried a tree-less parse. Now I get
ValueError: Callbacks must return a token (returned 48.0)
. To make the grammar work for a tree-less parse, I must wrap all my terminals in rules just to be able to transform them.This seems to me unnecessarily tedious, and expands the size of the grammar unhelpfully. Also, transformer callbacks for rules are more long-winded to write – you can't do
decimal = float
, at minimum you have to dodecimal = lambda self, n: return float(n[0])
To Reproduce
The text was updated successfully, but these errors were encountered: