I don't like the idea of just passing a stream of tokens to the macro processor, for three reasons.
1. It makes life unnecessarily difficult for macro implementers by forcing them to parse the stream of tokens. Macros may contain difficult to parse constructs, and other macros. If all that ends up happening is that macro processors reconstruct the source and pass it to ast.parse(), then what's the point?
2. I want macros to have familiar syntactic structure, even if the semantics are novel.
3. Pablo has already pointed out the potential performance impact of exposing the AST at the Python level. Passing tokens around would mean that both the AST nodes and lexer tokens would need to be Python objects, making the performance impact much greater.
On 16/09/2020 2:33 pm, Benjamin Peterson wrote:
On Tue, Sep 15, 2020, at 23:54, Guido van Rossum wrote:
On Tue, Sep 15, 2020 at 7:31 PM Benjamin Peterson email@example.com wrote:
On Tue, Sep 15, 2020, at 20:10, Greg Ewing wrote:
Maybe the PEP should propose an AST of its own, which would initially be a third thing separate from either of the existing ones, with the possibility of adopting it as the ast module representation some time in the future.
The rust approach to this problem to pass only a token stream to the macro. Then, the macro writer brings their own parser.
And (I looked this up ) it returns another token stream which is then re-parsed by the standard parser. I'm not quite sure who is responsible for determining where the macro ends -- the standard parser or the macro parser.
The standard parser determines the macro boundaries and passes the macro implementation only the relevant tokens.
A nice side effect of this would be that (hopefully) the macro syntax doesn't need to have the ugly special cases needed to allow `import! foo as bar` and `from! mod import foo as bar` (though these still need to be predefined macros).
I'm not entirely sure how this would preserve the hygiene of the macros though, since the macro's parser could just do a token-by-token substitution without checking the context, effectively what a macro in C does. Forcing an AST on the macros restricts macros somewhat (e.g. it requires the input arguments to look like well-formed expressions) but that's probably a good thing.
The rust macro TokenStream is a bit higher level than a traditional token stream. In particular, the macro implementation can request that new identifiers it inserts into the stream are only resolvable within the macro for hygiene. There's a bit about that here: https://doc.rust-lang.org/proc_macro/struct.Ident.html#impl
(Hm, or not. How would you implement something like PEP 505 using macros? It introduces new tokens like `?.` and `??`.
Is there any proposal that would let you add new tokens? _______________________________________________ Python-Dev mailing list -- firstname.lastname@example.org To unsubscribe send an email to email@example.com https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://firstname.lastname@example.org/message/GFDSVQ4Q... Code of Conduct: http://python.org/psf/codeofconduct/