On Wed, Sep 16, 2020 at 5:31 AM Chris Angelico <rosuav@gmail.com> wrote:

Users would be encouraged to try it but NOT to publish code using it.

Clarification: by "publish" I mean in production, or as libraries that people will be depending on. It'd be on par with playing around in a fork of Python with the new feature, except that people can try it out without actually patching the C code. ChrisA

September 15, 2020 4:02 PM, "Daniel Butler" <dabutler89@gmail.com (mailto:dabutler89@gmail.com?to=%22Daniel%20Butler%22%20<dabutler89@gmail.com>)> wrote:

Users would be encouraged to try it but

NOT to publish code using it. Thinking out loud, macros could be enabled with a command line flag. Advanced users would know how to use it but others would not. If the macro flag is not enabled it raises a syntax error. Thoughts? --Thank you!

Daniel Butler A command line flag would be slightly better. All the documentation warnings in the world will not be enough to prevent all the cargo culters from creating some of the hardest to read code you ever saw. --Edwin​​​​​​​

I had not read it. Thanks for the clarification On Tue, Sep 15, 2020 at 4:32 PM Gregory P. Smith <greg@krypto.org> wrote:

On Tue, Sep 15, 2020 at 1:27 PM <edwin@211mainstreet.net> wrote:

...

September 15, 2020 4:02 PM, "Daniel Butler" <dabutler89@gmail.com <dabutler89@gmail.com?to=%22Daniel%20Butler%22%20%3Cdabutler89@gmail.com%3E>> wrote:

...

Users would be encouraged to try it but

...

NOT to publish code using it.

Thinking out loud, macros could be enabled with a command line flag. Advanced users would know how to use it but others would not. If the macro flag is not enabled it raises a syntax error. Thoughts? -- Thank you!

Daniel Butler

A command line flag would be slightly better. All the documentation warnings in the world will not be enough to prevent all the cargo culters from creating some of the hardest to read code you ever saw.

If you're talking about a command line flag, I suggest you read the pre-PEP. The proposal requires explicit import-like syntax to bring the macro in for parsing of code in the duration of the scope of that import. Which is actually what you'd want for this kind of thing: explicitly declaring which additional / alternate syntax features you use in the code using it. It is similar from the existing `from __future__ import behavior` system.

-gps

--

Thank you! Daniel Butler

Hi Daniel, On 15/09/2020 9:02 pm, Daniel Butler wrote:

> Users would be encouraged to try it but

> NOT to publish code using it.

Thinking out loud, macros could be enabled with a command line flag. Advanced users would know how to use it but others would not. If the macro flag is not enabled it raises a syntax error. Thoughts?

I don't like command line options that change behaviour. It seems to me that it would makes macro uses harder to understand. Cheers, Mark.

Hi David, On 15/09/2020 9:20 pm, David Mertz wrote:

Three-ish questions:

1. What could this do that Macropy does not already do? (yes, I know "run as top-level script", but that's uninspiring for me).

Macros in MacroPy don't look like macros, they look like normal Python code. The proposed macros have distinct syntax, making it obvious at a glance that they are macros. Since the macro name is a distinct token it makes it obvious, especially once editors start highlighting them differently. The proposed macros support additional syntactic forms. The additional stmt_expr AST node make it possible to support complex flow control like finally blocks, which MacroPy cannot do. The import hooks that MacroPy uses have no means to support macro versioning, resulting in either stale pyc files or excessive compilation. I'm not sure which.

2. Do you have any evidence Numba developers would actually want this?! (as claimed in draft FAQ).  I know a lot of them, and teach Numba sometimes, although have not directly contributed.  I do not believe they would prefer the AST to the bytecode for the analysis.

When writing a compiler you want the AST. It doesn't really matter what the compiler is. I'll change the example, so that I'm not attributing things to the numba developers without their consent.

3. I guess not really a question, but I just can't see a language feature like this as fully "opt-in".  Any feature that is added is going to occur in code I wind up having to maintain.  That's already a pain point with e.g. metaclasses/decorators, monkey patching, and other constructs, where code often doesn't mean what it *looks like* it would mean.

Whether a macro does what it looks like it does is largely down to naming and design. The same could be said of library code in general.

... macros, even if hygienic, seem to make that need to learn a new DSL or language for every code base much worse than those other constructs. Looking at code with entirely novel flow control constructs adds a lot of cognitive burden.  Not least because there are likely to be a hundred different implementations that make "case" or "do until" do very subtly different things in each one.

On Tue, Sep 15, 2020 at 9:27 AM Mark Shannon <mark@hotpy.org <mailto:mark@hotpy.org>> wrote:

Hi all,

I'd like to propose a new PEP for hygienic macros.

I'm attempting to reduce the controversy and pain of adding new syntax to Python, ... by adding new syntax.

That might sound silly, but I hope to persuade that it isn't.

Adding a new feature to the language is always controversial.

Before a new module or functionality is added to the standard library, it must usually prove itself as a package on PyPI.

(Not always the same package, the value of dataclasses was demonstrated by "attrs".)

However, this isn't currently possible with new language features.

Any new language feature has pros and cons. It adds expressive power for some (usually experienced) users, but potentially confuses and inconveniences other users (especially beginners). However, it is very hard, if not impossible, to determine whether adding new syntax will be be beneficial or harmful before the syntax is added.

Is there a way to add new syntax in a way that allows it be battle tested before adding to the released version of the language? Yes there is, Macros.

I'm not talking about C-style macros which do textual substitution, but "hygienic" macros that rewrite the AST during compilation.

Macros allow domain specific new syntactic features, without bloating the language or confusing newcomers.

In general, I would except macros to be used within libraries, so that those libraries gain the power of their custom macros without making Python ever larger and more complex.

However, there is no reason why they cannot be made more widely available. Should a macro become widely used and popular, then it can be considered for adoption into the language.

The PEP:

https://github.com/markshannon/peps/blob/hygenic-macros/pep-0700.rst

Cheers, Mark. _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7BX4S6RO... Code of Conduct: http://python.org/psf/codeofconduct/

-- The dead increasingly dominate and strangle both the living and the not-yet born.  Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.

On 16/09/20 9:34 am, Pablo Galindo Salgado wrote:

The first is that deleting a deep tree currently is just freeing the arena block, while if the nodes were PyObjects it will involve recursive destruction. That could potentially segfault

Not sure why that would be a concern. It's theoretically a possibility with any Python data, but we don't seem to worry about it. Why should it be a problem in the compiler specifically?

* The proposal seems to imply that the AST will be a fully public and stable API.

We already have two kinds of AST, the one used internally by the compiler, and the one exposed to Python in the ast module. The only time the latter should need to change is if there is a change in the syntax of the language, so why not promise to keep it stable and use it as the form taken and produced by the macro processors? -- Greg

On 16/09/20 12:37 pm, Guido van Rossum wrote:

IMO if we were to standardize the AST for times immemorial this would have to be a separate PEP.

Perhaps, but a stable AST seems to be a prerequisite for this kind of feature. Preferably one that corresponds as closely as possible to the language syntax so that users can keep it in their heads. Maybe the PEP should propose an AST of its own, which would initially be a third thing separate from either of the existing ones, with the possibility of adopting it as the ast module representation some time in the future. -- Greg

On Tue, Sep 15, 2020 at 7:31 PM Benjamin Peterson <benjamin@python.org> wrote:

On Tue, Sep 15, 2020, at 20:10, Greg Ewing wrote:

...

Maybe the PEP should propose an AST of its own, which would initially be a third thing separate from either of the existing ones, with the possibility of adopting it as the ast module representation some time in the future.

The rust approach to this problem to pass only a token stream to the macro. Then, the macro writer brings their own parser.

And (I looked this up [1]) it returns another token stream which is then re-parsed by the standard parser. I'm not quite sure who is responsible for determining where the macro ends -- the standard parser or the macro parser. A nice side effect of this would be that (hopefully) the macro syntax doesn't need to have the ugly special cases needed to allow `import! foo as bar` and `from! mod import foo as bar` (though these still need to be predefined macros). I'm not entirely sure how this would preserve the hygiene of the macros though, since the macro's parser could just do a token-by-token substitution without checking the context, effectively what a macro in C does. Forcing an AST on the macros restricts macros somewhat (e.g. it requires the input arguments to look like well-formed expressions) but that's probably a good thing. (Hm, or not. How would you implement something like PEP 505 using macros? It introduces new tokens like `?.` and `??`. [1] https://doc.rust-lang.org/book/ch19-06-macros.html -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

On Tue, Sep 15, 2020, at 23:54, Guido van Rossum wrote:

On Tue, Sep 15, 2020 at 7:31 PM Benjamin Peterson <benjamin@python.org> wrote:

...

On Tue, Sep 15, 2020, at 20:10, Greg Ewing wrote:

...

Maybe the PEP should propose an AST of its own, which would initially be a third thing separate from either of the existing ones, with the possibility of adopting it as the ast module representation some time in the future.

The rust approach to this problem to pass only a token stream to the macro. Then, the macro writer brings their own parser.

And (I looked this up [1]) it returns another token stream which is then re-parsed by the standard parser. I'm not quite sure who is responsible for determining where the macro ends -- the standard parser or the macro parser.

The standard parser determines the macro boundaries and passes the macro implementation only the relevant tokens.

A nice side effect of this would be that (hopefully) the macro syntax doesn't need to have the ugly special cases needed to allow `import! foo as bar` and `from! mod import foo as bar` (though these still need to be predefined macros).

I'm not entirely sure how this would preserve the hygiene of the macros though, since the macro's parser could just do a token-by-token substitution without checking the context, effectively what a macro in C does. Forcing an AST on the macros restricts macros somewhat (e.g. it requires the input arguments to look like well-formed expressions) but that's probably a good thing.

The rust macro TokenStream is a bit higher level than a traditional token stream. In particular, the macro implementation can request that new identifiers it inserts into the stream are only resolvable within the macro for hygiene. There's a bit about that here: https://doc.rust-lang.org/proc_macro/struct.Ident.html#impl

(Hm, or not. How would you implement something like PEP 505 using macros? It introduces new tokens like `?.` and `??`.

Is there any proposal that would let you add new tokens?

Hi Benjamin, I don't like the idea of just passing a stream of tokens to the macro processor, for three reasons. 1. It makes life unnecessarily difficult for macro implementers by forcing them to parse the stream of tokens. Macros may contain difficult to parse constructs, and other macros. If all that ends up happening is that macro processors reconstruct the source and pass it to ast.parse(), then what's the point? 2. I want macros to have familiar syntactic structure, even if the semantics are novel. 3. Pablo has already pointed out the potential performance impact of exposing the AST at the Python level. Passing tokens around would mean that both the AST nodes and lexer tokens would need to be Python objects, making the performance impact much greater. Cheers, Mark. On 16/09/2020 2:33 pm, Benjamin Peterson wrote:

On Tue, Sep 15, 2020, at 23:54, Guido van Rossum wrote:

...

On Tue, Sep 15, 2020 at 7:31 PM Benjamin Peterson <benjamin@python.org> wrote:

...

On Tue, Sep 15, 2020, at 20:10, Greg Ewing wrote:

...

Maybe the PEP should propose an AST of its own, which would initially be a third thing separate from either of the existing ones, with the possibility of adopting it as the ast module representation some time in the future.

The rust approach to this problem to pass only a token stream to the macro. Then, the macro writer brings their own parser.

And (I looked this up [1]) it returns another token stream which is then re-parsed by the standard parser. I'm not quite sure who is responsible for determining where the macro ends -- the standard parser or the macro parser.

The standard parser determines the macro boundaries and passes the macro implementation only the relevant tokens.

...

A nice side effect of this would be that (hopefully) the macro syntax doesn't need to have the ugly special cases needed to allow `import! foo as bar` and `from! mod import foo as bar` (though these still need to be predefined macros).

I'm not entirely sure how this would preserve the hygiene of the macros though, since the macro's parser could just do a token-by-token substitution without checking the context, effectively what a macro in C does. Forcing an AST on the macros restricts macros somewhat (e.g. it requires the input arguments to look like well-formed expressions) but that's probably a good thing.

The rust macro TokenStream is a bit higher level than a traditional token stream. In particular, the macro implementation can request that new identifiers it inserts into the stream are only resolvable within the macro for hygiene. There's a bit about that here: https://doc.rust-lang.org/proc_macro/struct.Ident.html#impl

...

(Hm, or not. How would you implement something like PEP 505 using macros? It introduces new tokens like `?.` and `??`.

Is there any proposal that would let you add new tokens? _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GFDSVQ4Q... Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, Sep 16, 2020 at 3:53 PM Mark Shannon <mark@hotpy.org> wrote:

[...]

maybe!(a.b)

which would translate to:

$tmp = a; None if $tmp is None else ($tmp.b)

This reminds me very much of PL/1's "compile-time procedures," which forty years ago were pretty much like PL/1 constructs with percent signs sprinkled about. I didn't like those much, either ;D.

* Not using an arena allocator for the nodes can introduce more challenges

...

than simplifications. The first is that deleting a deep tree currently is just freeing the arena block, while if the nodes were PyObjects it will involve recursive destruction. That could potentially segfault so we would need to use some custom trashcan mechanism of special deleters. All of this will certainly not simplify the code (at least the parser code) and will impact performance (although just in the parser/compiler phase).

* We would need to (potentially) reimplement the AST sequences into proper owning-containers. That will involve changing a considerable amount of code and some slowdown due to having to use C-API calls.

There is probably another way. We already have code to convert between the C-level AST (the one that's arena-allocated) and the Python-level AST (the one that the `ast` module provides). Mark doesn't seem to mind if processing macros slows down parsing (since .pyc file caching still works). So we could convert the C-level AST to a Python-level AST, give that to the macro processor, which returns another Python-level AST, and then we convert that back to a C-level AST that we graft into the parse tree for the source being parsed

Given that macros can return more macros I fear this process may be prohibitively slow because not only you need to create the python structures at each time, but also you would need to delete them as well and that will be always slower. How slow remains of course to be seen, but given that the chain can be unbounded is enough to worry me a little. Even if we make the argument that is fine because this is a compile phase, there are a considerable amount of cases where compilation may happen at runtime, so although is not that critical as something like the eval loop, it may become a potential concern.

Hi Pablo, On 15/09/2020 10:34 pm, Pablo Galindo Salgado wrote:

Thanks for the proposal Mark!

I wanted to make some comments regarding converting AST nodes to PyObjects internally. I see some challenges here:

* Not using an arena allocator for the nodes can introduce more challenges than simplifications. The first is that deleting a deep tree currently is just freeing the arena block, while if the nodes were PyObjects it will involve recursive destruction. That could potentially segfault so we would need to use some custom trashcan mechanism of special deleters. All of this will certainly not simplify the code (at least the parser code) and will impact performance (although just in the parser/compiler phase).

Don't we need to do all of this in the _ast module, already? We already have an AST composed of Python objects.

* We would need to (potentially) reimplement the AST sequences into proper owning-containers. That will involve changing a considerable amount of code and some slowdown due to having to use C-API calls.

I agree that the sequences would need additional code for them to be Python objects. They would be more expensive to create, but I don't the additional cost being prohibitive.

* The proposal seems to imply that the AST will be a fully public and stable API. This has some danger as any internal optimization regarding AST can be braking changes to macro users. This will raise any problems that now linter and static analyst tools could have to all users (of macros), making of even more difficult to change it.

This is a good point. I am going to change macros to take a testlist instead of an arglist. Historically, function arguments have been an unstable part of the API. Since the AST node for testlist is just a list of ast.expr, it should be stable. That way at least the top level of the AST will be stable. Cheers, Mark.

Regards from sunny London, Pablo Galindo

On Tue, 15 Sep 2020, 20:28 Mark Shannon, <mark@hotpy.org <mailto:mark@hotpy.org>> wrote:

Hi all,

I'd like to propose a new PEP for hygienic macros.

I'm attempting to reduce the controversy and pain of adding new syntax to Python, ... by adding new syntax.

That might sound silly, but I hope to persuade that it isn't.

Adding a new feature to the language is always controversial.

Before a new module or functionality is added to the standard library, it must usually prove itself as a package on PyPI.

(Not always the same package, the value of dataclasses was demonstrated by "attrs".)

However, this isn't currently possible with new language features.

Any new language feature has pros and cons. It adds expressive power for some (usually experienced) users, but potentially confuses and inconveniences other users (especially beginners). However, it is very hard, if not impossible, to determine whether adding new syntax will be be beneficial or harmful before the syntax is added.

Is there a way to add new syntax in a way that allows it be battle tested before adding to the released version of the language? Yes there is, Macros.

I'm not talking about C-style macros which do textual substitution, but "hygienic" macros that rewrite the AST during compilation.

Macros allow domain specific new syntactic features, without bloating the language or confusing newcomers.

In general, I would except macros to be used within libraries, so that those libraries gain the power of their custom macros without making Python ever larger and more complex.

However, there is no reason why they cannot be made more widely available. Should a macro become widely used and popular, then it can be considered for adoption into the language.

The PEP:

https://github.com/markshannon/peps/blob/hygenic-macros/pep-0700.rst

Cheers, Mark. _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7BX4S6RO... Code of Conduct: http://python.org/psf/codeofconduct/

On 16/09/2020 12:22 pm, Pablo Galindo Salgado wrote:

> Don't we need to do all of this in the _ast module, already? > We already have an AST composed of Python objects

We have AST of Python objects, but the python versions are not used internally, especially in the parser, where they are created. The parser and the compiler currently use exclusively the C versions of the nodes. Changing the parser to use Python versions and especially not to use the arena allocator can complicate the parser substantially, and destroying the resulting tree can be expensive (as destroying a deep tree composed of Python objects currently is).

Would it complicate the parser much to allocate memory using a different allocator? It's just a function call, presumably. As for performance, we don't know how much difference it will make until we try. Deallocation doesn't have to be recursive. Cheers, Mark.