Hello Paul (and everyone else),
On Sun, Nov 29, 2020 at 12:00 PM Paul Sokolovsky email@example.com wrote:
# Experimental Syntax Proposal
I would like to propose that Python adopts a modified process before introducing significant changes of its syntax.
## New suggested process
Assuming one of the options mentioned above is adopted, before changes to the syntax are introduced in a Python version, they would be made available to be used either as an encoding variant or an import hook giving enough time for interested pythonistas to experiment with the new syntax, writing actual code and possibly trying out various alternatives.
Thanks for posting this proposal. It should be pretty clear that's the best process which should be followed.
It's also should be pretty clear that (almost) nobody follows it. If anything, lack of responses by the substance of the matter proposed, i.e. the *process*, not technical means to achieve it, is very telling.
I had taken the lack of response as indicative that my idea was simply "wrong".
So again, I fully agree that trying to implement any Python changes in the Python itself (instead of rushing to hack a particular implementation in C) should the best practice.
Now let's talk why it's not that way. The main reasons would be lack of technical best practices to achieve that, and perceived complexity of existing ways to do that.
Even from your proposal lack of the "best" solution is clear:
By using an import hook. This is the most powerful approach as it allows one to do changes either at the source level (prior to parsing), or at the AST level, or both.
By using a custom encoding. This only allows transformations at the
So, "custom encoding" way is essentially a hack. As it operates on the surface "stream of characters" representation of a program, only trivial, or imprecise transformations can be implemented.
Aside: some fairly complex transformations can be implemented using codecs; see https://github.com/dropbox/pyxl for example.
Alternatively, a program can be parsed, modified and dumped again as a stream of characters, just to immediately be parsed by the Python interpreter again. In either case, it's a hack, and should be rejected as a viable approach to a problem domain.
Now you say that the second choice is "import hooks". But import hooks in Python is its own kingdom of "wonderland". Import hooks aren't intended to "allow to experiment with syntax", they are intended to do anything what's possible with module loading (and many things that aren't possible, I would add). So, they have rather complex, non-intuitive API, thru which anyone would need to wade, neck-deep, before implementing something useful.
In my original email, I linked to a project I created ("Ideas": https://aroberge.github.io/ideas/docs/html/) which meant to simplify this as much as possible.
In other words, both choices you list aren't really viable to experiment with syntax/semantics.
I respectfully disagree, based on my own experimentation, documented in "Ideas" mentioned previously. For example, I implemented module level constants using this approach. They *almost* work ... except for some corner cases due to the fact that I am using module objects created using Python's type() as I have not figured out (yet?) how to create a module object with a custom dict.
That may be a good explanation why nobody rushes to.
I'd formulate choices to experiment with syntax/semantics differently. Let's remember the compilation pipeline: the source is tokenized, then it's parsed into AST, then AST is compiled into bytecode.
I tried to represent this visually on
- The central part of the pipeline is the AST. Many interesting
features can be implemented on AST level. Others can be at least prototyped, using existing syntactic elements (by assigning to them new semantics). Of course, arbitrary syntax changes can't be implemented this way.
Exactly. Which is why I suggested using either import hooks or encodings.
[Snip: explanation leading to discussion of Python2 "compiler" package which is referred to again at the end]
What's needed is higher-level API specifically for the usecase of letting Python source of modules to run thru custom tokenizer/parser/bytecode compiler. The best known (non-adhoc) approach to that is PEP511: https://www.python.org/dev/peps/pep-0511/.
I believe that the intention of PEP-511 was to optimize the code generated, and not to experiment with different syntax. I *think* that my ideas project is essentially a superset of what is described in pep-511, but with a completely different goal.
This turned out to be a long intro. As I said, I fully agree with you, that changes to Python should be prototyped in Python. And there's no better way to "agree" than actually dogfood this approach to oneself.
Yes. One additional comment, point of emphasis about my original proposal.
Take PEP 505 (None-aware operators) written 5 years ago. It is still deferred.
Imagine that someone wants to experiment with a custom implementation. Using the approach I mentioned, it should be possible to do so.
Suppose that the same person wishes to also experiment with block-level scoping. By using the approach I mentioned in my original message, it would be possible to "stack" transformations and experiment with code that support either None aware operators and/or block-level scoping. Different syntax variants could be implemented separately and, provided they do not create conflicts, could be tested together so that people could get a feel as to whether or not some proposed change should be included in Python.
As I lately argued here on the list that implementing block-level scoping for Python is "not rocket science at all" and "a notch above trivial", I decided to do just that - and subject myself to coding it up, using the very python-compiler project I mentioned above. The result is a branch on that repo: https://github.com/pfalcon/python-compiler/tree/for-block-scope
(I'll post a separate message with details.)
I look forward to reading it.
-- Best regards, Paul mailto:firstname.lastname@example.org