Hello Paul (and everyone else),

On Sun, Nov 29, 2020 at 12:00 PM Paul Sokolovsky <pmiscml@gmail.com> wrote:

> # Experimental Syntax Proposal
> I would like to propose that Python adopts a modified process
> before introducing significant changes of its syntax.


> ## New suggested process
> Assuming one of the options mentioned above is adopted,
> before changes to the syntax are introduced in a Python version,
> they would be made available to be used either as an encoding variant
> or an import hook giving enough time for interested pythonistas to
> experiment with the new syntax, writing actual code and possibly
> trying out various alternatives.

Thanks for posting this proposal. It should be pretty clear that's
the best process which should be followed.
Thank you.

It's also should be pretty
clear that (almost) nobody follows it. If anything, lack of responses
by the substance of the matter proposed, i.e. the *process*, not
technical means to achieve it, is very telling.
I had taken the lack of response as indicative that my idea was simply "wrong".


So again, I fully agree that trying to implement any Python changes in
the Python itself (instead of rushing to hack a particular
implementation in C) should the best practice.

Now let's talk why it's not that way. The main reasons would be lack of
technical best practices to achieve that, and perceived complexity of
existing ways to do that.

Even from your proposal lack of the "best" solution is clear:

> * By using an import hook. This is the most powerful approach as it
>   allows one to do changes either at the source level
>   (prior to parsing), or at the AST level, or both.
> * By using a custom encoding. This only allows transformations at the
> source level.

So, "custom encoding" way is essentially a hack. As it operates on the
surface "stream of characters" representation of a program, only
trivial, or imprecise transformations can be implemented.

Aside: some fairly complex transformations can be implemented using codecs; see https://github.com/dropbox/pyxl for example.

Alternatively, a program can be parsed, modified and dumped again as a
stream of characters, just to immediately be parsed by the Python
interpreter again. In either case, it's a hack, and should be rejected
as a viable approach to a problem domain.

Now you say that the second choice is "import hooks". But import hooks
in Python is its own kingdom of "wonderland". Import hooks aren't
intended to "allow to experiment with syntax", they are intended to do
anything what's possible with module loading (and many things that
aren't possible, I would add). So, they have rather complex,
non-intuitive API, thru which anyone would need to wade, neck-deep,
before implementing something useful.

In my original email, I linked to a project I created ("Ideas": https://aroberge.github.io/ideas/docs/html/) which meant to simplify this as much as possible.

In other words, both choices you list aren't really viable to experiment
with syntax/semantics.
I respectfully disagree, based on my own experimentation, documented in "Ideas" mentioned previously. For example, I implemented module level constants using this approach. They *almost* work ... except for some corner cases due to the fact that I am using module objects created using Python's type() as I have not figured out (yet?) how to create a module object with a custom dict.
That may be a good explanation why nobody rushes

I'd formulate choices to experiment with syntax/semantics differently.
Let's remember the compilation pipeline: the source is tokenized, then
it's parsed into AST, then AST is compiled into bytecode.

I tried to represent this visually on https://aroberge.github.io/ideas/docs/html/possible.html

1. The central part of the pipeline is the AST. Many interesting
features can be implemented on AST level. Others can be at least
prototyped, using existing syntactic elements (by assigning to them
new semantics). Of course, arbitrary syntax changes can't be
implemented this way.
Exactly. Which is why I suggested using either import hooks or encodings.

[Snip: explanation leading to discussion of Python2 "compiler" package which is referred to again at the end]

What's needed is higher-level API specifically for the usecase of
letting Python source of modules to run thru custom
tokenizer/parser/bytecode compiler. The best known (non-adhoc) approach
to that is PEP511: https://www.python.org/dev/peps/pep-0511/.
I believe that the intention of PEP-511 was to optimize the code generated, and not to experiment with different syntax. I *think* that my ideas project is essentially a superset of what is described in pep-511, but with a completely different goal.




This turned out to be a long intro. As I said, I fully agree with you,
that changes to Python should be prototyped in Python. And there's no
better way to "agree" than actually dogfood this approach to oneself.

Yes. One additional comment, point of emphasis about my original proposal.

Take PEP 505 (None-aware operators) written 5 years ago. It is still deferred. 

Imagine that someone wants to experiment with a custom implementation. Using the approach I mentioned, it should be possible to do so.

Suppose that the same person wishes to also experiment with block-level scoping.  By using the approach I mentioned in my original message, it would be possible to "stack" transformations and experiment with code that support either None aware operators and/or block-level scoping.  Different syntax variants could be implemented separately and, provided they do not create conflicts, could be tested together so that people could get a feel as to whether or not some proposed change should be included in Python.


As I lately argued here on the list that implementing block-level
scoping for Python is "not rocket science at all" and "a notch above
trivial", I decided to do just that - and subject myself to coding it
up, using the very python-compiler project I mentioned above. The
result is a branch on that repo:

(I'll post a separate message with details.)

I look forward to reading it.

Best regards,


Best regards,
 Paul                          mailto:pmiscml@gmail.com