Mailman 3 December 2020 - Python-ideas

__init__ in module names
by Gregory Szorc 12 Dec '20

12 Dec '20

PyOxidizer's pure Rust implementation of a meta path importer ( https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer_oxidized_find…) has been surprisingly effective at finding corner cases and behavior quirks in Python's importing mechanisms. It was recently brought to my attention via https://github.com/indygreg/PyOxidizer/issues/317 that "__init__" in module names is something that exists in Python code in the wild. (See https://github.com/search?l=Python&q=%22from+.__init__+import%22&type=Code for some examples.) In that GitHub issue and https://bugs.python.org/issue42564, I discovered that what's happening is the stdlib PathFinder meta path importer is "dumb" and doesn't treat "__init__" in module names specially. If someone uses syntax like "import foo.__init__" or "from .__init__ import foo", PathFinder operates on "__init__" like any other string value and proceeds to probe the filesystem for the relevant {.py, .pyc, .so, etc} files. The "__init__" files do exist in probed locations and PathFinder summarily constructs a new module object, albeit with "__init__" in its name. The end result is you have 2 module objects and sys.modules entries referring to the same file, keyed to different names (e.g. "foo" and "foo.__init__"). There is a strong argument to be made that "__init__" in module names should be treated specially. It seems wrong to me that you are allowed to address the same module/file through different names (let's pretend filesystem path normalization doesn't exist) and that the filesystem encoding of Python module files/names is addressable through the importer names. This feels like a bug that inadvertently shipped. However, code in the wild is clearly relying on "__init__" in module names being allowed. And changing the behavior is backwards incompatible and could break this code. Anyway, I was encouraged by Brett Cannon to email this list to assess the appetite for introducing a backwards incompatible change to this behavior. So here's my strawman/hardline proposal: 1. 3.10 introduces a DeprecationWarning for "__init__" appearing as any module part component (`"__init__" in fullname.split(".")`). 2. Some future release (I'm unsure which) turns it into a hard error. (A less aggressive proposal would be to normalize "__init__" in module names to something more reasonable - maybe stripping trailing ".__init__" from module names. But I'll start by proposing the stricter solution.) What do others think we should do? Gregory

8 21

An itertools.interleave / itertools.join function
by aurelien.lambert.89＠gmail.com 11 Dec '20

11 Dec '20

Hi I like using itertools for creating long strings while not paying the cost of intermediate strings (by eventually calling str.join on the whole iterator). However, one missing feature is to mimic the behavior of str.join as an iterator: an iterator that returns the items of an iterable, separated by the separator. I suggest name "interleave" or "join" (whichever is the most clear / least ambigous). def interleave(sep, iterable): """ Makes an iterator that returns elements from an iterable, separated by the separator. """ notfirst = False for i in iterable: if notfirst: yield sep else: notfirst = True yield i Could imagine a more elaborate implementation that can take several iterators, and would be equivalent to lambda chain_zip_interleave sep, *iterables: itertools.chain.from_iterable(interleave((sep,), zip(*iterables))) But that may be seriously overkill, and I have hard time describing it.

6 8

Proposal for __encode__ method for json encoder and dataclasses
by sam bland 11 Dec '20

11 Dec '20

In response to the additional work required to convert the new python dataclass using the json encoder I propose an __encode__ method that will be included in the default dataclass attributes. This would then be picked up by the default json encoder and called to encode the object. Using a common __encode__ tag would allow the solution to also be applied to custom objects and other new objects without needing to update all json encoders. The __encode__ method would work similar to __repr__ but output a json string of the object.

5 5

Proposal: typing.affix_type_hints
by Paul Bryan 11 Dec '20

11 Dec '20

With PEP 563 behavior going mainstream in Python 3.10, I see some ramifications, especially for runtime validation of type hints: 1. PEP 563 is changing the scope of evaluation of type hints, and in some cases it is not feasible to move that scope to the annotated object's globals. (I appreciate the reality that such cases would evade static type checkers.) 2. The typing.get_type_hints function causes annotation(s) to be evaluated from strings for every call, incurring a potentially significant runtime penalty depending on how often type hints are evaluated at runtime. To address both of these issues, I'm proposing that a new function be added to typing: affix_type_hints. I'm thinking something like this: def affix_type_hints(obj, globalns=None, localns=None): """ Affixes an object's type hints to the object. Type hints are affixed by first being resolved through get_type_hints, then by storing the result in the object's __annotations__ attribute. If the object is a class, this function will affix annotations from all superclasses. """ if hints := typing.get_type_hints(obj, globalns, localns, include_extras=True): obj.__annotations__ = hints Advantages: 1. The time and scope of type hint affixation are under the control of the caller. Future references can be realized before type hints are affixed. If there is a novel local scope required to resolve the type hint, it can be supplied in-context; future calls to get_type_hints inherit the benefit of such pre-resolution. 2. Annotations will not be re-evaluated on repeated calls to get_type_hints. 3. It doesn't require a change to the behavior of the get_type_hints function. Disadvantages: 1. Subclasses inheriting superclass annotations may be undesirable. 2. Monkey-patching __annotations__ may be considered an anti-pattern. 3. Maybe allowing re-evaluation on every get_type_hints call is desirable? (I'd rather avoid the potential for side-effects.) 4. The cited example is a 2-line function; perhaps its triviality makes it unjustified to add to the stdlib. Alternatives: 1. A caching version of get_type_hints. It looks like ForwardRef.__forward_value__ could actually deliver if annotations were encoded as ForwardRef objects instead of string literals. Am I missing a way for ForwardRef to cache evaluated values when get_type_hints is called? 2. Something I haven't considered? Please let me know. Paul

2 2

Lazy Type Casting
by Mathew Elman 10 Dec '20

10 Dec '20

I am not sure if this has been suggested before, so my apologies if it has. I would like to propose adding lazy types for casting builtins in a lazy fashion. e.g. `lazy_tuple` which creates a reference to the source iterable and a morally immutable sequence but only populates the tupular container when it or the source is used. Note that if the original object is not a built-in, will not be used after casting _or_ is immutable, this is trivial to implement. The difficulty arises when this it not the case, since the lazy object needs to also freeze any values that are mutated in the original to avoid side-effects. An alternative to adding lazy types / lazy type casting (making it possible to implement these oneself) would be to add method call hooks to python, since this would allow having a "freeze value" callback hooked into the __setitem__ and __getitem__ methods. Hooks may be a more useful solution for wider use cases as well.

5 10

Make for/while loops nameable.
by Jonatan 08 Dec '20

08 Dec '20

Hi, sometimes I do nested loops and I want to break specific loop, outer/inner etc. I need to do an ugly thing with for..else and it's annoying. It'd be nice if we could do so: for i in range(10) as loop_i: for j in range(i, i + 10) as loop_j: if i + j == 9: break loop_i or something like this. Please comment what do you think about this idea.

15 17

Thinking about dead simple import hooks...
by Paul Sokolovsky 08 Dec '20

08 Dec '20

Hello, It would be nice to have a simple and easy way to override module importing, instead of that babylon which CPython has in that regard. (Even if initially "limited" to filesystem importing, most people won't ever find that limiting at all.) The desired semantics would be: For "import foo", before "/path1/foo.py", "/path2/foo.py" (where /path1:/path2 come from sys.path) is tried to be imported, a user-defined hook function can be called. It should take just one param - full basename of the module to be imported, e.g. "/path1/foo", "/path2/foo" for the cases above. The hook should either return a module object, the result of the import, or None to signal that standard processing should take place. The hook should be aware that there might have been previous hooks, and it's current hook's responsibility to call them. Example of a hook: def import_hook(path): if os.path.exists(path + ".my_ext"): mod = imp.new_module("") # populate mod here return mod A crucial semantic detail is that all possible module types from one sys.path entry should be tried first, before going to the next entry. (CPython import extensibility instead seems to be built around idea that it's fine to do multiple passes over sys.path, loading only a particular extension module type on each pass, which is not the desired semantics). Questions: 1. Does anybody know such thing to already exist? 2. Would you tweak anything in the API/boilerplate semantics of the above? -- Best regards, Paul mailto:pmiscml@gmail.com

3 6

Re: Make for/while loops nameable.
by Aveheuzed 06 Dec '20

06 Dec '20

> I think this is the kind of feature that can very easily be abused. > Whenever I want to break out of a nested loop I take this as an > opportunity to extract the loop into its own function and use the > return statement to break out of the loop. IMO this is a lot better > than having named or indexed loop control because a proper function > can later be reusable, and it lets you explain why you need to return > early in the docstring. I agree with you that this feature can — and certainly will — be abused. But I don't think that's relevant: when you think about it, any feature can be abused. Example : variable are meant to allow a programmer to keep a reference to some data, using a meaningful label. However, have you never seen a piece of code where the variable names are so obscure you can't easily guess what they point to ? I guess you did . But variable aren't removed from any language for this reason ! The feature we're discussing now is, I think, a useful tool to be used parsimoniously when the need arises, like any other. I can see several advantages to having it in the language, as opposed to building an ad-hoc function : * performance : a call is not free, even less in Python * readability : defining a function or similar moves the code away from where it is actually needed. In the case of functions called only once, such as these, this loss in readability can't be balanced against the advantages of writing a function. That would be an example a misused feature imo. I don't think big nested loops are often reusable. Globally, I think the indexed/labeled loops would be a useful tool in imperative programming. Some other code styles, such as functional programming, wouldn't need it, but isn't Python's policy to remain an open language to allow you to code however you like it best ? Think of other features, such as coroutines: they are a powerful tool, yet dangerous and easily misused (I'm mostly talking about the `yield` expression, witch can be used both to throw some output at the caller /and/ to request input). However, coroutines /did/ make it into Python, and I think they should remain there, because they can really be useful and powerful if used with care ! The same reasoning applies to our case. TL;DR : consenting adults : the blame for misusing a feature is on the user, not on the language designer, especially when the feature is opt-in Regards, Alexis

6 8

Experimental syntax proposal
by André Roberge 05 Dec '20

05 Dec '20

# Experimental Syntax Proposal I would like to propose that Python adopts a modified process before introducing significant changes of its syntax. ## Preamble Given the following file: ```python # coding: experimental-syntax from experimental-syntax import fraction_literal from experimental-syntax import decimal_literal assert 1 /3F == Fraction(1, 3) assert 0.33D == Decimal('0.33') print("simple_test.py ran successfully.") ``` This is what happens when I run it, with the standard Python interpreter: ``` $ python simple_test.py simple_test.py ran successfully. ``` In what follows, I use this as a concrete example for one of the three possible options mentioned. ## The problem Python evolves in many ways including with the addition of new modules in the standard library and with the introduction of new syntax. Before considering the addition of a module in the standard library, it is often suggested to have a version on Pypi so that users can experiment with it, which can lead to significant improvements. However, when it comes to proposed syntax changes, this is currently not possible to do, at least not in any standard way that would immediately be easily recognized as such by the wider community. ## Proposed solutions For those that agree that this is something that could be improved upon, I see at least three possible solutions. 1. Adoption of a simple convention to identify the use of non-standard syntax. This could take the form of a comment with a specific form introduced near the top of the file. This comment could be used as a search term, to identify modules or projects that make use of some custom syntax. 2. Addition of a special encoding module in the Python standard library that can be used to implement nonstandard syntax through source transformation. The short program written at the top provides an example of what this might look like. Note the comment at the top defining a special codec, which could also 3. Addition of a standard import hook in the standard library which could be used, like the special encoding example above, to implement syntactic changes. By doing searches in standard locations (Github, Gitlab, etc.) on, one could identify how many modules are making use of a given experimental syntax, giving an idea of the desirability of incorporating into Python. Imagine how different the discussion about the walrus operator would have been if people had had the opportunity to try it out on their own for a few months prior to a decision being made about adding it to Python. ## New Syntax: two approaches There are currently at least two ways in which one can write a Python program that can use some non standard syntax: * By using an import hook. This is the most powerful approach as it allows one to do changes either at the source level (prior to parsing), or at the AST level, or both. * By using a custom encoding. This only allows transformations at the source level. Both approaches currently require a two-step process. To use an import hook, one has first to load a function that sets up this import hook before importing the module that contains the experimental syntax. This makes it impossible to simply write ``` python module_with_new_syntax.py ``` and have it being executed. Import hooks, like the name implies, only apply to modules being imported - not to the first module executed by Python. With a custom encoding, it is however possible to first register a custom codec either via a site.py or usercustomize.py file. Once this is done, the above way of running a module with new syntax is possible, provided that the appropriate encoding declaration is included in the script. This is what I did with the example shown above. ### Current limitation of these two approaches Using an import hook for enabling new syntax does not work for code entered in the standard Python REPL, which limits the possibility of experimentation. While I have read that specifying the environment variable `PYTHONIOENCODING` enables the Python REPL to use a custom encoding, I have not been able to make it work on Windows and confirm that it is indeed possible to do so. However, I have been able to use a simple customized REPL that can make use of the above. Perhaps the standard REPL could be modified in a similar way. ## New suggested process Assuming one of the options mentioned above is adopted, before changes to the syntax are introduced in a Python version, they would be made available to be used either as an encoding variant or an import hook giving enough time for interested pythonistas to experiment with the new syntax, writing actual code and possibly trying out various alternatives. ## Proof of concept As a proof of concept shown at the beginning of this post, I have created two tiny modules that introduce new syntax that had been discussed many times on this list: 1. a new syntax for decimal literals 2. a new syntax for rationals Both modules have been uploaded separately to Pypi; I wanted to simulate what could happen if a proposal such as this one were adopted. To make use of these, you need to use the `experimental-syntax` codec found in my `ideas` package. To run the example as shown above, you first need to register the codec, which can be done by using either the `site.py` or `usercustomize.py` approach. I chose the latter, by setting the environment variable `PYTHONPATH` to be a path where the following `usercustomize.py` file is found. ```python from ideas import experimental_syntax_encoding print(f" --> {__file__} was executed") ``` Doing this on Windows, I found that it did not seem to work when using a virtual environment (I added the print statement to confirm that it was loaded). Here's a sample session using the code available currently on pypi. ``` C:\Users\andre\github\ideas>python simple_test.py --> C:\Users\andre\github\ideas\usercustomize.py was executed simple_test.py ran successfully. C:\Users\andre\github\ideas>python --> C:\Users\andre\github\ideas\usercustomize.py was executed Python 3.8.4 (tags/v3.8.4:dfa645a, Jul 13 2020, 16:30:28) [MSC v.1926 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from ideas import experimental_syntax_encoding >>> from ideas import console >>> console.start() Configuration values for the console: transform_source: <function transform_source at 0x00A8AE38> -------------------------------------------------- Ideas Console version 0.0.19. [Python version: 3.8.4] ~>> import simple_test simple_test.py ran successfully. ~>> from experimental-syntax import decimal_literal ~>> 3.46D Decimal('3.46') ~>> from experimental-syntax import fraction_literal ~>> 2/3F Fraction(2, 3) ~>> ``` Installation: python -m pip install ideas # will also install token-utils python -m pip install decimal-literal python -m pip install fraction-literal More information about "ideas" can be found at https://aroberge.github.io/ideas/docs/html/ It has not (yet) been updated to include anything about the `experimental_syntax_encoding` module, which I just worked on today. ## Final thoughts This proposal is really about the idea of adopting a standard process of some sort that enables users to experiment with any proposed new syntax, rather than the specific "silly" examples I have chosen to illustrate it. André Roberge P.S. If anyone knows how to make a usercustomize file run in a virtual environment on Windows, please let me know.

4 7

Making the for statement work better with nested functions
by Greg Ewing 05 Dec '20

05 Dec '20

Starting a new thread for this as suggested by Guido. On 18/11/20 7:20 pm, Guido van Rossum wrote: > On Tue, Nov 17, 2020 at 22:12 Greg Ewing <greg.ewing(a)canterbury.ac.nz > <mailto:greg.ewing@canterbury.ac.nz>> wrote: > > If there's anything I would change, it would be to have the for > statement create a new binding on each iteration, so that capturing > it with a def or lambda inside the loop works as expected. I even > came up with a way to do that while still allowing the last-bound > value to be seen afterwards, but the idea didn't gain any traction. > > I wonder if we should try that idea again? The problem may be backwards > compatibility — there’s always someone who depends on such a thing (even > if by accident, it would be a pain to debug). Maybe we could invent new > syntax to signal this behavior, though that might defeat the purpose. I think it can be made close enough to backwards compatible, although I'm not greatly opposed to new syntax if it's thought desirable. Essentially the idea is this: If the loop body contains a def or lambda that captures the loop variable, then on each iteration, a new binding is created, instead of changing an existing binding. Note that this is *not* the same as introducing a new scope. All the bindings share the same name, and from the perspective of code outside the nested function, nothing changes -- the name is still bound to the most recent value, and can still be accessed after the loop has finished. In CPython, it would be implemented by creating a new cell for each iteration, instead of changing the value of an existing cell. For other implementations, I don't know -- it would depend on how those implementations currently deal with nested functions. Do we need new syntax? My feeling is, probably not. I believe that the vast majority of existing code, perhaps all of it, would be unaffected by this change. All the existing workarounds for the current behaviour would continue to work. The only things that would break would be code that somehow relies on capturing the same instance of the loop variable every time, and I find it hard to imagine code that needs to do that. If we do want new syntax, my suggestion would be for new x in some_iterator: ... The downside of requiring special syntax is that we would still regularly get people asking why their lambdas in for statements don't do what they expect. We would have a better answer for them, but the questions wouldn't go away. -- Greg

17 76