Mailman 3 January 2021 - Python-ideas

Making TYPE_CHECKING builtin.
by Inada Naoki Jan. 19, 2021

Jan. 19, 2021

Hi, all. I want to write type hints without worrying about runtime overhead. Current best practice is: ``` from __future__ import annotations import typing if typing.TYPE_CHECKING: import xxx # modules used only in type hints. ``` But it would be nice if I can avoid importing even "typing" module. How about adding TYPE_CHECKING builtin that is False? ``` from __future__ import annotations if TYPE_CHECKING: from typing import Any, Optional # We can use Any, Optional, etc here. ``` I wonder if we can make TYPE_CHECKING constant like True, False, and None. But it will break existing `from typing import TYPE_CHECKING` codes. Regards, -- Inada Naoki <songofacandy(a)gmail.com>

4 4

Python with braces formal proposal?
by Paul Sokolovsky Jan. 19, 2021

Jan. 19, 2021

Hello, There're tons of projects which introduce alternative braces (i.e. C-like) syntax for Python. Most of them are however not properly documented, and definitely not spec'ed for what they do. I wonder, does anyone here remember more or less formal proposal for braces syntax? A "minimum viable product" criteria would be support for lossless indent -> braces -> indent syntax roundtripping. -- Best regards, Paul mailto:pmiscml@gmail.com

11 34

pathlib.Path.makedirs
by Antonio Cavallo Jan. 18, 2021

Jan. 18, 2021

Hi, I've found myself typing too many time this: pathlib.Path("some-dir-name").mkdir(parent=True, exist_ok=True) Wouldn't be better to have a pathlib.Path.makedirs with parent/exist_ok set to True by default? Thanks

3 2

pathlib enhancements
by Todd Jan. 18, 2021

Jan. 18, 2021

I know enhancements to pathlib gets brought up occasionally, but it doesn't look like anyone has been willing to take the initiative and see things through to completion. I am willing to keep the ball rolling here and even implement these myself. I have some suggestions and I would like to discuss them. I don't think any of them are significant enough to require a pep. These can be split it into independent threads if anyone prefers. 1. copy The big one people keep bringing up that I strongly agree on is a "copy" method. This is really the only common file manipulation task that currently isn't possible. You can make files, read them, move them, delete them, create directories, even do less common operations like change owners or create symlinks or hard links. A common objection is that pathlib doesn't work on multiple paths. But that isn't the case. There are a ton of methods that do that, including: * symlink_to * link_to * rename * replace * glob * rglob * iterdir * is_relative_to * relative_to * samefile I think this is really the only common file operation that someone would need to switch to a different module to do, and it seems pretty strange to me to be able to make symbolic or hard links to a file but not straight up copy one. 2. recursive remove This could be a "recursive" option to "rmdir" or a "rmtree" method (I prefer the option). The main reason for this is symmetry. It is possible to create a tree of folders (using "mkdir(parents=True)"), but once you do that you cannot remove it again in a straightforward way. 3. newLine for write_text This is the only relevant option that "Path.open" has but "Path.write_text" doesn't, and is a serious omission when dealing with multiple operating systems. 4. uid and gid You can get the owner and group name of a file (with the "owner" and "group" methods), but there is no easy way to get the corresponding number. 5. Stem with no suffixes The stem property only takes off the last suffix, but even in the example given ('my/library.tar.gz') it isn't really useful because the suffix has two parts ('.tar' and '.gz'). I suggest another property, probably called "rootstem" or "basestem", that takes off all the suffixes, using the same logic as the "suffixes" property. This is another symmetry issue: it is possible to extract all the suffixes, but not remove them. 6. with_suffixes Equivalent to with_suffix, but replacing all suffixes. Again, this is a symmetry issue. It is hard to manipulate all the suffixes right now, as the example show. You can add them or extract them, but not change them without doing several steps. 7. exist_ok for is_* methods Currently all the is_* methods (such as is_file) return False if the file doesn't exist or if it is a broken symlink. This can be dangerous, since it is not trivially easy to tell if you are dealing with the wrong type of file vs. a missing file. And it isn't obvious behavior just from the method name. I suggest adding an "exist_ok" argument to all of these, with the default being "True" for backwards-compatibility. This argument name is already in use elsewhere in pathlib. If this is False and the file is not present, a "FileNotFoundError" is raised.

10 21

Consider having numbers.Real provide __complex__?
by Neil Girdhar Jan. 18, 2021

Jan. 18, 2021

I've been following along various issues in mypy regarding challenges with getting the ABCs in numbers (https://docs.python.org/3/library/numbers.html) working. Currently, there's a lot of clever logic in the functions int, float, and complex. Currently, inheriting from Real doesn't give you __complex__. This is because the complex function doesn't need it—it falls back to __float__. I think it would be nice if it did for the purpose of type annotations. Best, Neil

3 3

Best practices of class "reopening" a-la Ruby?
by Paul Sokolovsky Jan. 17, 2021

Jan. 17, 2021

Hello, Ruby has following feature. Suppose the existing class "Cls" is scope (either defined before or imported from some module), then the code like: class Cls def mixin_method(args) ... end end Will "reopen" (Ruby term) that class and will add a new method "mixin_method" to it. Syntactically, that's complete madness - the same syntax is used to both define the class initially and augment it afterwards, and that's expectedly leads to user confusion, e.g. https://stackoverflow.com/questions/4464629/ruby-rails-reopening-vs-overwri… Semantically, Python can achieve the same with "imperative" syntax like: def mixin_method(self, args): ... Cls.mixin_method = mixin_method The question then: what are the best practices in *declarative* syntax to achieve the same effect in Python? (but of course, unlike Ruby, there should be explicit syntactic marker that we augment existing class, not redefine it). One of usecases should be clear from the example above - it's an easy way to add a mixin to the class, without relying on multiple inheritance (which has many its own problems). It's also emerging language-flexibility/expressivity best practice to be able to define interfaces on classes (types) post-factum. Such interfaces are called different buzzwords in different languages, e.g. Haskell calls them "typeclasses" and Rust calls them "traits". The question is again how do we syntactically encode it in Python (with a hint that the matter is rather similar to post-factum mixing-in). -- Best regards, Paul mailto:pmiscml@gmail.com

5 6

PEP Draft: Build Dependency Specification for Manylinux Wheels
by Chris Antonellis Jan. 16, 2021

Jan. 16, 2021

Hello- This is a draft of a future PEP regarding capturing wheel build dependencies in pyproject.toml. I am looking for any critique or input prior to creating a PR against the PEP repository, Thank you! Previous discussion can be seen here: * https://mail.python.org/archives/list/python-ideas@python.org/message/EQKE5… * https://mail.python.org/archives/list/python-ideas@python.org/message/CGUVG…) PEP: 9999 Title: Build Dependency Specification for Manylinux Wheels Author: Christopher Antonellis <cantonellis(a)wayfair.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 22-Dec-2020 Post-History: 21-Dec-2020 Abstract ======== This document specifies how Python software packages should specify what actions are required to modify a standard manylinux environment to correctly build and bundle the package into a manylinux wheel. Motivation ========== Python wheels with compiled extensions may link to system libraries, requiring that the system libraries be available on the host systems (the system the wheel is installed on) to operate correctly. The manylinux project solves this problem through the auditwheel package, which identifies system libraries at compile time and bundles required libraries with the python wheel. This allows the python library, its compiled extensions, and any required system libraries to be installed on a host system without having to install the system library directly. In an ideal world, all package authors would make use of the manylinux project and all python packages that require system libraries would provide compiled, bundled distributions on PyPI. However, this is not the case, and many packages do not. There are valid instances where an author may not provide a manylinux wheel by choice: for example when a required system library cannot be bundled due to licensing. **However, there are packages on PyPI that do not provide a wheel when one could be provided.** This means that these libraries require local compilation prior to use, resulting in multiple negative side effects for the end user: - Required system libraries are not easily determined. They must be gleaned from project-specific documentation with no standardized format. - Compiling extensions can take a long time, adding additional expense to rebuilding environments. - Compiling a wheel that requires system libraries is non-trivial; it is easy to mismatch system library and python library version and be presented with cryptic error messages. Some authors do provide manylinux wheels on PyPI by making use of the manylinux project. However, the manylinux project does not provide a standardized way to capture environment setup data. This results in package authors keeping this data in project documentation or sometimes not recording it at all. Rationale ========= This PEP proposes a common format for the data required to correctly setup manylinux environments to compile a wheel with required system libraries. This concept is borrowed from package managers such as RPM which make use of a .SPEC file to capture this data. This data can be used in a manylinux container to set up the environment prior to compiling, resulting in a valid manylinux wheel. This data can be standardized to allow for automated building of manylinux wheels. Standardization of this data will allow package consumers to more easily contribute to building manylinux wheels when an existing distribution is lacking or not available. Specification ============= The data will be located in the `pyproject.toml` file of a python project, in a main table titled `manylinux_build_specification`. The data will be grouped in sub tables titled per the manylinux version they are targeting, ex `manylinux2014`. **pyproject.toml** .. code-block:: toml [manylinux_build_specification.manylinux2014] extra_base_system_repositories = ["http://foo.com/packages/"] system_dependencies = ["foo-1.0.0", "bar-1.0.0"] python_dependencies = ["foo==1.0.0", "bar==1.0.0"] environment_variables = ["FOO=BAR"] steps = [ "./scripts/build_and_upload.sh --my_option" ] All actions will be performed within a manylinux image. Given that the manylinux project uses CentOS as the base linux flavor, we can assume the following: * Use of `yum` for system package management * Python versions available in `/opt/python/` `extra_base_system_repositories` -------------------------------- Repositories to add to `yum` prior to installing system dependencies Additional repositories from which to download system dependencies. This allows access to builds of system libraries with the most up to date patches etc. `system_dependencies` --------------------- System dependencies to install with `yum` prior to building. Entries are expected to be in `yum` `name-version` format. `environment_variables` ----------------------- Environment variables to set prior to building. `python_dependencies` --------------------- Python libraries to install with `pip` prior to building. Will be installed for each version of python available in `/opt/python/`. `steps` ------- Steps to be executed sequentially with bash. The entire build process can be captured here, or this can be a call to a separate script. How to Teach This ================= This will be taught through examples and documentation provided in a reference implementation. Reference Implementation ======================== A reference implementation is currently in development. This will include the following: 1. The data spec will be defined in an example python package. 2. A python package will be created that consumes the data spec and sets up a manylinux container appropriately. 3. A manylinux docker image will be created that runs the python package that consumes the data spec prior to building a wheel. References ========== - `PEP 508 -- Dependency specification for Python Software Packages <https://www.python.org/dev/peps/pep-0508/>`_ - `PEP 518 -- Specifying Minimum Build System Requirements for Python Projects <https://www.python.org/dev/peps/pep-0518/>`_ - `PEP 571 -- The manylinux2010 Platform Tag <https://www.python.org/dev/peps/pep-0571/>`_ - `PEP 599 -- The manylinux2014 Platform Tag <https://www.python.org/dev/peps/pep-0599/>`_ - `PEP 600 -- Future ‘manylinux’ Platform Tags for Portable Linux Built Distributions <https://www.python.org/dev/peps/pep-0600/>`_ - `PEP 631 -- Dependency specification in pyproject.toml based on PEP 508 <https://www.python.org/dev/peps/pep-0631/>`_ - `RPM Packaging Guide: What is a SPEC File? <https://rpm-packaging-guide.github.io/#what-is-a-spec-file>`_ Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

2 2

Off-topic: What costs NaN pounds for a null amount?
by Jonathan Fine Jan. 14, 2021

Jan. 14, 2021

Hi There's interest here in arithmetic operations on NaN . I've just seen a product listed as costing NaN pounds to buy a null amount. That was written as £NaN/null. The bargain item is Glade Shake & Vacuum Citrus, and you can see it at https://www.tesco.com/groceries/en-GB/products/253732570 Searching on this site for 'NaN' brings up many entries for https://en.wikipedia.org/wiki/Naan bread. with best regards Jonathan

2 1

Re: Additional LRU cache introspection facilities
by Raymond Hettinger Jan. 14, 2021

Jan. 14, 2021

> I propose a method: > ... > returns a dictionary {arg: value} representing the cache. > It wouldn't be the cache itself, just a shallow copy > of the cache data I recommend against going down this path. It exposes (and potentially locks in) implementation details such as how we distinguish positional arguments, keyword arguments, and type information (something that has changed more than once). Also, a shallow copy still leaves plenty of room for meddling with the contents of the keys, potentially breaking the integrity of the cache. Another concern is that we've worked hard to remove potential deadlocks from the lru_cache. Hanging on a lock while copying the whole cache complicates our efforts and risks breaking it as users exploit the new feature in unpredictable ways. FWIW, OrderedDict provides methods that make it easy to roll your own variants of the lru_cache(). It would better to do that than to complexify the base implementation in ways that I think we would regret. Raymond

2 1

Renaming variable 'mod' to 'readline' in pymain_import_readline()
by Jero Bado Jan. 13, 2021

Jan. 13, 2021

Hello, In Modules/main.c, wherever PyImport_ImportModule() is use, the variables (runpy and sys) are named as follows: runpy = PyImport_ImportModule("runpy"); // line 265 sys = PyImport_ImportModule("sys"); // line 444 Can we also rename 'mod' to 'readline' for consistency in line 215? PyObject *mod = PyImport_ImportModule("readline"); // line 215 if (mod == NULL) { PyErr_Clear(); } else { Py_DECREF(mod); } Suggested changes: PyObject *readline = PyImport_ImportModule("readline"); if (readline == NULL) { PyErr_Clear(); } else { Py_DECREF(readline); } Let me know your thoughts. New guy here, Gerol

1 0