Mailman 3 October 2017 - Python-ideas

PEP draft: context variables
by Stefan Krah Oct. 13, 2017

Oct. 13, 2017

Yury Selivanov wrote: [generators and decimal] > Specifically for decimal: I tried to find bug reports on > bugs.python.org (found not even one), questions on stackoverflow (IIRC > I didn't find anything), and used github code search and google > (again, nothing directly relevant). I also don't recall any single feature request to change the tls-context either for generators or for coroutines. Generators can quickly turn into a labyrinth, and that's not good for numerical code, … [View More]

2 1

PEP 561: Distributing Type Information V3
by Ethan Smith Oct. 10, 2017

Oct. 10, 2017

Hello, I have made some changes to my PEP on distributing type information. A summary of the changes: - Move to adding a new metadata specifier so that more packaging tools can participate - Clarify version matching between third party stubs and runtime packages. - various other fixes for clarity, readability, and removal of repetition As usual I have replicated a copy below. Cheers, Ethan PEP: 561 Title: Distributing and Packaging Type Information Author: Ethan Smith <… [View More]ethan(a)ethanhs.me> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7 Post-History: Abstract ======== PEP 484 introduced type hinting to Python, with goals of making typing gradual and easy to adopt. Currently, typing information must be distributed manually. This PEP provides a standardized means to package and distribute type information and an ordering for type checkers to resolve modules and collect this information for type checking using existing packaging architecture. Rationale ========= Currently, package authors wish to distribute code that has inline type information. However, there is no standard method to distribute packages with inline type annotations or syntax that can simultaneously be used at runtime and in type checking. Additionally, if one wished to ship typing information privately the only method would be via setting ``MYPYPATH`` or the equivalent to manually point to stubs. If the package can be released publicly, it can be added to typeshed [1]_. However, this does not scale and becomes a burden on the maintainers of typeshed. Additionally, it ties bugfixes to releases of the tool using typeshed. PEP 484 has a brief section on distributing typing information. In this section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for shipping stub files. However, manually adding a path to stub files for each third party library does not scale. The simplest approach people have taken is to add ``site-packages`` to their ``MYPYPATH``, but this causes type checkers to fail on packages that are highly dynamic (e.g. sqlalchemy and Django). Specification ============= There are several motivations and methods of supporting typing in a package. This PEP recognizes three (3) types of packages that may be created: 1. The package maintainer would like to add type information inline. 2. The package maintainer would like to add type information via stubs. 3. A third party would like to share stub files for a package, but the maintainer does not want to include them in the source of the package. This PEP aims to support these scenarios and make them simple to add to packaging and deployment. The two major parts of this specification are the packaging specifications and the resolution order for resolving module type information. The packaging spec is based on and extends PEP 345 metadata. The type checking spec is meant to replace the ``shared/typehints/pythonX.Y/`` spec of PEP 484 [2]_. New third party stub libraries are encouraged to distribute stubs via the third party packaging proposed in this PEP in place of being added to typeshed. Typeshed will remain in use, but if maintainers are found, third party stubs in typeshed are encouraged to be split into their own package. Packaging Type Information -------------------------- In order to make packaging and distributing type information as simple and easy as possible, the distribution of type information, and typed Python code is done through existing packaging frameworks. This PEP adds a new item to the ``*.distinfo/METADATA`` file to contain metadata about a package's support for typing. The new item is optional, but must have a name of ``Typed`` and have a value of either ``inline`` or ``stubs``, if present. Metadata Examples:: Typed: inline Typed: stubs Stub Only Packages '''''''''''''''''' For package maintainers wishing to ship stub files containing all of their type information, it is prefered that the ``*.pyi`` stubs are alongside the corresponding ``*.py`` files. However, the stubs may be put in a sub-folder of the Python sources, with the same name the ``*.py`` files are in. For example, the ``flyingcircus`` package would have its stubs in the folder ``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are not found in ``flyingcircus/`` the type checker may treat the subdirectory as a normal package. The normal resolution order of checking ``*.pyi`` before ``*.py`` will be maintained. Third Party Stub Packages ''''''''''''''''''''''''' Third parties seeking to distribute stub files are encouraged to contact the maintainer of the package about distribution alongside the package. If the maintainer does not wish to maintain or package stub files or type information inline, then a "third party stub package" should be created. The structure is similar, but slightly different from that of stub only packages. If the stubs are for the library ``flyingcircus`` then the package should be named ``flyingcircus-stubs`` and the stub files should be put in a sub-directory named ``flyingcircus``. This allows the stubs to be checked as if they were in a regular package. In addition, the third party stub package should indicate which version(s) of the runtime package are supported by indicating the runtime package's version(s) through the normal dependency data. For example, if there was a stub package ``flyingcircus-stubs``, it can indicate the versions of the runtime ``flyingcircus`` package supported through ``install_requires`` in distutils based tools, or the equivalent in other packaging tools. Type Checker Module Resolution Order ------------------------------------ The following is the order that type checkers supporting this PEP should resolve modules containing type information: 1. User code - the files the type checker is running on. 2. Stubs or Python source manually put in the beginning of the path. Type checkers should provide this to allow the user complete control of which stubs to use, and patch broken stubs/inline types from packages. 3. Third party stub packages - these packages can supersede the installed untyped packages. They can be found at ``pkg-stubs`` for package ``pkg``, however it is encouraged to check the package's metadata using packaging query APIs such as ``pkg_resources`` to assure that the package is meant for type checking, and is compatible with the installed version. 4. Inline packages - finally, if there is nothing overriding the installed package, and it opts into type checking. 5. Typeshed (if used) - Provides the stdlib types and several third party libraries Type checkers that check a different Python version than the version they run on must find the type information in the ``site-packages``/``dist-packages`` of that Python version. This can be queried e.g. ``pythonX.Y -c 'import site; print(site.getsitepackages())'``. It is also recommended that the type checker allow for the user to point to a particular Python binary, in case it is not in the path. To check if a package has opted into type checking, type checkers are recommended to use the ``pkg_resources`` module to query the package metadata. If the ``typed`` package metadata has ``None`` as its value, the package has not opted into type checking, and the type checker should skip that package. References ========== .. [1] Typeshed (https://github.com/python/typeshed) .. [2] PEP 484, Storing and Distributing Stub Files (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-fil…) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

2 1

Fwd: Fwd: A PEP to define basical metric which allows to guarantee minimal code quality
by Nick Coghlan Oct. 9, 2017

Oct. 9, 2017

Forwarding my reply, since Google Groups still can't get the Reply-To headers for the mailing list right, and we still don't know how to categorically prohibit posting from there. ---------- Forwarded message ---------- From: Nick Coghlan <ncoghlan(a)gmail.com> Date: 26 September 2017 at 12:51 Subject: Re: [Python-ideas] Fwd: A PEP to define basical metric which allows to guarantee minimal code quality To: Alexandre GALODE <alexandre.galode(a)gmail.com> Cc: python-ideas <python-… [View More]ideas(a)googlegroups.com> On 25 September 2017 at 21:49, <alexandre.galode(a)gmail.com> wrote: > Hi, > > Sorry from being late, i was in professional trip to Pycon FR. > > I see that the subject is divising advises. > > Reading responses, i have impression that my proposal has been saw as > mandatory, that i don't want of course. As previously said, i see this "PEP" > as an informational PEP. So it's a guideline, not a mandatory. Each > developer will have right to ignore it, as each developer can choose to > ignore PEP8 or PEP20. > > Perfect solution does not exist, i know it, but i think this "PEP" could, > partially, be a good guideline. Your question is essentially "Are python-dev prepared to offer generic code quality assessment advice to Python developers?" The answer is "No, we're not". It's not our role, and it's not a role we're the least bit interested in taking on. Just because we're the ones making the software equivalent of hammers and saws doesn't mean we're also the ones that should be drafting or signing off on people's building codes :) Python's use cases are too broad, and what's appropriate for my ad hoc script to download desktop wallpaper backgrounds, isn't going to be what's appropriate for writing an Ansible module, which in turn isn't going to be the same as what's appropriate for writing a highly scalable web service or a complex data analysis job. So the question of "What does 'good enough for my purposes' actually mean?" is something for end users to tackle for themselves, either individually or collaboratively, without seeking specific language designer endorsement of their chosen criteria. However, as mentioned earlier in the thread, it would be *entirely* appropriate for the folks participating in PyCQA to decide to either take on this work themselves, or else endorse somebody else taking it on. I'd see such an effort as being similar to the way that packaging.python.org originally started as an independent PyPA project hosted at python-packaging-user-guide.readthedocs.io, with a fair bit of content already being added before we later requested and received the python.org subdomain. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

2 1

Ternary operators in list comprehensions
by Jason H Oct. 5, 2017

Oct. 5, 2017

>>> a = [1,2,3] >>> [ x for x in a if x & 1] [1, 3] >>> [ x for x in a if x & 1 else 'even'] File "<stdin>", line 1 [ x for x in a if x & 1 else 'even'] ^ SyntaxError: invalid syntax I expected [1, 'even', 3] I would expect that the if expression would be able to provide alternative values through else. The work around blows it out to: l = [] for x in a: if x&1: l.append(x) else: l.append('… [View More]

5 4

Changes to the existing optimization levels
by Diana Clarke Oct. 5, 2017

Oct. 5, 2017

Hi folks: I was recently looking for an entry-level cpython task to work on in my spare time and plucked this off of someone's TODO list. "Make optimizations more fine-grained than just -O and -OO" There are currently three supported optimization levels (0, 1, and 2). Briefly summarized, they do the following. 0: no optimizations 1: remove assert statements and __debug__ blocks 2: remove docstrings, assert statements, and __debug__ blocks >From what I gather, their use-… [View More]case is assert statements in production code. More specifically, they want to be able to optimize away docstrings, but keep the assert statements, which currently isn't possible with the existing optimization levels. As a first baby-step, I considered just adding a new optimization level 3 that keeps asserts but continues to remove docstrings and __debug__ blocks. 3: remove docstrings and __debug__ blocks >From a command-line perspective, there is already support for additional optimization levels. That is, without making any changes, the optimization level will increase with the number of 0s provided. $ python -c "import sys; print(sys.flags.optimize)" 0 $ python -OO -c "import sys; print(sys.flags.optimize)" 2 $ python -OOOOOOO -c "import sys; print(sys.flags.optimize)" 7 And the PYTHONOPTIMIZE environment variable will happily assign something like 42 to sys.flags.optimize. $ unset PYTHONOPTIMIZE $ python -c "import sys; print(sys.flags.optimize)" 0 $ export PYTHONOPTIMIZE=2 $ python -c "import sys; print(sys.flags.optimize)" 2 $ export PYTHONOPTIMIZE=42 $ python -c "import sys; print(sys.flags.optimize)" 42 Finally, the resulting __pycache__ folder also already contains the expected bytecode files for the new optimization levels ( __init__.cpython-37.opt-42.pyc was created for optimization level 42, for example). $ tree . └── test ├── __init__.py └── __pycache__ ├── __init__.cpython-37.opt-1.pyc ├── __init__.cpython-37.opt-2.pyc ├── __init__.cpython-37.opt-42.pyc ├── __init__.cpython-37.opt-7.pyc └── __init__.cpython-37.pyc Adding optimization level 3 is an easy change to make. Here's that quick proof of concept (minus changes to the docs, etc). I've also attached that diff as 3.diff. https://github.com/dianaclarke/cpython/commit/4bd7278d87bd762b2989178e5bfed… I was initially looking for a more elegant solution that allowed you to specify exactly which optimizations you wanted, and when I floated this naive ("level 3") approach off-list to a few core developers, their feedback confirmed my hunch (too hacky). So for my second pass at this task, I started with the following two pronged approach. 1) Changed the various compile signatures to accept a set of string optimization flags rather than an int value. 2) Added a new command line option N that allows you to specify any number of individual optimization flags. For example: python -N nodebug -N noassert -N nodocstring The existing optimization options (-O and -OO) still exist in this approach, but they are mapped to the new optimization flags ("nodebug", "noassert", "nodocstring"). With the exception of the builtin complile() function, all underlying compile functions would only accept optimization flags going forward, and the builtin compile() function would accept both an integer optimize value or a set of optimization flags for backwards compatibility. You can find that work-in-progress approach here on github (also attached as N.diff). https://github.com/dianaclarke/cpython/commit/3e36cea1fc8ee6f4cdc584851e4c1… All in all, that approach is going fairly well, but there's a lot of work remaining, and that diff is already getting quite large (for my new-contributor status). Note for example, that I haven't yet tackled adding bytecode files to __pycache__ that reflect these new optimization flags. Something like: $ tree . └── test ├── __init__.py └── __pycache__ ├── __init__.cpython-37.opt-nodebug-noassert.pyc ├── __init__.cpython-37.opt-nodebug-nodocstring.pyc ├── __init__.cpython-37.opt-nodebug-noassert-nodocstring.pyc └── __init__.cpython-37.pyc I'm also not certain if the various compile signatures are even open for change (int optimize => PyObject *optimizations), or if that's a no-no. And there are still a ton of references to "-O", "-OO", "sys.flags.optimize", "Py_OptimizeFlag", "PYTHONOPTIMIZE", "optimize", etc that all need to be audited and their implications considered. I've really enjoyed this task and I'm learning a lot about the c api, but I think this is a good place to stop and solicit feedback and direction. My gut says that the amount of churn and resulting risk is too high to continue down this path, but I would love to hear thoughts from others (alternate approaches, ways to limit scope, confirmation that the existing approach is too entrenched for change, etc). Regardless, I think the following subset change could merge without any bigger picture changes, as it just adds test coverage for a case not yet covered. I can reopen that pull request once I clean up the commit message a bit (I closed it in the mean time). https://github.com/python/cpython/pull/3450/commits/bfdab955a94a7fef431548f… Thanks for your time! Cheers, --diana [View Less]

6 16

Add pop_callback to deque signature
by Michel Desmoulin Oct. 4, 2017

Oct. 4, 2017

My initial proposal would have been to add a data structure to allow something like deque, but allowing cheaper random accesses. However I realized it was a use case more suitable for an external lib. Thinking about how I would implement such a new data structure, I imagined several possibilities, one of them would be to couple a deque with a dict or a list. However, sharing data between a deque and another data structure is hard because, while you can easily hook on the element going in (… [View More]

6 9

Re: [Python-ideas] allow overriding files used for the input builtin
by Wren Turkal Oct. 2, 2017

Oct. 2, 2017

Steven, Sorry for not replying to your message due to my mailing list incompetence (configured for digest at first). I am replying to your message here: https://mail.python.org/pipermail/python-ideas/2017-September/047239.html I have not figured out the issue with readline/arrow keys on the overridden files. Consider this current iteration an RFC on the concept. I’m sure there are bugs in there. ☺ wt

1 0

Re: [Python-ideas] allow overriding files used for the input builtin
by Wren Turkal Oct. 2, 2017

Oct. 2, 2017

Paul, Sorry for not replying to your message yet, I lost it in a well-intentioned try at digest delivery. Sorry about that, I am replying to this message: https://mail.python.org/pipermail/python-ideas/2017-September/047236.html I would guess that the context manager to override stdout does it globally. One of the reasons I like the params for input is that is doesn’t have to override the value globally by setting the module variable, which would affect other concurrent uses of input(). Thoughts? Thanks, wt

1 0

Re: [Python-ideas] allow overriding files used for the input builtin
by Wren Turkal Oct. 2, 2017

Oct. 2, 2017

Adam, I just realized that I had not responded to you. I’m sorry for not replying to the message as this was one of the messages that came in digest form. I am referring to your message here: https://mail.python.org/pipermail/python-ideas/2017-September/047238.html I looked over your comment about how input() should be like print(), but there is an issue that I see with it. It seems print() allows overriding the file used for the output. It seems like input() should allow that for symmetry. … [View More]

1 0