Python-Dev
Threads by month
- ----- 2025 -----
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2008 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2007 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2006 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2005 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2004 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2003 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2002 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2001 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2000 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 1999 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
May 2020
- 57 participants
- 39 discussions
Hello there,
I would like to discuss a proposal regarding one aspect which AFAIK is
currently missing from cPython's test suite: the ability to detect memory
leaks of functions implemented in the C extension modules.
In psutil I use a test class/framework which calls a function many times,
and fails if the process memory increased after doing so. I do this in
order to quickly detect missing free() or Py_DECREF calls in the C code,
but I suppose there may be other use cases. Here's the class:
…
[View More]https://github.com/giampaolo/psutil/blob/913d4b1d6dcce88dea6ef9382b93883a04…
Detecting a memory leak is no easy task, and that's because the process
memory fluctuates. Sometimes it may increase (or even decrease!) even if
there's no leak, I suppose because of how the OS handles memory, the
Python's garbage collector, the fact that RSS is an approximation, and who
knows what else. In order to compensate fluctuations I did the following:
in case of failure (mem > 0 after calling fun() N times) I retry the test
for up to 5 times, increasing N (repetitions) each time, so I consider the
test a failure only if the memory keeps increasing across all runs. So for
instance, here's a legitimate failure:
psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_disk_partitions
...
Run #1: extra-mem=696.0K, per-call=3.5K, calls=200
Run #2: extra-mem=1.4M, per-call=3.5K, calls=400
Run #3: extra-mem=2.1M, per-call=3.5K, calls=600
Run #4: extra-mem=2.7M, per-call=3.5K, calls=800
Run #5: extra-mem=3.4M, per-call=3.5K, calls=1000
FAIL
If, on the other hand, the memory increased on one run (say 200 calls) but
decreased on the next run (say 400 calls), then it clearly means it's a
false positive, because memory consumption may be > 0 on the second run,
but if it's lower than the previous run with less repetitions, then it
cannot possibly represent a leak (just a fluctuation):
psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_net_connections
...
Run #1: extra-mem=568.0K, per-call=2.8K, calls=200
Run #2: extra-mem=24.0K, per-call=61.4B, calls=400
OK
This is the best I could come up with as a simple leak detection mechanism
to integrate with CI services, and keep more advanced tools like Valgrind
out of the picture (I just wanted to know if there's a leak, not to debug
the leak itself). In addition, since psutil is able to get the number of
fds (UNIX) and handles (Windows) opened by a process, I also run a separate
set of tests to make sure I didn't forget to call close(2) or CloseHandle()
in C.
Would something like this make sense to have in cPython? Here's a quick PoC
I put together just to show how this thing would look like in practice:
https://github.com/giampaolo/cpython/pull/2/files
A proper work in terms of API coverage would result being quite huge (test
all C modules), and ideally should also include cases where functions raise
an exception when being fed with an improper input. The biggest stopper
here is, of course, psutil, since it's a third party dep, but before
getting to that I wanted to see how this idea is perceived in general.
Cheers,
--
Giampaolo - http://grodola.blogspot.com
[View Less]
5
7

inspect.getdoc and (Not) returning type/superclass docstrings in 3.9
by Matthias Bussonnier May 12, 2020
by Matthias Bussonnier May 12, 2020
May 12, 2020
Hi All,
# Too long didn't read:
In 3.9 inspect.getdoc(instance) behavior was changed and does not return the documentation of type(instance) or it's superclass(es) – I think this is a problematic change of for some project and interactive use to get info on objects that get rarely directly constructed by users. For example pandas dataframe obtained via `pandas.read_csv(filepath)`.
I'd like to ask for reconsideration, and that change of behavior are better suited in a new function; …
[View More]potentially deprecating the old one.
# Longer version
In https://bugs.python.org/issue40257 attempts are made to improve the output of `pydoc`, it particular it is difficult to have fine grained logic depending on where the documentation comes from (instance, class , superclass, etc..). Which sometime can lead to nonsensical help.
The following are given as examples:
> inspect.getdoc(1) returns the same as inspect.getdoc(int)
or
>>> import wave
>>> help(wave.Error)
Help on class Error in module wave:
class Error(builtins.Exception)
| Common base class for all non-exit exceptions.
|
| Method resolution order:
...
In 3.9 the behavior of `inspect.getdoc()` has been changed to be way more restrictive in what it returns, often returning None where it used to return docstrings.
I agree with the end goal of having more controllable way of finding where the documentation/docstrings is coming from and avoiding incorrect docs in pydoc and help,
though I find that change of behavior of `getdocs()` might not be the right approach.
I'm quite worried many project rely on current behavior of `getdocs()` – at least IPython/Jupyter does to provide user with help/superhelp accessible via obj? and obj??.
I would also argue that inaccurate help is also often better than no help.
With current state on Python 3.9, a few things like asking for help on a pandas dataframe instance will currently loose informations.
>>> import pandas as pd
>>> from inspect import getdoc
>>> df = pd.read_csv('mydata.csv')
>>> print(getdoc(df))
None
I'm taking the example of pandas as this is typically the kind of objects you don't construct directly, and get via for example `read_csv()`, or that another API/Package return to you.
I haven't been able to confirm yet exactly how this affects sphinx rendering of docs, and how other IDEs provide help (Spyder, Pycharm...), or other projects that use `getdocs()`.
I've found mentions of `getdocs()` in numpy, scipy, jedi, matplotlib ... as well (sphinx extension and various dynamic docs), and working on building them on 3.9 to check the effect.
In general though the effect of `getdoc()` rarely seem to be tested as they will directly be user facing is my feeling – I was lucky to catch it in IPython/Jupyter as the failing test was unrelated and indirectly relying on the exact output of a subprocess.
From the IPython/Jupyter perspective I would prefer to keep current behavior of `inspect.getdocs()` potentially deprecating it if you wish to, and provide an alternative that have a behavior of your choosing. Dealing with functions with slightly chaging behavior across Python version is not the best experience, and this would let the ecosystem get some chance to adapt. Updated project get rarely released in synchrony with new Python versions.
Your thoughts on this issue are welcome, thanks for all your work on core python, and I'll support any decision that get made.
--
Matthias
[View Less]
1
0
Hi,
PEP 543, the new TLS api for Python, was published several years ago as a way to a new library unencumbered by the legacy issues around the current ssl library.
In the meantime, no actual implementation has appeared. The closest appears to be https://github.com/Synss/python-mbedtls/tree/0.13.0 or https://github.com/python-hyper/pep543 both of which haven't seen much development in years not to mention that neither one has been accepted into the stdlib.
The current ssl library could use …
[View More]some improvements to make it easier to use and to expose some more of openssl's functionality (e.g. https://github.com/python/cpython/pull/2449) but it appears that changes have been shelved in favor of PEP 543.
Is there any hope for further development on ssl? Is PEP 543 abandoned?
Nimish
[View Less]
1
0
Hey all,
The feature freeze is coming up on us fast, and the PEP 615
implementation is more or less ready to be integrated into the standard
library (may need one or two little tweaks, but it's well past the
"minimum viable product" stage).
Normally I'd wait longer for someone to volunteer for the task of
reviewing, but given the somewhat tight timeline and the fact that the
code and tests alone (not including the documentation) are 6000 lines, I
figured it's better to give people a head's up …
[View More]that I'm looking for
reviewers. I've already had a few reviews on this code when it was first
merged to the reference implementation, but there's also a decent chunk
of otherwise unreviewed code.
- The implementation and tests are in PR #19909:
https://github.com/python/cpython/pull/19909
- The documentation and What's New entry is in #20006:
https://github.com/python/cpython/pull/20006
There's also one other feature that I did not originally include in the
PEP but which I think is a reasonable feature request that we are likely
to get, which is a way to list all the time zones available on the
system. The reference implementation includes an implementation for that
feature in the property test suite, and it would be easy for me to port
it over; I'll do that if there are no objections before the feature
freeze, I've opened this BPO issue to track the discussion:
https://bugs.python.org/issue40536
Thanks!
Paul
[View Less]
1
0
ACTIVITY SUMMARY (2020-05-01 - 2020-05-08)
Python tracker at https://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open 7459 (+32)
closed 44857 (+64)
total 52316 (+96)
Open issues with patches: 2996
Issues opened (66)
==================
#36543: Remove old-deprecated ElementTree features (part 2)
https://bugs.python.org/issue36543 reopened by scoder
#40028: Math module method …
[View More]to find prime factors for non-negative int
https://bugs.python.org/issue40028 reopened by mark.dickinson
#40228: Make setting line number in frame more robust.
https://bugs.python.org/issue40228 reopened by vstinner
#40471: Grammar typo in issubclass docstring
https://bugs.python.org/issue40471 opened by alexpovel
#40474: Code coverage report not entirely accurate
https://bugs.python.org/issue40474 opened by Lewis Ball
#40477: Launcher on Catalina
https://bugs.python.org/issue40477 opened by Auerhahn
#40478: allow finding nmake.exe executable in MSVCCompiler
https://bugs.python.org/issue40478 opened by japm48
#40479: Port _hashlib to OpenSSL 3.0.0
https://bugs.python.org/issue40479 opened by christian.heimes
#40481: Add include and exclude filters to zipapp cli
https://bugs.python.org/issue40481 opened by Jarosław Wygoda
#40482: _hashlib: register Python names as OpenSSL aliases
https://bugs.python.org/issue40482 opened by christian.heimes
#40483: Implementing a verifier function to verify integrity of AST no
https://bugs.python.org/issue40483 opened by BTaskaya
#40484: Document existing compiler flags under ast
https://bugs.python.org/issue40484 opened by BTaskaya
#40485: Provide an abstraction for a select-able Event
https://bugs.python.org/issue40485 opened by paravoid
#40486: pathlib's iterdir doesn't specify what happens if directory co
https://bugs.python.org/issue40486 opened by facundobatista
#40487: Unexpected exception handler behavior in Jupyter when returnin
https://bugs.python.org/issue40487 opened by jeanmonet
#40492: -m cProfile -o f.pstats with a script that does chdir() writes
https://bugs.python.org/issue40492 opened by Anthony Sottile
#40494: collections.abc.Callable and type variables
https://bugs.python.org/issue40494 opened by serhiy.storchaka
#40495: compileall: option to hardlink duplicate optimization levels b
https://bugs.python.org/issue40495 opened by frenzy
#40497: subprocess.check_output() accept the check keyword argument
https://bugs.python.org/issue40497 opened by remi.lapeyre
#40498: Holding spacebar on button widget permanently makes it SUNKEN
https://bugs.python.org/issue40498 opened by PythonAmateur742
#40500: test_multiprocessing_fork leaks processes on PPC64LE RHEL8 LTO
https://bugs.python.org/issue40500 opened by vstinner
#40501: Deprecate and remove ctypes usage in uuid
https://bugs.python.org/issue40501 opened by steve.dower
#40502: PyNode_New() does not initialize n->n_col_offset
https://bugs.python.org/issue40502 opened by Tomasz Pytel
#40503: PEP 615: Add zoneinfo module
https://bugs.python.org/issue40503 opened by p-ganssle
#40505: getpath.c doesn't know about lib64
https://bugs.python.org/issue40505 opened by lemburg
#40506: add support for os.Pathlike filenames in zipfile.ZipFile.write
https://bugs.python.org/issue40506 opened by d.ragusa
#40507: FileNotFound error raised by os.exec* doesn't contain filename
https://bugs.python.org/issue40507 opened by russelldavis
#40509: In argparse, allow REMAINDER(...) arguments in a mutually excl
https://bugs.python.org/issue40509 opened by Shani Armon
#40510: [regression] ZipFile fails to round trip on some files
https://bugs.python.org/issue40510 opened by DragonSA
#40511: IDLE yellow hint frame blinks when entering () in strings in f
https://bugs.python.org/issue40511 opened by wyz23x2
#40512: Meta issue: per-interpreter GIL
https://bugs.python.org/issue40512 opened by vstinner
#40513: Move _PyRuntimeState.ceval to PyInterpreterState
https://bugs.python.org/issue40513 opened by vstinner
#40514: Add --experimental-isolated-subinterpreters build option
https://bugs.python.org/issue40514 opened by vstinner
#40515: test_ssl.py hangs with SSL 1.1 built with no threads
https://bugs.python.org/issue40515 opened by mig28suarez
#40516: GCC 9 compiler warnings on MacOS Catalina
https://bugs.python.org/issue40516 opened by remi.lapeyre
#40519: Preserve AttributeError exception context in __getattr__
https://bugs.python.org/issue40519 opened by Arusekk
#40521: Make tuple, dict, frame free lists, unicode interned strings,
https://bugs.python.org/issue40521 opened by vstinner
#40522: Subinterpreters: get the current Python interpreter state from
https://bugs.python.org/issue40522 opened by vstinner
#40525: zipapps execute symlinks as if they are code
https://bugs.python.org/issue40525 opened by Anthony Sottile
#40528: Improve / Clear ASDL generator
https://bugs.python.org/issue40528 opened by BTaskaya
#40529: Auto Completions with case insensitive
https://bugs.python.org/issue40529 opened by mkasula
#40530: distutils/cygwinccompiler.py doesn't support recent msvc versi
https://bugs.python.org/issue40530 opened by Vadim Godunko
#40533: Subinterpreters: don't share Python objects between interprete
https://bugs.python.org/issue40533 opened by vstinner
#40534: ShUtil doc deficiencies
https://bugs.python.org/issue40534 opened by T UA
#40536: Addition of a "list of available time zones" function to zonei
https://bugs.python.org/issue40536 opened by p-ganssle
#40540: inconstent stdin buffering/seeking behaviour
https://bugs.python.org/issue40540 opened by PeterJCLaw
#40543: Tamil locale is using outdated encoding
https://bugs.python.org/issue40543 opened by Muthu A
#40544: Formatter exception when using logging.config.fileConfig
https://bugs.python.org/issue40544 opened by mirii1994
#40545: Expose _PyErr_GetTopmostException
https://bugs.python.org/issue40545 opened by jd
#40546: Inconsistencies between PEG parser and traceback SyntaxErrors
https://bugs.python.org/issue40546 opened by lys.nikolaou
#40547: 2to3 raise can silently remove code from old 2.4 string except
https://bugs.python.org/issue40547 opened by Rémy Oudompheng
#40548: Always run GitHub action jobs, even on documentation-only jobs
https://bugs.python.org/issue40548 opened by vstinner
#40549: Convert posixmodule.c to multiphase initialization (PEP 489)
https://bugs.python.org/issue40549 opened by vstinner
#40550: Popen.terminate fails with ProcessLookupError under certain co
https://bugs.python.org/issue40550 opened by Alexander Overvoorde
#40551: PRs should be rebased on top of master before running the buil
https://bugs.python.org/issue40551 opened by FFY00
#40552: Enhance for loop and copy example in tutorial
https://bugs.python.org/issue40552 opened by mdk
#40553: Python 3.8.2 Mac freezing/not responding when saving new progr
https://bugs.python.org/issue40553 opened by Zain
#40554: Add escape to the glossary?
https://bugs.python.org/issue40554 opened by mdk
#40556: test__xxsubinterpreters leaked [1486, 1484, 1484, 1484] refere
https://bugs.python.org/issue40556 opened by pablogsal
#40557: Move captured_stdin(), captured_stdout(), and captured_stderr(
https://bugs.python.org/issue40557 opened by felixxm
#40558: update CONTRIBUTING.rst docs
https://bugs.python.org/issue40558 opened by shihai1991
#40561: Provide docstrings for public-facing webbrowser functions
https://bugs.python.org/issue40561 opened by bsolomon1124
#40562: SEO: differentiate between Python 2 and Python 3 docs on Googl
https://bugs.python.org/issue40562 opened by simonw
#40563: Support pathlike objects on dbm/shelve
https://bugs.python.org/issue40563 opened by BTaskaya
#40564: Using zipfile.Path with several files prematurely closes zip
https://bugs.python.org/issue40564 opened by bustawin
#40566: Apply PEP 573 to abc module
https://bugs.python.org/issue40566 opened by corona10
Most recent 15 issues with no replies (15)
==========================================
#40566: Apply PEP 573 to abc module
https://bugs.python.org/issue40566
#40564: Using zipfile.Path with several files prematurely closes zip
https://bugs.python.org/issue40564
#40561: Provide docstrings for public-facing webbrowser functions
https://bugs.python.org/issue40561
#40556: test__xxsubinterpreters leaked [1486, 1484, 1484, 1484] refere
https://bugs.python.org/issue40556
#40551: PRs should be rebased on top of master before running the buil
https://bugs.python.org/issue40551
#40549: Convert posixmodule.c to multiphase initialization (PEP 489)
https://bugs.python.org/issue40549
#40547: 2to3 raise can silently remove code from old 2.4 string except
https://bugs.python.org/issue40547
#40544: Formatter exception when using logging.config.fileConfig
https://bugs.python.org/issue40544
#40543: Tamil locale is using outdated encoding
https://bugs.python.org/issue40543
#40540: inconstent stdin buffering/seeking behaviour
https://bugs.python.org/issue40540
#40536: Addition of a "list of available time zones" function to zonei
https://bugs.python.org/issue40536
#40534: ShUtil doc deficiencies
https://bugs.python.org/issue40534
#40530: distutils/cygwinccompiler.py doesn't support recent msvc versi
https://bugs.python.org/issue40530
#40510: [regression] ZipFile fails to round trip on some files
https://bugs.python.org/issue40510
#40507: FileNotFound error raised by os.exec* doesn't contain filename
https://bugs.python.org/issue40507
Most recent 15 issues waiting for review (15)
=============================================
#40566: Apply PEP 573 to abc module
https://bugs.python.org/issue40566
#40561: Provide docstrings for public-facing webbrowser functions
https://bugs.python.org/issue40561
#40558: update CONTRIBUTING.rst docs
https://bugs.python.org/issue40558
#40552: Enhance for loop and copy example in tutorial
https://bugs.python.org/issue40552
#40549: Convert posixmodule.c to multiphase initialization (PEP 489)
https://bugs.python.org/issue40549
#40548: Always run GitHub action jobs, even on documentation-only jobs
https://bugs.python.org/issue40548
#40545: Expose _PyErr_GetTopmostException
https://bugs.python.org/issue40545
#40544: Formatter exception when using logging.config.fileConfig
https://bugs.python.org/issue40544
#40533: Subinterpreters: don't share Python objects between interprete
https://bugs.python.org/issue40533
#40529: Auto Completions with case insensitive
https://bugs.python.org/issue40529
#40528: Improve / Clear ASDL generator
https://bugs.python.org/issue40528
#40525: zipapps execute symlinks as if they are code
https://bugs.python.org/issue40525
#40522: Subinterpreters: get the current Python interpreter state from
https://bugs.python.org/issue40522
#40521: Make tuple, dict, frame free lists, unicode interned strings,
https://bugs.python.org/issue40521
#40519: Preserve AttributeError exception context in __getattr__
https://bugs.python.org/issue40519
Top 10 most discussed issues (10)
=================================
#40513: Move _PyRuntimeState.ceval to PyInterpreterState
https://bugs.python.org/issue40513 15 msgs
#40334: PEP 617: new PEG-based parser
https://bugs.python.org/issue40334 13 msgs
#36543: Remove old-deprecated ElementTree features (part 2)
https://bugs.python.org/issue36543 12 msgs
#40528: Improve / Clear ASDL generator
https://bugs.python.org/issue40528 11 msgs
#40028: Math module method to find prime factors for non-negative int
https://bugs.python.org/issue40028 10 msgs
#40426: Unable to use lowercase hexadecimal digits for percent encodin
https://bugs.python.org/issue40426 10 msgs
#40512: Meta issue: per-interpreter GIL
https://bugs.python.org/issue40512 9 msgs
#40545: Expose _PyErr_GetTopmostException
https://bugs.python.org/issue40545 9 msgs
#40458: test_bad_getattr crashes on APPX test
https://bugs.python.org/issue40458 7 msgs
#40546: Inconsistencies between PEG parser and traceback SyntaxErrors
https://bugs.python.org/issue40546 7 msgs
Issues closed (64)
==================
#2380: Raise a Py3K warning for catching nested tuples with non-BaseE
https://bugs.python.org/issue2380 closed by zach.ware
#5879: multiprocessing example "pool of http servers " fails on windo
https://bugs.python.org/issue5879 closed by terry.reedy
#17848: can't compile with clang and build a shared lib due to libffi
https://bugs.python.org/issue17848 closed by ned.deily
#18499: mingw: setup _ctypes module with system libffi
https://bugs.python.org/issue18499 closed by ned.deily
#25413: ctypes (libffi) fails to compile on Solaris X86
https://bugs.python.org/issue25413 closed by ned.deily
#26192: python3 k1om dissociation permanence: libffi
https://bugs.python.org/issue26192 closed by ned.deily
#27133: python 3.5.1 will not compile because libffi module uses wrong
https://bugs.python.org/issue27133 closed by ned.deily
#31710: setup.py: _ctypes won't get built when system ffi is only in $
https://bugs.python.org/issue31710 closed by ned.deily
#34823: libffi detection doesn’t work in my setup
https://bugs.python.org/issue34823 closed by ned.deily
#35061: Specify libffi.so soname for ctypes
https://bugs.python.org/issue35061 closed by ned.deily
#36944: Add support for ARM64 to libffi
https://bugs.python.org/issue36944 closed by ned.deily
#39470: Indicate that os.makedirs is equivalent to Path.mkdir
https://bugs.python.org/issue39470 closed by nanjekyejoannah
#39691: Allow passing Pathlike objects to io.open_code
https://bugs.python.org/issue39691 closed by steve.dower
#40135: multiprocessing: test_shared_memory_across_processes() cannot
https://bugs.python.org/issue40135 closed by vstinner
#40178: Convert the remaining os funtions to Argument Clinic
https://bugs.python.org/issue40178 closed by serhiy.storchaka
#40316: Add zero function to time, datetime, which acts as the use cas
https://bugs.python.org/issue40316 closed by rhettinger
#40355: The ast module fails to reject certain malformed nodes
https://bugs.python.org/issue40355 closed by BTaskaya
#40393: Auto-response from Python Help points to Python 2 reference
https://bugs.python.org/issue40393 closed by terry.reedy
#40398: typing.get_args(Callable) fails
https://bugs.python.org/issue40398 closed by serhiy.storchaka
#40408: GenericAlias does not support nested type variables
https://bugs.python.org/issue40408 closed by serhiy.storchaka
#40414: Incorrect mouse and keyboard mapping
https://bugs.python.org/issue40414 closed by terry.reedy
#40416: Calling TextIOWrapper.tell() in the middle of reading a gb2312
https://bugs.python.org/issue40416 closed by terry.reedy
#40417: PyImport_ReloadModule emits deprecation warning
https://bugs.python.org/issue40417 closed by brett.cannon
#40419: timeit CLI docs still mention old power sequence
https://bugs.python.org/issue40419 closed by serhiy.storchaka
#40433: Equality operations between lists and arrays
https://bugs.python.org/issue40433 closed by rhettinger
#40449: multi-line f-string, syntaxerror points to wrong line
https://bugs.python.org/issue40449 closed by lys.nikolaou
#40459: [easy] undefined names in platform.py
https://bugs.python.org/issue40459 closed by vstinner
#40461: execution of file with pictures doesn't work in command --onef
https://bugs.python.org/issue40461 closed by ned.deily
#40465: Deprecate the optional *random* argument to random.shuffle()
https://bugs.python.org/issue40465 closed by rhettinger
#40466: asyncio.ensure_future() breaks implicit exception chaining
https://bugs.python.org/issue40466 closed by chris.jerdonek
#40472: PEG parser disables IDLE Shell input > 2 lines
https://bugs.python.org/issue40472 closed by terry.reedy
#40473: Universal newline not recognizing Mac newline (CR) when using
https://bugs.python.org/issue40473 closed by larrykuhn
#40475: json.JSONEncoder override default method
https://bugs.python.org/issue40475 closed by xsmyqf
#40476: Write file failed on OS X 10.15.4
https://bugs.python.org/issue40476 closed by 韩振宇
#40480: "fnmatch" exponential execution time
https://bugs.python.org/issue40480 closed by tim.peters
#40488: setup.py shall search by default for libffi.so in /usr/local/l
https://bugs.python.org/issue40488 closed by ned.deily
#40489: INCREF/DECREFs around the rich comparison needs tests
https://bugs.python.org/issue40489 closed by rhettinger
#40490: peg_generator module has unused imports
https://bugs.python.org/issue40490 closed by pablogsal
#40491: Typo in SyntaxError produced by pegen for numeric literals
https://bugs.python.org/issue40491 closed by serhiy.storchaka
#40493: Pegen can't parse some function type comments
https://bugs.python.org/issue40493 closed by gvanrossum
#40496: re.findall() takes a long time (100% cup usage) on Python 3.6.
https://bugs.python.org/issue40496 closed by serhiy.storchaka
#40499: asyncio.wait documentation on non-emptiness requirement lost i
https://bugs.python.org/issue40499 closed by gvanrossum
#40504: Restore weakref support for lru_cache wrappers
https://bugs.python.org/issue40504 closed by rhettinger
#40508: IDLE won't open
https://bugs.python.org/issue40508 closed by ned.deily
#40517: Syntax highlighting for ASDL
https://bugs.python.org/issue40517 closed by rhettinger
#40518: ValueError when using resource.setrlimit on macOS Catalina
https://bugs.python.org/issue40518 closed by ned.deily
#40520: Remove redundant comment in pydebug.h
https://bugs.python.org/issue40520 closed by corona10
#40523: Weakref proxy missing pass throughs for hash() and reversed()
https://bugs.python.org/issue40523 closed by pablogsal
#40524: Print() issue
https://bugs.python.org/issue40524 closed by eric.smith
#40526: documentation bad on asyncio
https://bugs.python.org/issue40526 closed by yselivanov
#40527: Multiple "unknown option" errors when passing unknown argument
https://bugs.python.org/issue40527 closed by corona10
#40531: Adding the method find() to list
https://bugs.python.org/issue40531 closed by rhettinger
#40532: Persmission error
https://bugs.python.org/issue40532 closed by eric.smith
#40535: While build python 3.8.2 in linux ctypes.so is using libffi.s
https://bugs.python.org/issue40535 closed by ned.deily
#40537: Typo in Doc/library/sqlite3.rst
https://bugs.python.org/issue40537 closed by nanjekyejoannah
#40538: struct.calcsize('L')== 8 but 4 is specified in documentation
https://bugs.python.org/issue40538 closed by mark.dickinson
#40539: Docs - difflib.SequenceMatcher quick_ratio and real_quick_rati
https://bugs.python.org/issue40539 closed by tim.peters
#40541: Add optional weights parameter to random.sample()
https://bugs.python.org/issue40541 closed by rhettinger
#40542: path environment variable not created correctly
https://bugs.python.org/issue40542 closed by steve.dower
#40555: pegen (PEP 617): bug in error handling
https://bugs.python.org/issue40555 closed by pablogsal
#40559: Missing Py_DECREF in task_step_impl() in _asynciomodule.c
https://bugs.python.org/issue40559 closed by chris.jerdonek
#40560: uuid.uuid4().hex not random
https://bugs.python.org/issue40560 closed by KingUdo
#40565: is comparison returns False while ids are the same.
https://bugs.python.org/issue40565 closed by eric.smith
#1648957: HP-UX: _ctypes/libffi/src/ia64/ffi/__attribute__/native cc
https://bugs.python.org/issue1648957 closed by ned.deily
[View Less]
1
0

PoC: Subinterpreters 4x faster than sequential execution or threads on CPU-bound workaround
by Victor Stinner May 8, 2020
by Victor Stinner May 8, 2020
May 8, 2020
Hi,
I wrote a "per-interpreter GIL" proof-of-concept: each interpreter
gets its own GIL. I chose to benchmark a factorial function in pure
Python to simulate a CPU-bound workload. I wrote the simplest possible
function just to be able to run a benchmark, to check if the PEP 554
would be relevant.
The proof-of-concept proves that subinterpreters can make a CPU-bound
workload faster than sequential execution or threads and that they
have the same speed than multiprocessing. The performance …
[View More]scales well
with the number of CPUs.
Performance
===========
Factorial:
n = 50_000
fact = 1
for i in range(1, n + 1):
fact = fact * i
2 CPUs:
Sequential: 1.00 sec +- 0.01 sec
Threads: 1.08 sec +- 0.01 sec
Multiprocessing: 529 ms +- 6 ms
Subinterpreters: 553 ms +- 6 ms
4 CPUs:
Sequential: 1.99 sec +- 0.01 sec
Threads: 3.15 sec +- 0.97 sec
Multiprocessing: 560 ms +- 12 ms
Subinterpreters: 583 ms +- 7 ms
8 CPUs:
Sequential: 4.01 sec +- 0.02 sec
Threads: 9.91 sec +- 0.54 sec
Multiprocessing: 1.02 sec +- 0.01 sec
Subinterpreters: 1.10 sec +- 0.00 sec
Benchmarks run on my laptop which has 8 logical CPUs (4 physical CPU
cores with Hyper Threading).
Threads are between 1.1x (2 CPUs) and 2.5x (8 CPUs) SLOWER than
sequential execution.
Subinterpreters are between 1.8x (2 CPUs) and 3.6x (8 CPUs) FASTER
than sequential execution.
Subinterpreters and multiprocessing have basically the same speed on
this benchmark.
See demo-pyperf.py attached to https://bugs.python.org/issue40512 for
the code of the benchmark.
Implementation
==============
See https://bugs.python.org/issue40512 and related issues for the
implementation. I already merged changes, but most code is disabled by
default: a new special undocumented
--with-experimental-isolated-subinterpreters build mode is required to
test it.
To reproduce the benchmark, use::
# up to date checkout of Python master branch
./configure \
--with-experimental-isolated-subinterpreters \
--enable-optimizations \
--with-lto
make
./python demo-pyperf.py
Limits of subinterpreters design
================================
Subinterpreters have a few design limits:
* A Python object must not be shared between two interpreters.
* Each interpreter has a minimum memory footprint, since Python
internal states and modules are duplicated.
* Others that I forgot :-)
Incomplete implementation
=========================
My proof-of-concept is just good enough to compute factorial with the
code that I wrote above :-) Any other code is very likely to crash in
various funny ways.
I added a few "#ifdef EXPERIMENTAL_ISOLATED_SUBINTERPRETERS" for the
proof-of-concept. Most are temporary workarounds until some parts of
the code are modified to become compatible with subinterpreters, like
tuple free lists or Unicode interned strings.
Right now, there are still some states which are shared between
subinterpreters: like None and True singletons, but also statically
allocated types. Avoid shared states should enhance performances.
See https://bugs.python.org/issue40512 for the current status and a
list of tasks.
Most of these tasks are already tracked in Eric Snow's "Multi Core
Python" project:
https://github.com/ericsnowcurrently/multi-core-python/issues
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
[View Less]
15
21
As part of PEP 399 <https://www.python.org/dev/peps/pep-0399/>, an idiom
for testing both C and pure Python versions of a library is suggested
making use if import_fresh_module.
Unfortunately, I'm finding that this is not amazingly robust. We have
this issue: https://bugs.python.org/issue40058, where the tester for
datetime needs to do some funky manipulations
<https://github.com/python/cpython/blob/302e5a8f79514fd84bafbc44b7c97ec63630…>to
the state of sys.modules for reasons that …
[View More]are now somewhat unclear, and
still sys.modules is apparently left in a bad state.
When implementing PEP 615, I ran into similar issues and found it very
difficult to get two independent instances of the same module – one with
the C extension blocked and one with it intact. I ended up manually
importing the C and Python extensions and grafting them onto two "fresh"
imports with nothing blocked
<https://github.com/pganssle/zoneinfo/blob/ffd21a6d065e04725e04b37bb430c2559…>.
This seems to work most of the time in my repo, but when I import it
into CPython, I'm now seeing failures due to this issue. The immediate
symptom is that assertRaises is seeing a mismatch between the exception
raised by the module and the exception *on* the module. Here's the
Travis error
<https://travis-ci.org/github/python/cpython/jobs/683080079#L3154-L3186>
(ignore the part about `tzdata`, that needs to be removed as
misleading), and here's the test
<https://github.com/python/cpython/pull/19909/files#diff-885914f7d01a0c33a25…>.
Evidently calling module.ZoneInfo("Bad_Zone") is raising a different
module's ZoneInfoNotFoundError in some cases and I have no idea why.
Is anyone familiar more familiar with the import system willing to take
a look at these issues?
Thanks,
Paul
[View Less]
5
9
It seems to me that os.removedirs() and os.renames() was added just for
symmetry with os.makedirs(). All three functions have similar structure
and was added in the same commit. Seems they were initially code
examples of using some os.path and os functions.
Unlike to quite popular os.makedirs(), os.removedirs() and os.renames()
are not used in the stdlib and rarely used in third party code.
os.removedirs() is considered as an opposite to os.makedirs(), and
os.renames() is a combination …
[View More]of os.makedirs(), os.rename() and
os.removedirs(). The problems with them are:
1. They do not remove directory if any file or other subdirectory is
left. They just stop removing and return success. ZTo the user it looks
like they do not work as expected, but he need to test the existence of
directory explicitly to check this.
2. They can remove more than expected. If the parent directory was empty
before calling os.makedirs(), the following os.removedirs() will remove
not just the newly created directories, but the parent directory, and
its parent if it contained a single directory, and so on.
os.removedirs() is not an opposite to os.makedirs(). It can remove less
or more, and you have no control on how much it will remove. It is
better to use shutil.rmtree().
os.renames() correspondingly can be replaced by os.rename() or
shutil.move(), with possible addition of os.makedirs() and
shutil.rmtree() if needed.
I propose to deprecate these functions and remove them in future Python
versions.
[View Less]
2
1
Hi all,
Thanks for the great feedback. I've updated PEP 554 (Multiple
Interpreters in the Stdlib) following feedback.
https://www.python.org/dev/peps/pep-0554/
Here's a summary of the main changes:
* [API] dropped/deferred the "release" and "close" methods from
RecvChannel and SendChannel (they were unnecessary and
the "association" stuff was too confusing)
* [API] dropped RecvChannel/SendChannel.interpreters
* [API] dropped/deferred SendChannel.send_buffer()
* [API] renamed …
[View More]Interpreter.destroy() to Interpreter.close()
* [API] added a per-interpreter "isolated" mode (default: on)
* added a section about "Help for Extension Module Maintainers"
* added a section about documentation
* added many entries to the "deferred" and "rejected" sections
Further feedback is welcome, though I feel like the PR is ready (or
very close to ready) for pronouncement. Thanks again to all.
-eric
------------------------------------------------------
PEP: 554
Title: Multiple Interpreters in the Stdlib
Author: Eric Snow <ericsnowcurrently(a)gmail.com>
BDFL-Delegate: Antoine Pitrou <antoine(a)python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2017-09-05
Python-Version: 3.9
Post-History: 07-Sep-2017, 08-Sep-2017, 13-Sep-2017, 05-Dec-2017,
09-May-2018, 20-Apr-2020, 01-May-2020
Abstract
========
CPython has supported multiple interpreters in the same process (AKA
"subinterpreters") since version 1.5 (1997). The feature has been
available via the C-API. [c-api]_ Subinterpreters operate in
`relative isolation from one another <Interpreter Isolation_>`_, which
facilitates novel alternative approaches to
`concurrency <Concurrency_>`_.
This proposal introduces the stdlib ``interpreters`` module. The module
will be `provisional <Provisional Status_>`_. It exposes the basic
functionality of subinterpreters already provided by the C-API, along
with new (basic) functionality for sharing data between interpreters.
A Disclaimer about the GIL
==========================
To avoid any confusion up front: This PEP is unrelated to any efforts
to stop sharing the GIL between subinterpreters. At most this proposal
will allow users to take advantage of any results of work on the GIL.
The position here is that exposing subinterpreters to Python code is
worth doing, even if they still share the GIL.
Proposal
========
The ``interpreters`` module will be added to the stdlib. To help
authors of extension modules, a new page will be added to the
`Extending Python <extension-docs_>`_ docs. More information on both
is found in the immediately following sections.
The "interpreters" Module
-------------------------
The ``interpreters`` module will
provide a high-level interface to subinterpreters and wrap a new
low-level ``_interpreters`` (in the same way as the ``threading``
module). See the `Examples`_ section for concrete usage and use cases.
Along with exposing the existing (in CPython) subinterpreter support,
the module will also provide a mechanism for sharing data between
interpreters. This mechanism centers around "channels", which are
similar to queues and pipes.
Note that *objects* are not shared between interpreters since they are
tied to the interpreter in which they were created. Instead, the
objects' *data* is passed between interpreters. See the `Shared data`_
section for more details about sharing between interpreters.
At first only the following types will be supported for sharing:
* None
* bytes
* str
* int
* PEP 554 channels
Support for other basic types (e.g. bool, float, Ellipsis) will be added later.
API summary for interpreters module
-----------------------------------
Here is a summary of the API for the ``interpreters`` module. For a
more in-depth explanation of the proposed classes and functions, see
the `"interpreters" Module API`_ section below.
For creating and using interpreters:
+---------------------------------------------+----------------------------------------------+
| signature | description
|
+=============================================+==============================================+
| ``list_all() -> [Interpreter]`` | Get all existing
interpreters. |
+---------------------------------------------+----------------------------------------------+
| ``get_current() -> Interpreter`` | Get the currently
running interpreter. |
+---------------------------------------------+----------------------------------------------+
| ``get_main() -> Interpreter`` | Get the main
interpreter. |
+---------------------------------------------+----------------------------------------------+
| ``create(*, isolated=True) -> Interpreter`` | Initialize a new
(idle) Python interpreter. |
+---------------------------------------------+----------------------------------------------+
|
+----------------------------------------+-----------------------------------------------------+
| signature | description
|
+========================================+=====================================================+
| ``class Interpreter(id)`` | A single interpreter.
|
+----------------------------------------+-----------------------------------------------------+
| ``.id`` | The interpreter's ID
(read-only). |
+----------------------------------------+-----------------------------------------------------+
| ``.isolated`` | The interpreter's mode
(read-only). |
+----------------------------------------+-----------------------------------------------------+
| ``.is_running() -> bool`` | Is the interpreter
currently executing code? |
+----------------------------------------+-----------------------------------------------------+
| ``.close()`` | Finalize and destroy the
interpreter. |
+----------------------------------------+-----------------------------------------------------+
| ``.run(src_str, /, *, channels=None)`` | | Run the given source code
in the interpreter. |
| | | (This blocks the current
thread until done.) |
+----------------------------------------+-----------------------------------------------------+
|
+--------------------+------------------+------------------------------------------------------+
| exception | base | description
|
+====================+==================+======================================================+
| ``RunFailedError`` | ``RuntimeError`` | Interpreter.run() resulted
in an uncaught exception. |
+--------------------+------------------+------------------------------------------------------+
For sharing data between interpreters:
+---------------------------------------------------------+--------------------------------------------+
| signature |
description |
+=========================================================+============================================+
| ``is_shareable(obj) -> Bool`` | | Can the
object's data be shared |
| | | between
interpreters? |
+---------------------------------------------------------+--------------------------------------------+
| ``create_channel() -> (RecvChannel, SendChannel)`` | | Create a
new channel for passing |
| | | data
between interpreters. |
+---------------------------------------------------------+--------------------------------------------+
| ``list_all_channels() -> [(RecvChannel, SendChannel)]`` | Get all
open channels. |
+---------------------------------------------------------+--------------------------------------------+
|
+------------------------------------------+-----------------------------------------------+
| signature | description
|
+==========================================+===============================================+
| ``class RecvChannel(id)`` | The receiving end of a
channel. |
+------------------------------------------+-----------------------------------------------+
| ``.id`` | The channel's unique ID.
|
+------------------------------------------+-----------------------------------------------+
| ``.recv() -> object`` | | Get the next object
from the channel, |
| | | and wait if none have
been sent. |
+------------------------------------------+-----------------------------------------------+
| ``.recv_nowait(default=None) -> object`` | | Like recv(), but return
the default |
| | | instead of waiting.
|
+------------------------------------------+-----------------------------------------------+
|
+------------------------------+--------------------------------------------------+
| signature | description
|
+==============================+==================================================+
| ``class SendChannel(id)`` | The sending end of a channel.
|
+------------------------------+--------------------------------------------------+
| ``.id`` | The channel's unique ID.
|
+------------------------------+--------------------------------------------------+
| ``.send(obj)`` | | Send the object (i.e. its data) to
the |
| | | receiving end of the channel and
wait. |
+------------------------------+--------------------------------------------------+
| ``.send_nowait(obj)`` | | Like send(), but return False if
not received. |
+------------------------------+--------------------------------------------------+
|
+--------------------------+------------------------+------------------------------------------------+
| exception | base | description
|
+==========================+========================+================================================+
| ``ChannelError`` | ``Exception`` | The base class
for channel-related exceptions. |
+--------------------------+------------------------+------------------------------------------------+
| ``ChannelNotFoundError`` | ``ChannelError`` | The identified
channel was not found. |
+--------------------------+------------------------+------------------------------------------------+
| ``ChannelEmptyError`` | ``ChannelError`` | The channel was
unexpectedly empty. |
+--------------------------+------------------------+------------------------------------------------+
| ``ChannelNotEmptyError`` | ``ChannelError`` | The channel was
unexpectedly not empty. |
+--------------------------+------------------------+------------------------------------------------+
| ``NotReceivedError`` | ``ChannelError`` | Nothing was
waiting to receive a sent object. |
+--------------------------+------------------------+------------------------------------------------+
Help for Extension Module Maintainers
-------------------------------------
Many extension modules do not support use in subinterpreters yet. The
maintainers and users of such extension modules will both benefit when
they are updated to support subinterpreters. In the meantime users may
become confused by failures when using subinterpreters, which could
negatively impact extension maintainers. See `Concerns`_ below.
To mitigate that impact and accelerate compatibility, we will do the
following:
* be clear that extension modules are *not* required to support use in
subinterpreters
* raise ``ImportError`` when an incompatible (no PEP 489 support) module
is imported in a subinterpreter
* provide resources (e.g. docs) to help maintainers reach compatibility
* reach out to the maintainers of Cython and of the most used extension
modules (on PyPI) to get feedback and possibly provide assistance
Examples
========
Run isolated code
-----------------
::
interp = interpreters.create()
print('before')
interp.run('print("during")')
print('after')
Run in a thread
---------------
::
interp = interpreters.create()
def run():
interp.run('print("during")')
t = threading.Thread(target=run)
print('before')
t.start()
print('after')
Pre-populate an interpreter
---------------------------
::
interp = interpreters.create()
interp.run(tw.dedent("""
import some_lib
import an_expensive_module
some_lib.set_up()
"""))
wait_for_request()
interp.run(tw.dedent("""
some_lib.handle_request()
"""))
Handling an exception
---------------------
::
interp = interpreters.create()
try:
interp.run(tw.dedent("""
raise KeyError
"""))
except interpreters.RunFailedError as exc:
print(f"got the error from the subinterpreter: {exc}")
Re-raising an exception
-----------------------
::
interp = interpreters.create()
try:
try:
interp.run(tw.dedent("""
raise KeyError
"""))
except interpreters.RunFailedError as exc:
raise exc.__cause__
except KeyError:
print("got a KeyError from the subinterpreter")
Note that this pattern is a candidate for later improvement.
Synchronize using a channel
---------------------------
::
interp = interpreters.create()
r, s = interpreters.create_channel()
def run():
interp.run(tw.dedent("""
reader.recv()
print("during")
"""),
shared=dict(
reader=r,
),
)
t = threading.Thread(target=run)
print('before')
t.start()
print('after')
s.send(b'')
Sharing a file descriptor
-------------------------
::
interp = interpreters.create()
r1, s1 = interpreters.create_channel()
r2, s2 = interpreters.create_channel()
def run():
interp.run(tw.dedent("""
fd = int.from_bytes(
reader.recv(), 'big')
for line in os.fdopen(fd):
print(line)
writer.send(b'')
"""),
shared=dict(
reader=r,
writer=s2,
),
)
t = threading.Thread(target=run)
t.start()
with open('spamspamspam') as infile:
fd = infile.fileno().to_bytes(1, 'big')
s.send(fd)
r.recv()
Passing objects via marshal
---------------------------
::
interp = interpreters.create()
r, s = interpreters.create_channel()
interp.run(tw.dedent("""
import marshal
"""),
shared=dict(
reader=r,
),
)
def run():
interp.run(tw.dedent("""
data = reader.recv()
while data:
obj = marshal.loads(data)
do_something(obj)
data = reader.recv()
"""))
t = threading.Thread(target=run)
t.start()
for obj in input:
data = marshal.dumps(obj)
s.send(data)
s.send(None)
Passing objects via pickle
--------------------------
::
interp = interpreters.create()
r, s = interpreters.create_channel()
interp.run(tw.dedent("""
import pickle
"""),
shared=dict(
reader=r,
),
)
def run():
interp.run(tw.dedent("""
data = reader.recv()
while data:
obj = pickle.loads(data)
do_something(obj)
data = reader.recv()
"""))
t = threading.Thread(target=run)
t.start()
for obj in input:
data = pickle.dumps(obj)
s.send(data)
s.send(None)
Running a module
----------------
::
interp = interpreters.create()
main_module = mod_name
interp.run(f'import runpy; runpy.run_module({main_module!r})')
Running as script (including zip archives & directories)
--------------------------------------------------------
::
interp = interpreters.create()
main_script = path_name
interp.run(f"import runpy; runpy.run_path({main_script!r})")
Running in a thread pool executor
---------------------------------
::
interps = [interpreters.create() for i in range(5)]
with concurrent.futures.ThreadPoolExecutor(max_workers=len(interps)) as pool:
print('before')
for interp in interps:
pool.submit(interp.run, 'print("starting"); print("stopping")'
print('after')
Rationale
=========
Running code in multiple interpreters provides a useful level of
isolation within the same process. This can be leveraged in a number
of ways. Furthermore, subinterpreters provide a well-defined framework
in which such isolation may extended.
Nick Coghlan explained some of the benefits through a comparison with
multi-processing [benefits]_::
[I] expect that communicating between subinterpreters is going
to end up looking an awful lot like communicating between
subprocesses via shared memory.
The trade-off between the two models will then be that one still
just looks like a single process from the point of view of the
outside world, and hence doesn't place any extra demands on the
underlying OS beyond those required to run CPython with a single
interpreter, while the other gives much stricter isolation
(including isolating C globals in extension modules), but also
demands much more from the OS when it comes to its IPC
capabilities.
The security risk profiles of the two approaches will also be quite
different, since using subinterpreters won't require deliberately
poking holes in the process isolation that operating systems give
you by default.
CPython has supported subinterpreters, with increasing levels of
support, since version 1.5. While the feature has the potential
to be a powerful tool, subinterpreters have suffered from neglect
because they are not available directly from Python. Exposing the
existing functionality in the stdlib will help reverse the situation.
This proposal is focused on enabling the fundamental capability of
multiple isolated interpreters in the same Python process. This is a
new area for Python so there is relative uncertainly about the best
tools to provide as companions to subinterpreters. Thus we minimize
the functionality we add in the proposal as much as possible.
Concerns
--------
* "subinterpreters are not worth the trouble"
Some have argued that subinterpreters do not add sufficient benefit
to justify making them an official part of Python. Adding features
to the language (or stdlib) has a cost in increasing the size of
the language. So an addition must pay for itself. In this case,
subinterpreters provide a novel concurrency model focused on isolated
threads of execution. Furthermore, they provide an opportunity for
changes in CPython that will allow simultaneous use of multiple CPU
cores (currently prevented by the GIL).
Alternatives to subinterpreters include threading, async, and
multiprocessing. Threading is limited by the GIL and async isn't
the right solution for every problem (nor for every person).
Multiprocessing is likewise valuable in some but not all situations.
Direct IPC (rather than via the multiprocessing module) provides
similar benefits but with the same caveat.
Notably, subinterpreters are not intended as a replacement for any of
the above. Certainly they overlap in some areas, but the benefits of
subinterpreters include isolation and (potentially) performance. In
particular, subinterpreters provide a direct route to an alternate
concurrency model (e.g. CSP) which has found success elsewhere and
will appeal to some Python users. That is the core value that the
``interpreters`` module will provide.
* "stdlib support for subinterpreters adds extra burden
on C extension authors"
In the `Interpreter Isolation`_ section below we identify ways in
which isolation in CPython's subinterpreters is incomplete. Most
notable is extension modules that use C globals to store internal
state. PEP 3121 and PEP 489 provide a solution for most of the
problem, but one still remains. [petr-c-ext]_ Until that is resolved
(see PEP 573), C extension authors will face extra difficulty
to support subinterpreters.
Consequently, projects that publish extension modules may face an
increased maintenance burden as their users start using subinterpreters,
where their modules may break. This situation is limited to modules
that use C globals (or use libraries that use C globals) to store
internal state. For numpy, the reported-bug rate is one every 6
months. [bug-rate]_
Ultimately this comes down to a question of how often it will be a
problem in practice: how many projects would be affected, how often
their users will be affected, what the additional maintenance burden
will be for projects, and what the overall benefit of subinterpreters
is to offset those costs. The position of this PEP is that the actual
extra maintenance burden will be small and well below the threshold at
which subinterpreters are worth it.
* "creating a new concurrency API deserves much more thought and
experimentation, so the new module shouldn't go into the stdlib
right away, if ever"
Introducing an API for a a new concurrency model, like happened with
asyncio, is an extremely large project that requires a lot of careful
consideration. It is not something that can be done a simply as this
PEP proposes and likely deserves significant time on PyPI to mature.
(See `Nathaniel's post <nathaniel-asyncio>`_ on python-dev.)
However, this PEP does not propose any new concurrency API. At most
it exposes minimal tools (e.g. subinterpreters, channels) which may
be used to write code that follows patterns associated with (relatively)
new-to-Python `concurrency models <Concurrency_>`_. Those tools could
also be used as the basis for APIs for such concurrency models.
Again, this PEP does not propose any such API.
* "there is no point to exposing subinterpreters if they still share
the GIL"
* "the effort to make the GIL per-interpreter is disruptive and risky"
A common misconception is that this PEP also includes a promise that
subinterpreters will no longer share the GIL. When that is clarified,
the next question is "what is the point?". This is already answered
at length in this PEP. Just to be clear, the value lies in::
* increase exposure of the existing feature, which helps improve
the code health of the entire CPython runtime
* expose the (mostly) isolated execution of subinterpreters
* preparation for per-interpreter GIL
* encourage experimentation
* "data sharing can have a negative impact on cache performance
in multi-core scenarios"
(See [cache-line-ping-pong]_.)
This shouldn't be a problem for now as we have no immediate plans
to actually share data between interpreters, instead focusing
on copying.
About Subinterpreters
=====================
Concurrency
-----------
Concurrency is a challenging area of software development. Decades of
research and practice have led to a wide variety of concurrency models,
each with different goals. Most center on correctness and usability.
One class of concurrency models focuses on isolated threads of
execution that interoperate through some message passing scheme. A
notable example is `Communicating Sequential Processes`_ (CSP) (upon
which Go's concurrency is roughly based). The isolation inherent to
subinterpreters makes them well-suited to this approach.
Shared data
-----------
Subinterpreters are inherently isolated (with caveats explained below),
in contrast to threads. So the same communicate-via-shared-memory
approach doesn't work. Without an alternative, effective use of
concurrency via subinterpreters is significantly limited.
The key challenge here is that sharing objects between interpreters
faces complexity due to various constraints on object ownership,
visibility, and mutability. At a conceptual level it's easier to
reason about concurrency when objects only exist in one interpreter
at a time. At a technical level, CPython's current memory model
limits how Python *objects* may be shared safely between interpreters;
effectively objects are bound to the interpreter in which they were
created. Furthermore the complexity of *object* sharing increases as
subinterpreters become more isolated, e.g. after GIL removal.
Consequently,the mechanism for sharing needs to be carefully considered.
There are a number of valid solutions, several of which may be
appropriate to support in Python. This proposal provides a single basic
solution: "channels". Ultimately, any other solution will look similar
to the proposed one, which will set the precedent. Note that the
implementation of ``Interpreter.run()`` will be done in a way that
allows for multiple solutions to coexist, but doing so is not
technically a part of the proposal here.
Regarding the proposed solution, "channels", it is a basic, opt-in data
sharing mechanism that draws inspiration from pipes, queues, and CSP's
channels. [fifo]_
As simply described earlier by the API summary,
channels have two operations: send and receive. A key characteristic
of those operations is that channels transmit data derived from Python
objects rather than the objects themselves. When objects are sent,
their data is extracted. When the "object" is received in the other
interpreter, the data is converted back into an object owned by that
interpreter.
To make this work, the mutable shared state will be managed by the
Python runtime, not by any of the interpreters. Initially we will
support only one type of objects for shared state: the channels provided
by ``create_channel()``. Channels, in turn, will carefully manage
passing objects between interpreters.
This approach, including keeping the API minimal, helps us avoid further
exposing any underlying complexity to Python users. Along those same
lines, we will initially restrict the types that may be passed through
channels to the following:
* None
* bytes
* str
* int
* channels
Limiting the initial shareable types is a practical matter, reducing
the potential complexity of the initial implementation. There are a
number of strategies we may pursue in the future to expand supported
objects and object sharing strategies.
Interpreter Isolation
---------------------
CPython's interpreters are intended to be strictly isolated from each
other. Each interpreter has its own copy of all modules, classes,
functions, and variables. The same applies to state in C, including in
extension modules. The CPython C-API docs explain more. [caveats]_
However, there are ways in which interpreters share some state. First
of all, some process-global state remains shared:
* file descriptors
* builtin types (e.g. dict, bytes)
* singletons (e.g. None)
* underlying static module data (e.g. functions) for
builtin/extension/frozen modules
There are no plans to change this.
Second, some isolation is faulty due to bugs or implementations that did
not take subinterpreters into account. This includes things like
extension modules that rely on C globals. [cryptography]_ In these
cases bugs should be opened (some are already):
* readline module hook functions (http://bugs.python.org/issue4202)
* memory leaks on re-init (http://bugs.python.org/issue21387)
Finally, some potential isolation is missing due to the current design
of CPython. Improvements are currently going on to address gaps in this
area:
* GC is not run per-interpreter [global-gc]_
* at-exit handlers are not run per-interpreter [global-atexit]_
* extensions using the ``PyGILState_*`` API are incompatible [gilstate]_
* interpreters share memory management (e.g. allocators, gc)
* interpreters share the GIL
Existing Usage
--------------
Subinterpreters are not a widely used feature. In fact, the only
documented cases of wide-spread usage are
`mod_wsgi <https://github.com/GrahamDumpleton/mod_wsgi>`_,
`OpenStack Ceph <https://github.com/ceph/ceph/pull/14971>`_, and
`JEP <https://github.com/ninia/jep>`_. On the one hand, these cases
provide confidence that existing subinterpreter support is relatively
stable. On the other hand, there isn't much of a sample size from which
to judge the utility of the feature.
Provisional Status
==================
The new ``interpreters`` module will be added with "provisional" status
(see PEP 411). This allows Python users to experiment with the feature
and provide feedback while still allowing us to adjust to that feedback.
The module will be provisional in Python 3.9 and we will make a decision
before the 3.10 release whether to keep it provisional, graduate it, or
remove it. This PEP will be updated accordingly.
While the module is provisional, any changes to the API (or to behavior)
do not need to be reflected here, nor get approval by the BDFL-delegate.
However, such changes will still need to go through the normal processes
(BPO for smaller changes and python-dev/PEP for substantial ones).
Alternate Python Implementations
================================
I've solicited feedback from various Python implementors about support
for subinterpreters. Each has indicated that they would be able to
support subinterpreters (if they choose to) without a lot of
trouble. Here are the projects I contacted:
* jython ([jython]_)
* ironpython (personal correspondence)
* pypy (personal correspondence)
* micropython (personal correspondence)
.. _interpreters-list-all:
.. _interpreters-get-current:
.. _interpreters-create:
.. _interpreters-Interpreter:
"interpreters" Module API
=========================
The module provides the following functions::
list_all() -> [Interpreter]
Return a list of all existing interpreters.
get_current() => Interpreter
Return the currently running interpreter.
get_main() => Interpreter
Return the main interpreter. If the Python implementation
has no concept of a main interpreter then return None.
create(*, isolated=True) -> Interpreter
Initialize a new Python interpreter and return it. The
interpreter will be created in the current thread and will remain
idle until something is run in it. The interpreter may be used
in any thread and will run in whichever thread calls
``interp.run()``. See "Interpreter Isolated Mode" below for
an explanation of the "isolated" parameter.
The module also provides the following class::
class Interpreter(id):
id -> int:
The interpreter's ID. (read-only)
isolated -> bool:
Whether or not the interpreter is operating in "isolated" mode.
(read-only)
is_running() -> bool:
Return whether or not the interpreter is currently executing
code. Calling this on the current interpreter will always
return True.
close():
Finalize and destroy the interpreter.
This may not be called on an already running interpreter.
Doing so results in a RuntimeError.
run(source_str, /, *, channels=None):
Run the provided Python source code in the interpreter. If
the "channels" keyword argument is provided (and is a mapping
of attribute names to channels) then it is added to the
interpreter's execution namespace (the interpreter's
"__main__" module). If any of the values are not RecvChannel
or SendChannel instances then ValueError gets raised.
This may not be called on an already running interpreter.
Doing so results in a RuntimeError.
A "run()" call is similar to a function call. Once it
completes, the code that called "run()" continues executing
(in the original interpreter). Likewise, if there is any
uncaught exception then it effectively (see below) propagates
into the code where ``run()`` was called. However, unlike
function calls (but like threads), there is no return value.
If any value is needed, pass it out via a channel.
The big difference from functions is that "run()" executes
the code in an entirely different interpreter, with entirely
separate state. The state of the current interpreter in the
current OS thread is swapped out with the state of the target
interpreter (the one that will execute the code). When the
target finishes executing, the original interpreter gets
swapped back in and its execution resumes.
So calling "run()" will effectively cause the current Python
thread to pause. Sometimes you won't want that pause, in
which case you should make the "run()" call in another thread.
To do so, add a function that calls "run()" and then run that
function in a normal "threading.Thread".
Note that the interpreter's state is never reset, neither
before "run()" executes the code nor after. Thus the
interpreter state is preserved between calls to "run()".
This includes "sys.modules", the "builtins" module, and the
internal state of C extension modules.
Also note that "run()" executes in the namespace of the
"__main__" module, just like scripts, the REPL, "-m", and
"-c". Just as the interpreter's state is not ever reset, the
"__main__" module is never reset. You can imagine
concatenating the code from each "run()" call into one long
script. This is the same as how the REPL operates.
Supported code: source text.
Uncaught Exceptions
-------------------
Regarding uncaught exceptions in ``Interpreter.run()``, we noted that
they are "effectively" propagated into the code where ``run()`` was
called. To prevent leaking exceptions (and tracebacks) between
interpreters, we create a surrogate of the exception and its traceback
(see ``traceback.TracebackException``), set it to ``__cause__`` on a
new ``RunFailedError``, and raise that.
Raising (a proxy of) the exception directly is problematic since it's
harder to distinguish between an error in the ``run()`` call and an
uncaught exception from the subinterpreter.
.. _interpreters-is-shareable:
.. _interpreters-create-channel:
.. _interpreters-list-all-channels:
.. _interpreters-RecvChannel:
.. _interpreters-SendChannel:
API for sharing data
--------------------
Subinterpreters are less useful without a mechanism for sharing data
between them. Sharing actual Python objects between interpreters,
however, has enough potential problems that we are avoiding support
for that here. Instead, only mimimum set of types will be supported.
Initially this will include ``None``, ``bytes``, ``str``, ``int``,
and channels. Further types may be supported later.
The ``interpreters`` module provides a function that users may call
to determine whether an object is shareable or not::
is_shareable(obj) -> bool:
Return True if the object may be shared between interpreters.
This does not necessarily mean that the actual objects will be
shared. Insead, it means that the objects' underlying data will
be shared in a cross-interpreter way, whether via a proxy, a
copy, or some other means.
This proposal provides two ways to share such objects between
interpreters.
First, channels may be passed to ``run()`` via the ``channels``
keyword argument, where they are effectively injected into the target
interpreter's ``__main__`` module. While passing arbitrary shareable
objects this way is possible, doing so is mainly intended for sharing
meta-objects (e.g. channels) between interpreters. It is less useful
to pass other objects (like ``bytes``) to ``run`` directly.
Second, the main mechanism for sharing objects (i.e. their data) between
interpreters is through channels. A channel is a simplex FIFO similar
to a pipe. The main difference is that channels can be associated with
zero or more interpreters on either end. Like queues, which are also
many-to-many, channels are buffered (though they also offer methods
with unbuffered semantics).
Python objects are not shared between interpreters. However, in some
cases data those objects wrap is actually shared and not just copied.
One example might be PEP 3118 buffers. In those cases the object in the
original interpreter is kept alive until the shared data in the other
interpreter is no longer used. Then object destruction can happen like
normal in the original interpreter, along with the previously shared
data.
The ``interpreters`` module provides the following functions related
to channels::
create_channel() -> (RecvChannel, SendChannel):
Create a new channel and return (recv, send), the RecvChannel
and SendChannel corresponding to the ends of the channel.
Both ends of the channel are supported "shared" objects (i.e.
may be safely shared by different interpreters. Thus they
may be passed as keyword arguments to "Interpreter.run()".
list_all_channels() -> [(RecvChannel, SendChannel)]:
Return a list of all open channel-end pairs.
The module also provides the following channel-related classes::
class RecvChannel(id):
The receiving end of a channel. An interpreter may use this to
receive objects from another interpreter. At first only a few
of the simple, immutable builtin types will be supported.
id -> int:
The channel's unique ID. This is shared with the "send" end.
recv():
Return the next object from the channel. If none have been
sent then wait until the next send.
At the least, the object will be equivalent to the sent object.
That will almost always mean the same type with the same data,
though it could also be a compatible proxy. Regardless, it may
use a copy of that data or actually share the data.
recv_nowait(default=None):
Return the next object from the channel. If none have been
sent then return the default. Otherwise, this is the same
as the "recv()" method.
class SendChannel(id):
The sending end of a channel. An interpreter may use this to
send objects to another interpreter. At first only a few of
the simple, immutable builtin types will be supported.
id -> int:
The channel's unique ID. This is shared with the "recv" end.
send(obj):
Send the object (i.e. its data) to the "recv" end of the
channel. Wait until the object is received. If the object
is not shareable then ValueError is raised.
send_nowait(obj):
Send the object to the "recv" end of the channel. This
behaves the same as "send()", except for the waiting part.
If no interpreter is currently receiving (waiting on the
other end) then queue the object and return False. Otherwise
return True.
Channel Lifespan
----------------
A channel is automatically closed and destoyed once there are no more
Python objects (e.g. ``RecvChannel`` and ``SendChannel``) referring
to it. So it is effectively triggered via garbage-collection of those
objects..
.. _isolated-mode:
Interpreter "Isolated" Mode
===========================
By default, every new interpreter created by ``interpreters.create()``
has specific restrictions on any code it runs. This includes the
following:
* importing an extension module fails if it does not implement the
PEP 489 API
* new threads are not allowed (including daemon threads)
* ``os.fork()`` is not allowed (so no ``multiprocessing``)
* ``os.exec*()``, AKA "fork+exec", is not allowed (so no ``subprocess``)
This represents the full "isolated" mode of subinterpreters. It is
applied when ``interpreters.create()`` is called with the "isolated"
keyword-only argument set to ``True`` (the default). If
``interpreters.create(isolated=False)`` is called then none of those
restrictions is applied.
One advantage of this approach is that it allows extension maintainers
to check subinterpreter compatibility before they implement the PEP 489
API. Also note that ``isolated=False`` represents the historical
behavior when using the existing subinterpreters C-API, thus providing
backward compatibility. For the existing C-API itself, the default
remains ``isolated=False``. The same is true for the "main" module, so
existing use of Python will not change.
We may choose to later loosen some of the above restrictions or provide
a way to enable/disable granular restrictions individually. Regardless,
requiring PEP 489 support from extension modules will always be a
default restriction.
Documentation
=============
The new stdlib docs page for the ``interpreters`` module will include
the following:
* (at the top) a clear note that subinterpreter support in extension
modules is not required
* some explanation about what subinterpreters are
* brief examples of how to use subinterpreters and channels
* a summary of the limitations of subinterpreters
* (for extension maintainers) a link to the resources for ensuring
subinterpreter compatibilty
* much of the API information in this PEP
A separate page will be added to the docs for resources to help
extension maintainers ensure their modules can be used safely in
subinterpreters, under `Extending Python <extension-docs_>`. The page
will include the following information:
* a summary about subinterpreters (similar to the same in the new
``interpreters`` module page and in the C-API docs)
* an explanation of how extension modules can be impacted
* how to implement PEP 489 support
* how to move from global module state to per-interpreter
* how to take advantage of PEP 384 (heap types), PEP 3121
(module state), and PEP 573
* strategies for dealing with 3rd party C libraries that keep their
own subinterpreter-incompatible global state
Note that the documentation will play a large part in mitigating any
negative impact that the new ``interpreters`` module might have on
extension module maintainers.
Also, the ``ImportError`` for imcompatible extgension modules will have
a message that clearly says it is due to missing subinterpreter
compatibility and that extensions are not required to provide it. This
will help set user expectations properly.
Deferred Functionality
======================
In the interest of keeping this proposal minimal, the following
functionality has been left out for future consideration. Note that
this is not a judgement against any of said capability, but rather a
deferment. That said, each is arguably valid.
Interpreter.call()
------------------
It would be convenient to run existing functions in subinterpreters
directly. ``Interpreter.run()`` could be adjusted to support this or
a ``call()`` method could be added::
Interpreter.call(f, *args, **kwargs)
This suffers from the same problem as sharing objects between
interpreters via queues. The minimal solution (running a source string)
is sufficient for us to get the feature out where it can be explored.
timeout arg to recv() and send()
--------------------------------
Typically functions that have a ``block`` argument also have a
``timeout`` argument. It sometimes makes sense to do likewise for
functions that otherwise block, like the channel ``recv()`` and
``send()`` methods. We can add it later if needed.
Interpreter.run_in_thread()
---------------------------
This method would make a ``run()`` call for you in a thread. Doing this
using only ``threading.Thread`` and ``run()`` is relatively trivial so
we've left it out.
Synchronization Primitives
--------------------------
The ``threading`` module provides a number of synchronization primitives
for coordinating concurrent operations. This is especially necessary
due to the shared-state nature of threading. In contrast,
subinterpreters do not share state. Data sharing is restricted to
channels, which do away with the need for explicit synchronization. If
any sort of opt-in shared state support is added to subinterpreters in
the future, that same effort can introduce synchronization primitives
to meet that need.
CSP Library
-----------
A ``csp`` module would not be a large step away from the functionality
provided by this PEP. However, adding such a module is outside the
minimalist goals of this proposal.
Syntactic Support
-----------------
The ``Go`` language provides a concurrency model based on CSP, so
it's similar to the concurrency model that subinterpreters support.
However, ``Go`` also provides syntactic support, as well several builtin
concurrency primitives, to make concurrency a first-class feature.
Conceivably, similar syntactic (and builtin) support could be added to
Python using subinterpreters. However, that is *way* outside the scope
of this PEP!
Multiprocessing
---------------
The ``multiprocessing`` module could support subinterpreters in the same
way it supports threads and processes. In fact, the module's
maintainer, Davin Potts, has indicated this is a reasonable feature
request. However, it is outside the narrow scope of this PEP.
C-extension opt-in/opt-out
--------------------------
By using the ``PyModuleDef_Slot`` introduced by PEP 489, we could easily
add a mechanism by which C-extension modules could opt out of support
for subinterpreters. Then the import machinery, when operating in
a subinterpreter, would need to check the module for support. It would
raise an ImportError if unsupported.
Alternately we could support opting in to subinterpreter support.
However, that would probably exclude many more modules (unnecessarily)
than the opt-out approach. Also, note that PEP 489 defined that an
extension's use of the PEP's machinery implies support for
subinterpreters.
The scope of adding the ModuleDef slot and fixing up the import
machinery is non-trivial, but could be worth it. It all depends on
how many extension modules break under subinterpreters. Given that
there are relatively few cases we know of through mod_wsgi, we can
leave this for later.
Poisoning channels
------------------
CSP has the concept of poisoning a channel. Once a channel has been
poisoned, any ``send()`` or ``recv()`` call on it would raise a special
exception, effectively ending execution in the interpreter that tried
to use the poisoned channel.
This could be accomplished by adding a ``poison()`` method to both ends
of the channel. The ``close()`` method can be used in this way
(mostly), but these semantics are relatively specialized and can wait.
Resetting __main__
------------------
As proposed, every call to ``Interpreter.run()`` will execute in the
namespace of the interpreter's existing ``__main__`` module. This means
that data persists there between ``run()`` calls. Sometimes this isn't
desirable and you want to execute in a fresh ``__main__``. Also,
you don't necessarily want to leak objects there that you aren't using
any more.
Note that the following won't work right because it will clear too much
(e.g. ``__name__`` and the other "__dunder__" attributes::
interp.run('globals().clear()')
Possible solutions include:
* a ``create()`` arg to indicate resetting ``__main__`` after each
``run`` call
* an ``Interpreter.reset_main`` flag to support opting in or out
after the fact
* an ``Interpreter.reset_main()`` method to opt in when desired
* ``importlib.util.reset_globals()`` [reset_globals]_
Also note that resetting ``__main__`` does nothing about state stored
in other modules. So any solution would have to be clear about the
scope of what is being reset. Conceivably we could invent a mechanism
by which any (or every) module could be reset, unlike ``reload()``
which does not clear the module before loading into it. Regardless,
since ``__main__`` is the execution namespace of the interpreter,
resetting it has a much more direct correlation to interpreters and
their dynamic state than does resetting other modules. So a more
generic module reset mechanism may prove unnecessary.
This isn't a critical feature initially. It can wait until later
if desirable.
Resetting an interpreter's state
--------------------------------
It may be nice to re-use an existing subinterpreter instead of
spinning up a new one. Since an interpreter has substantially more
state than just the ``__main__`` module, it isn't so easy to put an
interpreter back into a pristine/fresh state. In fact, there *may*
be parts of the state that cannot be reset from Python code.
A possible solution is to add an ``Interpreter.reset()`` method. This
would put the interpreter back into the state it was in when newly
created. If called on a running interpreter it would fail (hence the
main interpreter could never be reset). This would likely be more
efficient than creating a new subinterpreter, though that depends on
what optimizations will be made later to subinterpreter creation.
While this would potentially provide functionality that is not
otherwise available from Python code, it isn't a fundamental
functionality. So in the spirit of minimalism here, this can wait.
Regardless, I doubt it would be controversial to add it post-PEP.
File descriptors and sockets in channels
----------------------------------------
Given that file descriptors and sockets are process-global resources,
support for passing them through channels is a reasonable idea. They
would be a good candidate for the first effort at expanding the types
that channels support. They aren't strictly necessary for the initial
API.
Integration with async
----------------------
Per Antoine Pitrou [async]_::
Has any thought been given to how FIFOs could integrate with async
code driven by an event loop (e.g. asyncio)? I think the model of
executing several asyncio (or Tornado) applications each in their
own subinterpreter may prove quite interesting to reconcile multi-
core concurrency with ease of programming. That would require the
FIFOs to be able to synchronize on something an event loop can wait
on (probably a file descriptor?).
A possible solution is to provide async implementations of the blocking
channel methods (``recv()``, and ``send()``). However,
the basic functionality of subinterpreters does not depend on async and
can be added later.
Alternately, "readiness callbacks" could be used to simplify use in
async scenarios. This would mean adding an optional ``callback``
(kw-only) parameter to the ``recv_nowait()`` and ``send_nowait()``
channel methods. The callback would be called once the object was sent
or received (respectively).
(Note that making channels buffered makes readiness callbacks less
important.)
Support for iteration
---------------------
Supporting iteration on ``RecvChannel`` (via ``__iter__()`` or
``_next__()``) may be useful. A trivial implementation would use the
``recv()`` method, similar to how files do iteration. Since this isn't
a fundamental capability and has a simple analog, adding iteration
support can wait until later.
Channel context managers
------------------------
Context manager support on ``RecvChannel`` and ``SendChannel`` may be
helpful. The implementation would be simple, wrapping a call to
``close()`` (or maybe ``release()``) like files do. As with iteration,
this can wait.
Pipes and Queues
----------------
With the proposed object passing machanism of "channels", other similar
basic types aren't required to achieve the minimal useful functionality
of subinterpreters. Such types include pipes (like unbuffered channels,
but one-to-one) and queues (like channels, but more generic). See below
in `Rejected Ideas` for more information.
Even though these types aren't part of this proposal, they may still
be useful in the context of concurrency. Adding them later is entirely
reasonable. The could be trivially implemented as wrappers around
channels. Alternatively they could be implemented for efficiency at the
same low level as channels.
Return a lock from send()
-------------------------
When sending an object through a channel, you don't have a way of knowing
when the object gets received on the other end. One way to work around
this is to return a locked ``threading.Lock`` from ``SendChannel.send()``
that unlocks once the object is received.
Alternately, the proposed ``SendChannel.send()`` (blocking) and
``SendChannel.send_nowait()`` provide an explicit distinction that is
less likely to confuse users.
Note that returning a lock would matter for buffered channels
(i.e. queues). For unbuffered channels it is a non-issue.
Support prioritization in channels
----------------------------------
A simple example is ``queue.PriorityQueue`` in the stdlib.
Support inheriting settings (and more?)
---------------------------------------
Folks might find it useful, when creating a new subinterpreter, to be
able to indicate that they would like some things "inherited" by the
new interpreter. The mechanism could be a strict copy or it could be
copy-on-write. The motivating example is with the warnings module
(e.g. copy the filters).
The feature isn't critical, nor would it be widely useful, so it
can wait until there's interest. Notably, both suggested solutions
will require significant work, especially when it comes to complex
objects and most especially for mutable containers of mutable
complex objects.
Make exceptions shareable
-------------------------
Exceptions are propagated out of ``run()`` calls, so it isn't a big
leap to make them shareable in channels. However, as noted elsewhere,
it isn't essential or (particularly common) so we can wait on doing
that.
Make RunFailedError.__cause__ lazy
----------------------------------
An uncaught exception in a subinterpreter (from ``run()``) is copied
to the calling interpreter and set as ``__cause__`` on a
``RunFailedError`` which is then raised. That copying part involves
some sort of deserialization in the calling intepreter, which can be
expensive (e.g. due to imports) yet is not always necessary.
So it may be useful to use an ``ExceptionProxy`` type to wrap the
serialized exception and only deserialize it when needed. That could
be via ``ExceptionProxy__getattribute__()`` or perhaps through
``RunFailedError.resolve()`` (which would raise the deserialized
exception and set ``RunFailedError.__cause__`` to the exception.
It may also make sense to have ``RunFailedError.__cause__`` be a
descriptor that does the lazy deserialization (and set ``__cause__``)
on the ``RunFailedError`` instance.
Serialize everything through channels
-------------------------------------
We could use pickle (or marshal) to serialize everything sent through
channels. Doing this is potentially inefficient, but it may be a
matter of convenience in the end. We can add it later, but trying to
remove it later would be significantly more painful.
Return a value from ``run()``
-----------------------------
Currently ``run()`` always returns None. One idea is to return the
return value from whatever the subinterpreter ran. However, for now
it doesn't make sense. The only thing folks can run is a string of
code (i.e. a script). This is equivalent to ``PyRun_StringFlags()``,
``exec()``, or a module body. None of those "return" anything. We can
revisit this once ``run()`` supports functions, etc.
Add a "tp_share" type slot
--------------------------
This would replace the current global registry for shareable types.
Expose which interpreters have actually *used* a channel end.
-------------------------------------------------------------
Currently we associate interpreters upon access to a channel. We would
keep a separate association list for "upon use" and expose that.
Add a shareable synchronization primitive
-----------------------------------------
This would be ``_threading.Lock`` (or something like it) where
interpreters would actually share the underlying mutex. This would
provide much better efficiency than blocking channel ops. The main
concern is that locks and channels don't mix well (as learned in Go).
Note that the same functionality as a lock can be acheived by passing
some sort of "token" object through a channel. "send()" would be
equivalent to releasing the lock and "recv()" to acquiring the lock.
We can add this later if it proves desireable without much trouble.
Propagate SystemExit and KeyboardInterrupt Differently
------------------------------------------------------
The exception types that inherit from ``BaseException`` (aside from
``Exception``) are usually treated specially. These types are:
``KeyboardInterrupt``, ``SystemExit``, and ``GeneratorExit``. It may
make sense to treat them specially when it comes to propagation from
``run()``. Here are some options::
* propagate like normal via RunFailedError
* do not propagate (handle them somehow in the subinterpreter)
* propagate them directly (avoid RunFailedError)
* propagate them directly (set RunFailedError as __cause__)
We aren't going to worry about handling them differently. Threads
already ignore ``SystemExit``, so for now we will follow that pattern.
Add an explicit release() and close() to channel end classes
------------------------------------------------------------
It can be convenient to have an explicit way to close a channel against
further global use. Likewise it could be useful to have an explicit
way to release one of the channel ends relative to the current
interpreter. Among other reasons, such a mechanism is useful for
communicating overall state between interpreters without the extra
boilerplate that passing objects through a channel directly would
require.
The challenge is getting automatic release/close right without making
it hard to understand. This is especially true when dealing with a
non-empty channel. We should be able to get by without release/close
for now.
Add SendChannel.send_buffer()
-----------------------------
This method would allow no-copy sending of an object through a channel
if it supports the PEP 3118 buffer protocol (e.g. memoryview).
Support for this is not fundamental to channels and can be added on
later without much disruption.
Auto-run in a thread
--------------------
The PEP proposes a hard separation between subinterpreters and threads:
if you want to run in a thread you must create the thread yourself and
call ``run()`` in it. However, it might be convenient if ``run()``
could do that for you, meaning there would be less boilerplate.
Furthermore, we anticipate that users will want to run in a thread much
more often than not. So it would make sense to make this the default
behavior. We would add a kw-only param "threaded" (default ``True``)
to ``run()`` to allow the run-in-the-current-thread operation.
Rejected Ideas
==============
Explicit channel association
----------------------------
Interpreters are implicitly associated with channels upon ``recv()`` and
``send()`` calls. They are de-associated with ``release()`` calls. The
alternative would be explicit methods. It would be either
``add_channel()`` and ``remove_channel()`` methods on ``Interpreter``
objects or something similar on channel objects.
In practice, this level of management shouldn't be necessary for users.
So adding more explicit support would only add clutter to the API.
Use pipes instead of channels
-----------------------------
A pipe would be a simplex FIFO between exactly two interpreters. For
most use cases this would be sufficient. It could potentially simplify
the implementation as well. However, it isn't a big step to supporting
a many-to-many simplex FIFO via channels. Also, with pipes the API
ends up being slightly more complicated, requiring naming the pipes.
Use queues instead of channels
------------------------------
Queues and buffered channels are almost the same thing. The main
difference is that channels have a stronger relationship with context
(i.e. the associated interpreter).
The name "Channel" was used instead of "Queue" to avoid confusion with
the stdlib ``queue.Queue``.
"enumerate"
-----------
The ``list_all()`` function provides the list of all interpreters.
In the threading module, which partly inspired the proposed API, the
function is called ``enumerate()``. The name is different here to
avoid confusing Python users that are not already familiar with the
threading API. For them "enumerate" is rather unclear, whereas
"list_all" is clear.
Alternate solutions to prevent leaking exceptions across interpreters
---------------------------------------------------------------------
In function calls, uncaught exceptions propagate to the calling frame.
The same approach could be taken with ``run()``. However, this would
mean that exception objects would leak across the inter-interpreter
boundary. Likewise, the frames in the traceback would potentially leak.
While that might not be a problem currently, it would be a problem once
interpreters get better isolation relative to memory management (which
is necessary to stop sharing the GIL between interpreters). We've
resolved the semantics of how the exceptions propagate by raising a
``RunFailedError`` instead, for which ``__cause__`` wraps a safe proxy
for the original exception and traceback.
Rejected possible solutions:
* reproduce the exception and traceback in the original interpreter
and raise that.
* raise a subclass of RunFailedError that proxies the original
exception and traceback.
* raise RuntimeError instead of RunFailedError
* convert at the boundary (a la ``subprocess.CalledProcessError``)
(requires a cross-interpreter representation)
* support customization via ``Interpreter.excepthook``
(requires a cross-interpreter representation)
* wrap in a proxy at the boundary (including with support for
something like ``err.raise()`` to propagate the traceback).
* return the exception (or its proxy) from ``run()`` instead of
raising it
* return a result object (like ``subprocess`` does) [result-object]_
(unnecessary complexity?)
* throw the exception away and expect users to deal with unhandled
exceptions explicitly in the script they pass to ``run()``
(they can pass error info out via channels); with threads you have
to do something similar
Always associate each new interpreter with its own thread
---------------------------------------------------------
As implemented in the C-API, a subinterpreter is not inherently tied to
any thread. Furthermore, it will run in any existing thread, whether
created by Python or not. You only have to activate one of its thread
states (``PyThreadState``) in the thread first. This means that the
same thread may run more than one interpreter (though obviously
not at the same time).
The proposed module maintains this behavior. Subinterpreters are not
tied to threads. Only calls to ``Interpreter.run()`` are. However,
one of the key objectives of this PEP is to provide a more human-
centric concurrency model. With that in mind, from a conceptual
standpoint the module *might* be easier to understand if each
subinterpreter were associated with its own thread.
That would mean ``interpreters.create()`` would create a new thread
and ``Interpreter.run()`` would only execute in that thread (and
nothing else would). The benefit is that users would not have to
wrap ``Interpreter.run()`` calls in a new ``threading.Thread``. Nor
would they be in a position to accidentally pause the current
interpreter (in the current thread) while their subinterpreter
executes.
The idea is rejected because the benefit is small and the cost is high.
The difference from the capability in the C-API would be potentially
confusing. The implicit creation of threads is magical. The early
creation of threads is potentially wasteful. The inability to run
arbitrary interpreters in an existing thread would prevent some valid
use cases, frustrating users. Tying interpreters to threads would
require extra runtime modifications. It would also make the module's
implementation overly complicated. Finally, it might not even make
the module easier to understand.
Only associate interpreters upon use
------------------------------------
Associate interpreters with channel ends only once ``recv()``,
``send()``, etc. are called.
Doing this is potentially confusing and also can lead to unexpected
races where a channel is auto-closed before it can be used in the
original (creating) interpreter.
Add a "reraise" method to RunFailedError
----------------------------------------
While having ``__cause__`` set on ``RunFailedError`` helps produce a
more useful traceback, it's less helpful when handling the original
error. To help facilitate this, we could add
``RunFailedError.reraise()``. This method would enable the following
pattern::
try:
try:
interp.run(script)
except RunFailedError as exc:
exc.reraise()
except MyException:
...
This would be made even simpler if there existed a ``__reraise__``
protocol.
All that said, this is completely unnecessary. Using ``__cause__``
is good enough::
try:
try:
interp.run(script)
except RunFailedError as exc:
raise exc.__cause__
except MyException:
...
Note that in extreme cases it may require a little extra boilerplate::
try:
try:
interp.run(script)
except RunFailedError as exc:
if exc.__cause__ is not None:
raise exc.__cause__
raise # re-raise
except MyException:
...
Implementation
==============
The implementation of the PEP has 4 parts:
* the high-level module described in this PEP (mostly a light wrapper
around a low-level C extension
* the low-level C extension module
* additions to the ("private") C=API needed by the low-level module
* secondary fixes/changes in the CPython runtime that facilitate
the low-level module (among other benefits)
These are at various levels of completion, with more done the lower
you go:
* the high-level module has been, at best, roughly implemented.
However, fully implementing it will be almost trivial.
* the low-level module is mostly complete. The bulk of the
implementation was merged into master in December 2018 as the
"_xxsubinterpreters" module (for the sake of testing subinterpreter
functionality). Only 3 parts of the implementation remain:
"send_wait()", "send_buffer()", and exception propagation. All three
have been mostly finished, but were blocked by work related to ceval.
That blocker is basically resolved now and finishing the low-level
will not require extensive work.
* all necessary C-API work has been finished
* all anticipated work in the runtime has been finished
The implementation effort for PEP 554 is being tracked as part of
a larger project aimed at improving multi-core support in CPython.
[multi-core-project]_
References
==========
.. [c-api]
https://docs.python.org/3/c-api/init.html#sub-interpreter-support
.. _Communicating Sequential Processes:
.. [CSP]
https://en.wikipedia.org/wiki/Communicating_sequential_processes
https://github.com/futurecore/python-csp
.. [fifo]
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Pipe
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue
https://docs.python.org/3/library/queue.html#module-queue
http://stackless.readthedocs.io/en/2.7-slp/library/stackless/channels.html
https://golang.org/doc/effective_go.html#sharing
http://www.jtolds.com/writing/2016/03/go-channels-are-bad-and-you-should-fe…
.. [caveats]
https://docs.python.org/3/c-api/init.html#bugs-and-caveats
.. [petr-c-ext]
https://mail.python.org/pipermail/import-sig/2016-June/001062.html
https://mail.python.org/pipermail/python-ideas/2016-April/039748.html
.. [cryptography]
https://github.com/pyca/cryptography/issues/2299
.. [global-gc]
http://bugs.python.org/issue24554
.. [gilstate]
https://bugs.python.org/issue10915
http://bugs.python.org/issue15751
.. [global-atexit]
https://bugs.python.org/issue6531
.. [mp-conn]
https://docs.python.org/3/library/multiprocessing.html#connection-objects
.. [bug-rate]
https://mail.python.org/pipermail/python-ideas/2017-September/047094.html
.. [benefits]
https://mail.python.org/pipermail/python-ideas/2017-September/047122.html
.. [main-thread]
https://mail.python.org/pipermail/python-ideas/2017-September/047144.html
https://mail.python.org/pipermail/python-dev/2017-September/149566.html
.. [reset_globals]
https://mail.python.org/pipermail/python-dev/2017-September/149545.html
.. [async]
https://mail.python.org/pipermail/python-dev/2017-September/149420.html
https://mail.python.org/pipermail/python-dev/2017-September/149585.html
.. [result-object]
https://mail.python.org/pipermail/python-dev/2017-September/149562.html
.. [jython]
https://mail.python.org/pipermail/python-ideas/2017-May/045771.html
.. [multi-core-project]
https://github.com/ericsnowcurrently/multi-core-python
.. [cache-line-ping-pong]
https://mail.python.org/archives/list/python-dev@python.org/message/3HVRFWH…
.. [nathaniel-asyncio]
https://mail.python.org/archives/list/python-dev@python.org/message/TUEAZNZ…
.. [extension-docs]
https://docs.python.org/3/extending/index.html
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
[View Less]
5
9
[GvR]
> We should not try to import JavaScript's object model into Python.
Yes, I get that. Just want to point-out that working with heavily nested dictionaries (typical for JSON) is no fun with square brackets and quotation marks.
Raymond
8
10