Here’s an update to PEP 553, which makes $PYTHONBREAKPOINT a first class feature. I’ve also updated PR #3355 with the implementation to match.
Cheers,
-Barry
PEP: 553
Title: Built-in breakpoint()
Author: Barry Warsaw <barry(a)python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2017-09-05
Python-Version: 3.7
Post-History: 2017-09-05, 2017-09-07, 2017-09-13
Abstract
========
This PEP proposes adding a new built-in function called ``breakpoint()`` which
enters a Python debugger at the point of the call. Additionally, two new
names are added to the ``sys`` module to make the debugger pluggable.
Rationale
=========
Python has long had a great debugger in its standard library called ``pdb``.
Setting a break point is commonly written like this::
foo()
import pdb; pdb.set_trace()
bar()
Thus after executing ``foo()`` and before executing ``bar()``, Python will
enter the debugger. However this idiom has several disadvantages.
* It's a lot to type (27 characters).
* It's easy to typo. The PEP author often mistypes this line, e.g. omitting
the semicolon, or typing a dot instead of an underscore.
* It ties debugging directly to the choice of pdb. There might be other
debugging options, say if you're using an IDE or some other development
environment.
* Python linters (e.g. flake8 [linters]_) complain about this line because it
contains two statements. Breaking the idiom up into two lines further
complicates the use of the debugger,
These problems can be solved by modeling a solution based on prior art in
other languages, and utilizing a convention that already exists in Python.
Proposal
========
The JavaScript language provides a ``debugger`` statement [java]_ which enters
the debugger at the point where the statement appears.
This PEP proposes a new built-in function called ``breakpoint()``
which enters a Python debugger at the call site. Thus the example
above would be written like so::
foo()
breakpoint()
bar()
Further, this PEP proposes two new name bindings for the ``sys``
module, called ``sys.breakpointhook()`` and
``sys.__breakpointhook__``. By default, ``sys.breakpointhook()``
implements the actual importing and entry into ``pdb.set_trace()``,
and it can be set to a different function to change the debugger that
``breakpoint()`` enters. ``sys.__breakpointhook__`` then stashes the
default value of ``sys.breakpointhook()`` to make it easy to reset.
This exactly models the existing ``sys.displayhook()`` /
``sys.__displayhook__`` and ``sys.excepthook()`` /
``sys.__excepthook__`` [hooks]_.
The signature of the built-in is ``breakpoint(*args, **kws)``. The positional
and keyword arguments are passed straight through to ``sys.breakpointhook()``
and the signatures must match or a ``TypeError`` will be raised. The return
from ``sys.breakpointhook()`` is passed back up to, and returned from
``breakpoint()``. Since ``sys.breakpointhook()`` by default calls
``pdb.set_trace()`` by default it accepts no arguments.
Environment variable
====================
The default implementation of ``sys.breakpointhook()`` consults a new
environment variable called ``PYTHONBREAKPOINT``. This environment variable
can have various values:
* ``PYTHONBREAKPOINT=0`` disables debugging. Specifically, with this value
``sys.breakpointhook()`` returns ``None`` immediately.
* ``PYTHONBREAKPOINT= `` (i.e. the empty string). This is the same as not
setting the environment variable at all, in which case ``pdb.set_trace()``
is run as usual.
* ``PYTHONBREAKPOINT=some.importable.callable``. In this case,
``sys.breakpointhook()`` imports the ``some.importable`` module and gets the
``callable`` object from the resulting module, which it then calls. The
value may be a string with no dots, in which case it names a built-in
callable, e.g. ``PYTHONBREAKPOINT=int``. (Guido has expressed the
preference for normal Python dotted-paths, not setuptools-style entry point
syntax [syntax]_.)
This environment variable allows external processes to control how breakpoints
are handled. Some uses cases include:
* Completely disabling all accidental ``breakpoint()`` calls pushed to
production. This could be accomplished by setting ``PYTHONBREAKPOINT=0`` in
the execution environment. Another suggestion by reviewers of the PEP was
to set ``PYTHONBREAKPOINT=sys.exit`` in this case.
* IDE integration with specialized debuggers for embedded execution. The IDE
would run the program in its debugging environment with ``PYTHONBREAKPOINT``
set to their internal debugging hook.
``PYTHONBREAKPOINT`` is re-interpreted every time ``sys.breakpointhook()`` is
reached. This allows processes to change its value during the execution of a
program and have ``breakpoint()`` respond to those changes. It is not
considered a performance critical section since entering a debugger by
definition stops execution. (Of note, the implementation fast-tracks the
``PYTHONBREAKPOINT=0`` case.)
Overriding ``sys.breakpointhook`` defeats the default consultation of
``PYTHONBREAKPOINT``. It is up to the overriding code to consult
``PYTHONBREAKPOINT`` if they want.
If access to the ``PYTHONBREAKPOINT`` callable fails in any way (e.g. the
import fails, or the resulting module does not contain the callable), a
``RuntimeWarning`` is issued, and no breakpoint function is called.
Open issues
===========
Confirmation from other debugger vendors
----------------------------------------
We want to get confirmation from at least one alternative debugger
implementation (e.g. PyCharm) that the hooks provided in this PEP will
be useful to them.
Evaluation of $PYTHONBREAKPOINT
-------------------------------
There has been some mailing list discussion around this topic. The basic
behavior as described above does not appear to be controversial. Guido has
expressed a preference for ``$PYTHONBREAKPOINT`` consultation happening in
the default implementation of ``sys.breakpointhook`` [envar]_.
The one point of discussion relates to whether the value of
``$PYTHONBREAKPOINT`` should be loaded on interpreter start, and whether its
value should be cached the first time it's accessed.
It is the PEP author's opinion that the environment variable need only be
looked up at the time of use. It is also the author's opinion that the value
of the environment variable can be accessed each time ``sys.breakpointhook``
is run, to allow for maximum functional flexibility. Because this feature
enters the debugger, any performance improvements for caching will be
negligible and do not outweigh the flexibility. Further, because the special
case of ``PYTHONBREAKPOINT=0`` is fast-tracked, the no-op code path is quite
fast, and should be in the noise given the function calls of ``breakpoint()``
-> ``sys.breakpointhook()``.
Breakpoint bytecode
-------------------
Related, there has been an idea to add a bytecode that calls
``sys.breakpointhook()``. Whether built-in ``breakpoint()`` emits
this bytecode (or gets peephole optimized to the bytecode) is an open
issue. The bytecode is useful for debuggers that actively modify
bytecode streams to trampoline into their own debugger. Having a
"breakpoint" bytecode might allow them to avoid bytecode modification
in order to invoke this trampoline. *NOTE*: It probably makes sense to split
this idea into a separate PEP.
Call a fancier object by default
--------------------------------
Some folks want to be able to use other ``pdb`` interfaces such as
``pdb.pm()``. Although this is a less commonly used API, it could be
supported by binding ``sys.breakpointhook`` to an object that implements
``__call__()``. Calling this object would call ``pdb.set_trace()``, but the
object could expose other methods, such as ``pdb.pm()``, making invocation of
it as handy as ``breakpoint.pm()``.
Implementation
==============
A pull request exists with the proposed implementation [impl]_.
Rejected alternatives
=====================
A new keyword
-------------
Originally, the author considered a new keyword, or an extension to an
existing keyword such as ``break here``. This is rejected on several fronts.
* A brand new keyword would require a ``__future__`` to enable it since almost
any new keyword could conflict with existing code. This negates the ease
with which you can enter the debugger.
* An extended keyword such as ``break here``, while more readable and not
requiring a ``__future__`` would tie the keyword extension to this new
feature, preventing more useful extensions such as those proposed in
PEP 548.
* A new keyword would require a modified grammar and likely a new bytecode.
Each of these makes the implementation more complex. A new built-in breaks
no existing code (since any existing module global would just shadow the
built-in) and is quite easy to implement.
sys.breakpoint()
----------------
Why not ``sys.breakpoint()``? Requiring an import to invoke the debugger is
explicitly rejected because ``sys`` is not imported in every module. That
just requires more typing and would lead to::
import sys; sys.breakpoint()
which inherits several of the problems this PEP aims to solve.
Version History
===============
* 2017-09-13
* The ``PYTHONBREAKPOINT`` environment variable is made a first class
feature.
* 2017-09-07
* ``debug()`` renamed to ``breakpoint()``
* Signature changed to ``breakpoint(*args, **kws)`` which is passed straight
through to ``sys.breakpointhook()``.
References
==========
.. [linters]
http://flake8.readthedocs.io/en/latest/
.. [java]
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statement…
.. [hooks]
https://docs.python.org/3/library/sys.html#sys.displayhook
.. [syntax]
http://setuptools.readthedocs.io/en/latest/setuptools.html?highlight=consol…
.. [impl]
https://github.com/python/cpython/pull/3355
.. [envar]
https://mail.python.org/pipermail/python-dev/2017-September/149447.html
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
The Python build factories have been busy the last several weeks preparing
our fall lineup of releases. Today we are happy to announce three
additions: 3.6.3rc1, 3.7.0a1, and 3.3.7 final, which join last weekend's
2.7.14 and last month's 3.5.4 bug-fix releases and 3.4.7 security-fix
update (https://www.python.org/downloads/).
1. Python 3.6.3rc1 is the first release candidate for Python 3.6.3, the next
maintenance release of Python 3.6. While 3.6.3rc1 is a preview release and,
thus, not intended for production environments, we encourage you to explore
it and provide feedback via the Python bug tracker (https://bugs.python.org).
3.6.3 is planned for final release on 2017-10-02 with the next maintenance
release expected to follow in about 3 months. You can find Python 3.6.3rc1
and more information here:
https://www.python.org/downloads/release/python-363rc1/
2. Python 3.7.0a1 is the first of four planned alpha releases of Python 3.7,
the next feature release of Python. During the alpha phase, Python 3.7
remains under heavy development: additional features will be added
and existing features may be modified or deleted. Please keep in mind
that this is a preview release and its use is not recommended for
production environments. The next preview release, 3.6.0a2, is planned
for 2017-10-16. You can find Python 3.7.0a1 and more information here:
https://www.python.org/downloads/release/python-370a1/
3. Python 3.3.7 is also now available. It is a security-fix source-only
release and is expected to be the final release of any kind for Python
3.3.x before it reaches end-of-life status on 2017-09-29, five years after
its initial release. Because 3.3.x has long been in security-fix mode,
3.3.7 may no longer build correctly on all current operating system
releases and some tests may fail. If you are still using Python 3.3.x,
we **strongly** encourage you to upgrade now to a more recent, fully
supported version of Python 3. You can find Python 3.3.7 here:
https://www.python.org/downloads/release/python-337/
--
Ned Deily
nad(a)python.org -- []
The syntax for decorators in the grammar is quite specific:
decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
Decorators can be either a dotted name, or a dotted name followed by
arguments. This disallows:
@mapping['async'] # looking something up in a mapping
@func(1, 2, 3).async # getting the attribute of the /result/ of a
function call
Obviously these restrictions haven't been too burdensome, as I've
labored under them for 13+ years and didn't even notice until just now.
But it set me to wondering.
If it were me, perpetually lazy sod that I am, I'd have written the
grammar as
decorator: '@' expr NEWLINE
and been done with it.
So why don't decorators allow arbitrary expressions? The PEP discusses
the syntax for decorators, but that whole debate only concerned itself
with where the decorator goes relative to "def", and what funny
punctuation might it use. It never says "decorators shouldn't permit
arbitrary expressions because--". Nor is there any info on wiki page
with the extensive summary of alternative syntax proposals.
Anybody remember?
I'm not proposing that we allow arbitrary expressions as decorators...
well, I'm not doing that /yet/ at least. But like I said, the syntax
has been this way for 13 years and I don't recall anybody complaining.
//arry
/p.s. I experimentally tried modifying the grammar for descriptors to
allow arbitrary expressions. The syntax was fine but it crashed in
ast_for_decorator(). When I updated that to match, it worked fine! The
diff is short and mainly consists of deleted code.
I've updated the PEP in response to some excellent feedback. Thanks
to all who helped. Most notably, I've added a basic object passing
mechanism. I've included the PEP below. Please let me know what you
think. Thanks!
-eric
****************************************************
PEP: 554
Title: Multiple Interpreters in the Stdlib
Author: Eric Snow <ericsnowcurrently(a)gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2017-09-05
Python-Version: 3.7
Post-History:
Abstract
========
This proposal introduces the stdlib ``interpreters`` module. It exposes
the basic functionality of subinterpreters that already exists in the
C-API. Each subinterpreter runs with its own state (see
``Interpreter Isolation`` below). The module will be "provisional", as
described by PEP 411.
Rationale
=========
Running code in multiple interpreters provides a useful level of
isolation within the same process. This can be leveraged in number
of ways. Furthermore, subinterpreters provide a well-defined framework
in which such isolation may extended.
CPython has supported subinterpreters, with increasing levels of
support, since version 1.5. While the feature has the potential
to be a powerful tool, subinterpreters have suffered from neglect
because they are not available directly from Python. Exposing the
existing functionality in the stdlib will help reverse the situation.
This proposal is focused on enabling the fundamental capability of
multiple isolated interpreters in the same Python process. This is a
new area for Python so there is relative uncertainly about the best
tools to provide as companions to subinterpreters. Thus we minimize
the functionality we add in the proposal as much as possible.
Concerns
--------
* "subinterpreters are not worth the trouble"
Some have argued that subinterpreters do not add sufficient benefit
to justify making them an official part of Python. Adding features
to the language (or stdlib) has a cost in increasing the size of
the language. So it must pay for itself. In this case, subinterpreters
provide a novel concurrency model focused on isolated threads of
execution. Furthermore, they present an opportunity for changes in
CPython that will allow simulateous use of multiple CPU cores (currently
prevented by the GIL).
Alternatives to subinterpreters include threading, async, and
multiprocessing. Threading is limited by the GIL and async isn't
the right solution for every problem (nor for every person).
Multiprocessing is likewise valuable in some but not all situations.
Direct IPC (rather than via the multiprocessing module) provides
similar benefits but with the same caveat.
Notably, subinterpreters are not intended as a replacement for any of
the above. Certainly they overlap in some areas, but the benefits of
subinterpreters include isolation and (potentially) performance. In
particular, subinterpreters provide a direct route to an alternate
concurrency model (e.g. CSP) which has found success elsewhere and
will appeal to some Python users. That is the core value that the
``interpreters`` module will provide.
* "stdlib support for subinterpreters adds extra burden
on C extension authors"
In the ``Interpreter Isolation`` section below we identify ways in
which isolation in CPython's subinterpreters is incomplete. Most
notable is extension modules that use C globals to store internal
state. PEP 3121 and PEP 489 provide a solution for most of the
problem, but one still remains. [petr-c-ext]_ Until that is resolved,
C extension authors will face extra difficulty to support
subinterpreters.
Consequently, projects that publish extension modules may face an
increased maintenance burden as their users start using subinterpreters,
where their modules may break. This situation is limited to modules
that use C globals (or use libraries that use C globals) to store
internal state.
Ultimately this comes down to a question of how often it will be a
problem in practice: how many projects would be affected, how often
their users will be affected, what the additional maintenance burden
will be for projects, and what the overall benefit of subinterpreters
is to offset those costs. The position of this PEP is that the actual
extra maintenance burden will be small and well below the threshold at
which subinterpreters are worth it.
Proposal
========
The ``interpreters`` module will be added to the stdlib. It will
provide a high-level interface to subinterpreters and wrap the low-level
``_interpreters`` module. The proposed API is inspired by the
``threading`` module.
The module provides the following functions:
``list()``::
Return a list of all existing interpreters.
``get_current()``::
Return the currently running interpreter.
``get_main()``::
Return the main interpreter.
``create()``::
Initialize a new Python interpreter and return it. The
interpreter will be created in the current thread and will remain
idle until something is run in it.
The module also provides the following classes:
``Interpreter(id)``::
id:
The interpreter's ID (read-only).
is_running():
Return whether or not the interpreter is currently executing code.
destroy():
Finalize and destroy the interpreter. If called on a running
interpreter then raise RuntimeError in the interpreter that
called ``destroy()``.
run(code):
Run the provided Python code in the interpreter, in the current
OS thread. If the interpreter is already running then raise
RuntimeError in the interpreter that called ``run()``.
The current interpreter (which called ``run()``) will block until
the subinterpreter finishes running the requested code. Any
uncaught exception in that code will bubble up to the current
interpreter.
Supported code: source text.
get_fifo(name):
Return the FIFO object with the given name that is associated
with this interpreter. If no such FIFO exists then raise
KeyError. The FIFO will be either a "FIFOReader" or a
"FIFOWriter", depending on which "add_*_fifo()" was called.
list_fifos():
Return a list of all fifos associated with the interpreter.
add_recv_fifo(name=None):
Create a new FIFO, associate the two ends with the involved
interpreters, and return the side associated with the interpreter
in which "add_recv_fifo()" was called. A FIFOReader gets tied to
this interpreter. A FIFOWriter gets tied to the interpreter that
called "add_recv_fifo()".
The FIFO's name is set to the provided value. If no name is
provided then a dynamically generated one is used. If a FIFO
with the given name is already associated with this interpreter
(or with the one in which "add_recv_fifo()" was called) then raise
KeyError.
add_send_fifo(name=None):
Create a new FIFO, associate the two ends with the involved
interpreters, and return the side associated with the interpreter
in which "add_recv_fifo()" was called. A FIFOWriter gets tied to
this interpreter. A FIFOReader gets tied to the interpreter that
called "add_recv_fifo()".
The FIFO's name is set to the provided value. If no name is
provided then a dynamically generated one is used. If a FIFO
with the given name is already associated with this interpreter
(or with the one in which "add_send_fifo()" was called) then raise
KeyError.
remove_fifo(name):
Drop the association between the named FIFO and this interpreter.
If the named FIFO is not found then raise KeyError.
``FIFOReader(name)``::
The receiving end of a FIFO. An interpreter may use this to receive
objects from another interpreter. At first only bytes and None will
be supported.
name:
The FIFO's name.
__next__():
Return the next bytes object from the pipe. If none have been
pushed on then block.
pop(*, block=True):
Return the next bytes object from the pipe. If none have been
pushed on and "block" is True (the default) then block.
Otherwise return None.
``FIFOWriter(name)``::
The sending end of a FIFO. An interpreter may use this to send
objects to another interpreter. At first only bytes and None will
be supported.
name:
The FIFO's name.
push(object, *, block=True):
Add the object to the FIFO. If "block" is true then block
until the object is popped off. If the FIFO does not support
the object's type then TypeError is raised.
About FIFOs
-----------
Subinterpreters are inherently isolated (with caveats explained below),
in contrast to threads. This enables a different concurrency model than
currently exists in Python. CSP (Communicating Sequential Processes),
upon which Go's concurrency is based, is one example of this model.
A key component of this approach to concurrency is message passing. So
providing a message/object passing mechanism alongside ``Interpreter``
is a fundamental requirement. This proposal includes a basic mechanism
upon which more complex machinery may be built. That basic mechanism
draws inspiration from pipes, queues, and CSP's channels.
The key challenge here is that sharing objects between interpreters
faces complexity due in part to CPython's current memory model.
Furthermore, in this class of concurrency, the ideal is that objects
only exist in one interpreter at a time. However, this is not practical
for Python so we initially constrain supported objects to ``bytes`` and
``None``. There are a number of strategies we may pursue in the future
to expand supported objects and object sharing strategies.
Note that the complexity of object sharing increases as subinterpreters
become more isolated, e.g. after GIL removal. So the mechanism for
message passing needs to be carefully considered. Keeping the API
minimal and initially restricting the supported types helps us avoid
further exposing any underlying complexity to Python users.
Deferred Functionality
======================
In the interest of keeping this proposal minimal, the following
functionality has been left out for future consideration. Note that
this is not a judgement against any of said capability, but rather a
deferment. That said, each is arguably valid.
Interpreter.call()
------------------
It would be convenient to run existing functions in subinterpreters
directly. ``Interpreter.run()`` could be adjusted to support this or
a ``call()`` method could be added::
Interpreter.call(f, *args, **kwargs)
This suffers from the same problem as sharing objects between
interpreters via queues. The minimal solution (running a source string)
is sufficient for us to get the feature out where it can be explored.
timeout arg to pop() and push()
-------------------------------
Typically functions that have a ``block`` argument also have a
``timeout`` argument. We can add it later if needed.
Interpreter Isolation
=====================
CPython's interpreters are intended to be strictly isolated from each
other. Each interpreter has its own copy of all modules, classes,
functions, and variables. The same applies to state in C, including in
extension modules. The CPython C-API docs explain more. [c-api]_
However, there are ways in which interpreters share some state. First
of all, some process-global state remains shared, like file descriptors.
There are no plans to change this.
Second, some isolation is faulty due to bugs or implementations that did
not take subinterpreters into account. This includes things like
at-exit handlers and extension modules that rely on C globals. In these
cases bugs should be opened (some are already).
Finally, some potential isolation is missing due to the current design
of CPython. This includes the GIL and memory management. Improvements
are currently going on to address gaps in this area.
Provisional Status
==================
The new ``interpreters`` module will be added with "provisional" status
(see PEP 411). This allows Python users to experiment with the feature
and provide feedback while still allowing us to adjust to that feedback.
The module will be provisional in Python 3.7 and we will make a decision
before the 3.8 release whether to keep it provisional, graduate it, or
remove it.
References
==========
.. [c-api]
https://docs.python.org/3/c-api/init.html#bugs-and-caveats
.. [petr-c-ext]
https://mail.python.org/pipermail/import-sig/2016-June/001062.htmlhttps://mail.python.org/pipermail/python-ideas/2016-April/039748.html
Copyright
=========
This document has been placed in the public domain.
I'm happy to announce to the immediate availability of Python 2.7.14,
yet another bug fix release in the Python 2.7 series. 2.7.14 includes 9
months of conservative bug fixes from the 3.x branch.
Downloads of source code and binaries are at:
https://www.python.org/downloads/release/python-2714/
Bugs may be reported at
https://bugs.python.org/
Warmly,
Benjamin
2.7 release manager
(on behalf of all of 2.7's contributors)
There are times when you deal with completely independent input/output
'pipes' - where parallelizing would really help speed things up.
Can't there be a way to capture that idiom and multi thread it in the
language itself?
Example:
loop:
read an XML
produce a JSON like
Regards,
MB
I know there's a lot of excitement around lots of new ideas. And the 3.7
feature freeze is looming (January is in a few months). But someone has to
review and accept all those PEPs, and I can't do it all by myself.
If you want your proposal to be taken seriously, you need to include a
summary of the discussion on the mailing list (including objections, even
if you disagree!) in your PEP, e.g. as an extended design rationale or
under the Rejected Ideas heading.
If you don't do this you risk having to repeat yourself -- also you risk
having your PEP rejected, because at this point there's no way I am going
to read all the discussions.
--
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
Lots of releases coming up soon!
2017-09-16:
- Python 2.7.14 final release
2017-09-18 1200 UTC cutoff:
- Python 3.6.3 rc1
- Python 3.3.7 final release (following by retirement)
If you know of any issues that you feel need to be addressed in these releases, please make sure there is an open issue on the bug tracker set to "release blocker".
Also on 2017-09-18:
- Python 3.7.0 alpha 1
This will be the first preview snapshot of 3.7.0. It's still very early in the development cycle for 3.7. The main purpose of early alpha releases is to exercise the build and release process and to make it easier for Windows and Macs users to help test. Many new features and build changes are yet to appear in subsequent releases prior to feature code cutoff on 2018-01-29 at 3.7.0b1.
Thanks for your help!
--Ned
--
Ned Deily
nad(a)python.org -- []
Hi,
In March 2015, I proposed the PEP 490 to chain implicitly C exceptions
by default:
https://www.python.org/dev/peps/pep-0490/
The discussion on python-dev was quickly in favor of keeping the
status quo: don't chain C exceptions by default, but only do that
explicitly where it makes sense.
I forgot this PEP, but I just rejected it to close the old discussion :-)
Note: The PEP is not yet rejected on python.org, it will be done at
the next cron job run.
Victor
