On 11/18/2016 11:34 AM, Chris Angelico wrote:
> Hmm. I'm seeing them. Where are you reading -ideas? Is there possibly
> a glitch between mailing list and newsgroup?
Yeah, I'm using the mailing list.
On 11/18/2016 11:39 AM, Mikhail V wrote:
> I am really sorry :( Didn't intended it
> Something with my mailer it showed me python-ideas in to: in Stevens reply.
Looks like it's not an issue with you, Mikhail, my apologies.
I'll ask Brett to check it out.
--
~Ethan~
Hi!
Why am I bringing this up:
Security is hard !
Implementing a standard correctly is not easy.
I know about the later because the last 2 years I’ve been involved in
certifying OpenID Connect Provider instances. Lately I’ve been doing the
same for OpenID Connect Relying Party libraries.
All of what I’ve done in written in Python and on Github.
Regarding the first opinion that has been shown time and time again so
I won’t go into that here.
Now, voices has been raise within the OpenID Foundation that it would pick
a number of implementations, one per language, and stamp them with
a sign of approval. Those implementations would all be thoroughly tested
for compliance and usability before approved.
My Python implementation (https://github.com/rohe/pyoidc) is probably the forerunner when it comes to being
the chosen Python implementation. It’s been around for a number of years and it’s the
basis for the test tools. Which means, it has been thoroughly tested by many independent parties.
My question to you is if it would be possible to get an OAuth2/OIDC implementation like mine
to be part of the Python standard distribution.
I realise that I will have to rewrite parts of pyoidc because presently it uses modules
(for instance pycryptdome and requests) that are not part of the standard distribution.
The bottom line is of course that it would benefit the community to have a
high quality OAuth2/OIDC implementation within easy reach.
— Roland
On 15 November 2016 at 10:46, Paul Moore <p.f.moore(a)gmail.com> wrote:
> If you're proposing a = a + b to introspect at runtime the type of a,
> and produce different bytecode depending on the answer, you're
> proposing a fundamental change to the runtime semantics of Python
> (such that the resulting language is arguably not even Python any
> more). That's not to say it can't be done, just that it's not in scope
> for "ideas about new features for Python" in any practical sense.
I thank you all again for informing me about assignment operator
and its further consequences. I read accurately and try to absorb
the information, still however I feel like I need just a little bit of more
understanding in this particular case.
So I can do better if we imagine, just imagine, such a scenario:
1. += and friends are now not allowed for integers,
but all _add()_, _iadd()_ are still of course there,
it does not concern me.
2. Somebody says: now I want to write something like:
halfofanextremelylongvariable = halfofanextremelylongvariable + 1
3. I come up, scratch my head and say: Ok, now we can write that as:
halfofanextremelylongvariable = self + 1
So, considering previous comments, there was an impression
that I wanted to shake the fundament of all Python world or something.
But is it necesserily so? So I don't know how the Python parser
works exactly, but why cannot the above roughly speaking "translated", e.g.:
x = self + 1
would mean exactly the same that was before
x += 1
and call __iadd()__ in turn. Isn't it a very simple thing technically seen?
Since augmented assignment is always applied to exactly one argument
I believe (correct me if I am wrong), would not it be simple and
non-ambigios operation at parsing stage?
So in other words sequence " = self + " is same as sequence " += " was.
Another example, lets take a Numpy array for example.
So it was also stated that += is vital for doing in-place sum.
Now I say, we write it in such way, having "A+=1" in mind:
exec A + 1
since again, += can be only applied only on one operand, cannot I
just say that a sequence "exec {1} + {2}" is what sequence "{1}+={2}" was?
What semantics it will fundamentally break or so hard to implement?
I was actually trying to give similar ideas in previous comments,
but I was not precise in my ideas, so it came
to misunderstanding, or it can be that I'm still missing
something very important here.
Mikhail
Hello,
Following is my code :
#!/usr/bin/python
import socket
import struct
import binascii
rawSocket = socket.socket(socket.PF_PACKET,socket.SOCK_RAW,socket.htons(0x0800))
# use 0x0800 for IPv4 packets , 0x0003 is for sniffing all kinds of packets
while True:
pkt= rawSocket.recvfrom(2048)
ethernetHeader = pkt[0][0:14]
pr = unicode(ethernetHeader, errors='replace')
print pr
eth_hdr = struct.unpack("!6s6s2s",ethernetHeader)
print "Source MAC Address :" , binascii.hexlify(eth_hdr[1])
print "Destination MAC Address : " , binascii.hexlify(eth_hdr[0])
print "Protocol : " , binascii.hexlify(eth_hdr[2])
ipHeader = pkt[0][14:34]
ip_hdr = struct.unpack("!12s4s4s",ipHeader)
print "Source ip ADDRESS : " + socket.inet_ntoa(ip_hdr[1])
print "Destination IP Address: " + socket.inet_ntoa(ip_hdr[2])
# initial part of the tcp header
tcpHeader = pkt[0][34:54]
tcp_hdr = struct.unpack("!HH16s",tcpHeader)
print "Source Port ADDRESS : " ,tcp_hdr[0]
print "Destination Port ADDRESS : " , tcp_hdr[1]
Issues :
1. Unable to capture any outgoing IPv4 traffic. I ran the sniff()
method in Scapy and it does capture the outgoing packets.
2. I am NOT USING PROMISCUOUS MODE , still most of the packes I am
receiving neither have my IP or MAC in either of the source or
destination fields.
3. Captured data is different from the one observed using Scapy or Wireshark.
Request you to kindly clarify these observations.
Thanks and Regards,
Ayush
Hi folks,
As promised, here's a follow-up to the withdrawn PEP 531 that focuses
on coming up with a common protocol driven solution to conditional
evaluation of subexpressions that also addresses the element-wise
comparison chaining problem that Guido noted when rejecting PEP 335.
I quite like how it came out, but see the "Risks & Concerns" section
for a discussion of an internal inconsistency it would introduce into
the language if accepted as currently written, and the potentially
far-reaching consequences actually resolving that inconsistency might
have on the way people write their Python code (if the PEP was
subsequently approved).
Regards,
Nick.
=================================
PEP: 532
Title: A circuit breaking operator and protocol
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan(a)gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 30-Oct-2016
Python-Version: 3.7
Post-History: 5-Nov-2016
Abstract
========
Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this PEP
proposes the addition of a new protocol-driven circuit breaking operator to
Python that allows the left operand to decide whether or not the expression
should short circuit and return a result immediately, or else continue
on with evaluation of the right operand::
exists(foo) else bar
missing(foo) else foo.bar()
These two expressions can be read as:
* "the expression result is 'foo' if it exists, otherwise it is 'bar'"
* "the expression result is 'foo' if it is missing, otherwise it is 'foo.bar()'"
Execution of these expressions relies on a new circuit breaking protocol that
implicitly avoids repeated evaluation of the left operand while letting
that operand fully control the result of the expression, regardless of whether
it skips evaluation of the right operand or not::
_lhs = LHS
type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS)
To properly support logical negation of circuit breakers, a new ``__not__``
protocol method would also be introduced allowing objects to control
the result of ``not obj`` expressions.
As shown in the basic example above, the PEP further proposes the addition of
builtin ``exists`` and ``missing`` circuit breakers that provide conditional
branching based on whether or not an object is ``None``, but return the
original object rather than the existence checking wrapper when the expression
evaluation short circuits.
In addition to being usable as simple boolean operators (e.g. as in
``assert all(exists, items)`` or ``if any(missing, items):``), these circuit
breakers will allow existence checking fallback operations (aka None-coalescing
operations) to be written as::
value = exists(expr1) else exists(expr2) else expr3
and existence checking precondition operations (aka None-propagating
or None-severing operations) to be written as::
value = missing(obj) else obj.field.of.interest
value = missing(obj) else obj["field"]["of"]["interest"]
A change to the definition of chained comparisons is also proposed, where
the comparison chaining will be updated to use the circuit breaking operator
rather than the logical disjunction (``and``) operator if the left hand
comparison returns a circuit breaker as its result.
While there are some practical complexities arising from the current handling
of single-valued arrays in NumPy, this change should be sufficient to allow
elementwise chained comparison operations for matrices, where the result
is a matrix of boolean values, rather than tautologically returning ``True``
or raising ``ValueError``.
Relationship with other PEPs
============================
This PEP is a direct successor to PEP 531, replacing the existence checking
protocol and the new ``?then`` and ``?else`` syntactic operators defined there
with a single protocol driven ``else`` operator and adjustments to the ``not``
operator. The existence checking use cases are taken from that PEP.
It is also a direct successor to PEP 335, which proposed the ability to
overload the ``and`` and ``or`` operators directly, with the ability to
overload the semantics of comparison chaining being one of the consequences
of that change. The proposal in this PEP to instead handle the element-wise
comparison use case by changing the semantic definition of comparison chaining
is drawn from Guido's rejection of PEP 335.
This PEP competes with the dedicated null-coalescing operator in PEP 505,
proposing that improved support for null-coalescing operations be offered
through a more general protocol-driven short circuiting operator and related
builtins, rather than through a dedicated single-purpose operator.
It doesn't compete with PEP 505's proposed shorthands for existence checking
attribute access and subscripting, but instead offers an alternative underlying
semantic framework for defining them:
* ``EXPR?.attr`` would be syntactic sugar for ``missing(EXPR) else EXPR.attr``
* ``EXPR?[key]`` would be syntactic sugar for ``missing(EXPR) else EXPR[key]``
In both cases, the dedicated syntactic form could be optimised to avoid
actually creating the circuit breaker instance.
Specification
=============
The circuit breaking operator (``else``)
----------------------------------------
Circuit breaking expressions would be written using ``else`` as a new binary
operator, akin to the existing ``and`` and ``or`` logical operators::
LHS else RHS
Ignoring the hidden variable assignment, this is semantically equivalent to::
_lhs = LHS
type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS)
The key difference relative to the existing ``or`` operator is that the value
determining which branch of the conditional expression gets executed *also*
gets a chance to postprocess the results of the expressions on each of the
branches.
As part of the short-circuiting behaviour, interpreter implementations
are expected to access only the protocol method needed for the branch
that is actually executed, but it is still recommended that circuit
breaker authors that always return ``True`` or always return ``False`` from
``__bool__`` explicitly raise ``NotImplementedError`` with a suitable
message from branch methods that are never expected to be executed (see the
comparison chaining use case in the Rationale section below for an example
of that).
It is proposed that the ``else`` operator use a new precedence level that binds
less tightly than the ``or`` operator by adjusting the relevant line in
Python's grammar from the current::
test: or_test ['if' or_test 'else' test] | lambdef
to instead be::
test: else_test ['if' or_test 'else' test] | lambdef
else_test: or_test ['else' test]
The definition of ``test_nocond`` would remain unchanged, so circuit
breaking expressions would require parentheses when used in the ``if``
clause of comprehensions and generator expressions just as conditional
expressions themselves do.
This grammar definition means precedence/associativity in the otherwise
ambiguous case of ``expr1 if cond else expr2 else epxr3`` resolves as
``(expr1 if cond else expr2) else epxr3``.
A guideline will also be added to PEP 8 to say "don't do that", as such a
construct will be inherently confusing for readers, regardless of how the
interpreter executes it.
Overloading logical inversion (``not``)
---------------------------------------
Any circuit breaker definition will have a logical inverse that is still a
circuit breaker, but inverts the answer as to whether or not to short circuit
the expression evaluation. For example, the ``exists`` and ``missing`` circuit
breakers proposed in this PEP are each other's logical inverse.
A new protocol method, ``__not__(self)``, will be introduced to permit circuit
breakers and other types to override ``not`` expressions to return their
logical inverse rather than a coerced boolean result.
To preserve the semantics of existing language optimisations, ``__not__``
implementations will be obliged to respect the following invariant::
assert not bool(obj) == bool(not obj)
Chained comparisons
-------------------
A chained comparison like ``0 < x < 10`` written as::
LEFT_BOUND left_op EXPR right_op RIGHT_BOUND
is currently roughly semantically equivalent to::
_expr = EXPR
_lhs_result = LEFT_BOUND left_op _expr
_expr_result = _lhs_result and (_expr right_op RIGHT_BOUND)
This PEP proposes that this be changed to explicitly check if the left
comparison returns a circuit breaker, and if so, use ``else`` rather than
``and`` to implement the comparison chaining::
_expr = EXPR
_lhs_result = LEFT_BOUND left_op _expr
if hasattr(type(_lhs_result), "__then__"):
_expr_result = _lhs_result else (_expr right_op RIGHT_BOUND)
else:
_expr_result = _lhs_result and (_expr right_op RIGHT_BOUND)
This allows types like NumPy arrays to control the behaviour of chained
comparisons by returning circuit breakers from comparison operations.
Existence checking comparisons
------------------------------
Two new builtins implementing the new protocol are proposed to encapsulate the
notion of "existence checking": seeing if a value is ``None`` and either
falling back to an alternative value (an operation known as "None-coalescing")
or passing it through as the result of the overall expression (an operation
known as "None-severing" or "None-propagating").
These builtins would be defined as follows::
class CircuitBreaker:
"""Base circuit breaker type (available as types.CircuitBreaker)"""
def __init__(self, value, condition, inverse_type):
self.value = value
self._condition = condition
self._inverse_type = inverse_type
def __bool__(self):
return self._condition
def __not__(self):
return self._inverse_type(self.value)
def __then__(self):
return self.value
def __else__(self, other):
if other is self:
return self.value
return other
class exists(types.CircuitBreaker):
"""Circuit breaker for 'EXPR is not None' checks"""
def __init__(self, value):
super().__init__(value, value is not None, missing)
class missing(types.CircuitBreaker):
"""Circuit breaker for 'EXPR is None' checks"""
def __init__(self, value):
super().__init__(value, value is None, exists)
Aside from changing the definition of ``__bool__`` to be based on
``is not None`` rather than normal truth checking, the key characteristic of
``exists`` is that when it is used as a circuit breaker, it is *ephemeral*:
when it is told that short circuiting has taken place, it returns the original
value, rather than the existence checking wrapper.
``missing`` is defined as the logically inverted counterpart of ``exists``:
``not exists(obj)`` is semantically equivalent to ``missing(obj)``.
The ``__else__`` implementations for both builtin circuit breakers are defined
such that the wrapper will always be removed even if you explicitly pass the
circuit breaker to both sides of the ``else`` expression::
breaker = exists(foo)
assert (breaker else breaker) is foo
breaker = missing(foo)
assert (breaker else breaker) is foo
Other conditional constructs
----------------------------
No changes are proposed to if statements, while statements, conditional
expressions, comprehensions, or generator expressions, as the boolean clauses
they contain are already used for control flow purposes.
However, it's worth noting that while such proposals are outside the scope of
this PEP, the circuit breaking protocol defined here would be sufficient to
support constructs like::
while exists(dynamic_query()) as result:
... # Code using result
and:
if exists(re.search(pattern, text)) as match:
... # Code using match
Leaving the door open to such a future extension is the main reason for
recommending that circuit breaker implementations handle the ``self is other``
case in ``__else__`` implementations the same way as they handle the
short-circuiting behaviour in ``__then__``.
Style guide recommendations
---------------------------
The following additions to PEP 8 are proposed in relation to the new features
introduced by this PEP:
* In the absence of other considerations, prefer the use of the builtin
circuit breakers ``exists`` and ``missing`` over the corresponding
conditional expressions
* Do not combine conditional expressions (``if-else``) and circuit breaking
expressions (the ``else`` operator) in a single expression - use one or the
other depending on the situation, but not both.
Rationale
=========
Adding a new operator
---------------------
Similar to PEP 335, early drafts of this PEP focused on making the existing
``and`` and ``or`` operators less rigid in their interpretation, rather than
proposing new operators. However, this proved to be problematic for a few
reasons:
* defining a shared protocol for both ``and`` and ``or`` was confusing, as
``__then__`` was the short-circuiting outcome for ``or``, while ``__else__``
was the short-circuiting outcome for ``and``
* the ``and`` and ``or`` operators have a long established and stable meaning,
so readers would inevitably be surprised if their meaning now became
dependent on the type of the left operand. Even new users would be confused
by this change due to 25+ years of teaching material that assumes the
current well-known semantics for these operators
* Python interpreter implementations, including CPython, have taken advantage
of the existing semantics of ``and`` and ``or`` when defining runtime and
compile time optimisations, which would all need to be reviewed and
potentially discarded if the semantics of those operations changed
Proposing a single new operator instead resolves all of those issues -
``__then__`` always indicates short circuiting, ``__else__`` only indicates
"short circuiting" if the circuit breaker itself is also passed in as the
right operand, and the semantics of ``and`` and ``or`` remain entirely
unchanged. While the semantics of the unary ``not`` operator do change, the
invariant required of ``__not__`` implementations means that existing
expression optimisations in boolean contexts will remain valid.
As a result of that design simplification, the new protocol and operator would
even allow us to expose ``operator.true`` and ``operator.false``
as circuit breaker definitions if we chose to do so::
class true(types.CircuitBreaker):
"""Circuit breaker for 'bool(EXPR)' checks"""
def __init__(self, value):
super().__init__(value, bool(value), when_false)
class false(types.CircuitBreaker):
"""Circuit breaker for 'not bool(EXPR)' checks"""
def __init__(self, value):
super().__init__(value, not bool(value), when_true)
Given those circuit breakers:
* ``LHS or RHS`` would be roughly ``operator.true(LHS) else RHS``
* ``LHS and RHS`` would be roughly ``operator.false(LHS) else RHS``
Naming the operator and protocol
--------------------------------
The names "circuit breaking operator", "circuit breaking protocol" and
"circuit breaker" are all inspired by the phrase "short circuiting operator":
the general language design term for operators that only conditionally
evaluate their right operand.
The electrical analogy is that circuit breakers in Python detect and handle
short circuits in expressions before they trigger any exceptions similar to the
way that circuit breakers detect and handle short circuits in electrical
systems before they damage any equipment or harm any humans.
The Python level analogy is that just as a ``break`` statement lets you
terminate a loop before it reaches its natural conclusion, a circuit breaking
expression lets you terminate evaluation of the expression and produce a result
immediately.
Using an existing keyword
-------------------------
Using an existing keyword has the benefit of allowing the new expression to
be introduced without a ``__future__`` statement.
``else`` is semantically appropriate for the proposed new protocol, and the
only syntactic ambiguity introduced arises when the new operator is combined
with the explicit ``if-else`` conditional expression syntax.
Element-wise chained comparisons
--------------------------------
In ultimately rejecting PEP 335, Guido van Rossum noted [1_]:
The NumPy folks brought up a somewhat separate issue: for them,
the most common use case is chained comparisons (e.g. A < B < C).
To understand this observation, we first need to look at how comparisons work
with NumPy arrays::
>>> import numpy as np
>>> increasing = np.arange(5)
>>> increasing
array([0, 1, 2, 3, 4])
>>> decreasing = np.arange(4, -1, -1)
>>> decreasing
array([4, 3, 2, 1, 0])
>>> increasing < decreasing
array([ True, True, False, False, False], dtype=bool)
Here we see that NumPy array comparisons are element-wise by default, comparing
each element in the lefthand array to the corresponding element in the righthand
array, and producing a matrix of boolean results.
If either side of the comparison is a scalar value, then it is broadcast across
the array and compared to each individual element::
>>> 0 < increasing
array([False, True, True, True, True], dtype=bool)
>>> increasing < 4
array([ True, True, True, True, False], dtype=bool)
However, this broadcasting idiom breaks down if we attempt to use chained
comparisons::
>>> 0 < increasing < 4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element
is ambiguous. Use a.any() or a.all()
The problem is that internally, Python implicitly expands this chained
comparison into the form::
>>> 0 < increasing and increasing < 4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element
is ambiguous. Use a.any() or a.all()
And NumPy only permits implicit coercion to a boolean value for single-element
arrays where ``a.any()`` and ``a.all()`` can be assured of having the same
result::
>>> np.array([False]) and np.array([False])
array([False], dtype=bool)
>>> np.array([False]) and np.array([True])
array([False], dtype=bool)
>>> np.array([True]) and np.array([False])
array([False], dtype=bool)
>>> np.array([True]) and np.array([True])
array([ True], dtype=bool)
The proposal in this PEP would allow this situation to be changed by updating
the definition of element-wise comparison operations in NumPy to return a
dedicated subclass that implements the new circuit breaking protocol and also
changes the result array's interpretation in a boolean context to always
return ``False`` and hence never trigger the short-circuiting behaviour::
class ComparisonResultArray(np.ndarray):
def __bool__(self):
return False
def _raise_NotImplementedError(self):
msg = ("Comparison array truth values are ambiguous outside "
"chained comparisons. Use a.any() or a.all()")
raise NotImplementedError(msg)
def __not__(self):
self._raise_NotImplementedError()
def __then__(self):
self._raise_NotImplementedError()
def __else__(self, other):
return np.logical_and(self, other.view(ComparisonResultArray))
With this change, the chained comparison example above would be able to return::
>>> 0 < increasing < 4
ComparisonResultArray([ False, True, True, True, False], dtype=bool)
Existence checking expressions
------------------------------
An increasingly common requirement in modern software development is the need
to work with "semi-structured data": data where the structure of the data is
known in advance, but pieces of it may be missing at runtime, and the software
manipulating that data is expected to degrade gracefully (e.g. by omitting
results that depend on the missing data) rather than failing outright.
Some particularly common cases where this issue arises are:
* handling optional application configuration settings and function parameters
* handling external service failures in distributed systems
* handling data sets that include some partial records
At the moment, writing such software in Python can be genuinely awkward, as
your code ends up littered with expressions like:
* ``value1 = expr1.field.of.interest if expr1 is not None else None``
* ``value2 = expr2["field"]["of"]["interest"] if expr2 is not None else None``
* ``value3 = expr3 if expr3 is not None else expr4 if expr4 is not
None else expr5``
PEP 531 goes into more detail on some of the challenges of working with this
kind of data, particularly in data transformation pipelines where dealing with
potentially missing content is the norm rather than the exception.
The combined impact of the proposals in this PEP is to allow the above sample
expressions to instead be written as:
* ``value1 = missing(expr1) else expr1.field.of.interest``
* ``value2 = missing(expr2) else expr2.["field"]["of"]["interest"]``
* ``value3 = exists(expr3) else exists(expr4) else expr5``
In these forms, significantly more of the text presented to the reader is
immediately relevant to the question "What does this code do?", while the
boilerplate code to handle missing data by passing it through to the output
or falling back to an alternative input, has shrunk to two uses of the new
``missing`` builtin, and two uses of the new ``exists`` builtin.
In the first two examples, the 31 character boilerplate suffix
``if exprN is not None else None`` (minimally 27 characters for a single letter
variable name) has been replaced by a 19 character ``missing(expr1) else``
prefix (minimally 15 characters with a single letter variable name), markedly
improving the signal-to-pattern-noise ratio of the lines (especially if it
encourages the use of more meaningful variable and field names rather than
making them shorter purely for the sake of expression brevity). The additional
syntactic sugar proposals in PEP 505 would further reduce this boilerplate to
a single ``?`` character that also eliminated the repetition of the expession
being checked for existence.
In the last example, not only are two instances of the 21 character boilerplate,
`` if exprN is not None`` (minimally 17 characters) replaced with the
8 character function call ``exists()``, but that function call is placed
directly around the original expression, eliminating the need to duplicate it
in the conditional existence check.
Risks and concerns
==================
This PEP has been designed specifically to address the risks and concerns
raised when discussing PEPs 335, 505 and 531.
* it defines a new operator and adjusts the definition of chained comparison
rather than impacting the existing ``and`` and ``or`` operators
* the changes to the ``not`` unary operator are defined in such a way that
control flow optimisations based on the existing semantics remain valid
* rather than the cryptic ``??``, it uses ``else`` as the operator keyword in
exactly the same sense as it is already used in conditional expressions
* it defines a general purpose short-circuiting binary operator that can even
be used to express the existing semantics of ``and`` and ``or`` rather than
focusing solely and inflexibly on existence checking
* it names the proposed builtins in such a way that they provide a strong
mnemonic hint as to when the expression containing them will short-circuit
and skip evaluating the right operand
Possible confusion with conditional expressions
-----------------------------------------------
The proposal in this PEP is essentially for an "implied ``if``" where if you
omit the ``if`` clause from a conditional expression, you invoke the circuit
breaking protocol instead. That is::
exists(foo) else calculate_default()
invokes the new protocol, but::
foo.field.of.interest if exists(foo) else calculate_default()
bypasses it entirely, *including* the non-short-circuiting ``__else__`` method.
This mostly wouldn't be a problem for the proposed ``types.CircuitBreaker``
implementation (and hence the ``exists`` and ``missing`` builtins), as the
only purpose the extended protocol serves in that case is to remove the
wrapper in the short-circuiting case - the ``__else__`` method passes the
right operand through unchanged.
However, this discrepancy could potentially be eliminated entirely by also
updating conditional expressions to use the circuit breaking protocol if
the condition defines those methods. In that case, ``__then__`` would need
to be updated to accept the left operand as a parameter, with short-circuiting
indicated by passing in the circuit breaker itself::
class CircuitBreaker:
"""Base circuit breaker type (available as types.CircuitBreaker)"""
def __init__(self, value, condition, inverse_type):
self.value = value
self._condition = condition
self._inverse_type = inverse_type
def __bool__(self):
return self._condition
def __not__(self):
return self._inverse_type(self.value)
def __then__(self, other):
if other is not self:
return other
return self.value # Short-circuit, remove the wrapper
def __else__(self, other):
if other is not self:
return other
return self.value # Short-circuit, remove the wrapper
With this symmetric protocol, the definition of conditional expressions
could be updated to also make the ``else`` clause optional::
test: else_test ['if' or_test ['else' test]] | lambdef
else_test: or_test ['else' test]
(We would avoid the apparent simplification to ``else_test ('if' else_test)*``
in order to make it easier to correctly preserve the semantics of normal
conditional expressions)
Given that expanded definition, the following statements would be
functionally equivalent::
foo = calculate_default() if missing(foo)
foo = calculate_default() if foo is None else foo
Just as the base proposal already makes the following equivalent::
foo = exists(foo) else calculate_default()
foo = foo if foo is not None else calculate_default()
The ``if`` based circuit breaker form has the virtue of reading significantly
better when used for conditional imperative commands like debug messages::
print(some_expensive_query()) if verbosity > 2
If we went down this path, then ``operator.true`` would need to be declared
as the nominal implicit circuit breaker when the condition didn't define the
circuit breaker protocol itself (so the above example would produce ``None``
if the debugging message was printed, and ``False`` otherwise)
The main objection to this expansion of the proposal is that it makes it a
more intrusive change that may potentially affect the behaviour of existing
code, while the main point in its favour is that allowing both ``if`` and
``else`` as circuit breaking operators and also supporting the circuit breaking
protocol for normal conditional expressions would be significantly more
self-consistent than special-casing a bare ``else`` as a stand-alone operator.
Design Discussion
=================
Arbitrary sentinel objects
--------------------------
Unlike PEPs 505 and 531, this proposal readily handles custom sentinel objects::
class defined(types.CircuitBreaker):
MISSING = object()
def __init__(self, value):
super().__init__(self, value is not self.MISSING, undefined)
class undefined(types.CircuitBreaker):
def __init__(self, value):
super().__init__(self, value is defined.MISSING, defined)
# Using the sentinel to check whether or not an argument was supplied
def my_func(arg=defined.MISSING):
arg = defined(arg) else calculate_default()
Implementation
==============
As with PEP 505, actual implementation has been deferred pending in-principle
interest in the idea of making these changes - aside from the possible
syntactic ambiguity concerns covered by the grammer proposals above, the
implementation isn't really the hard part of these proposals, the hard part
is deciding whether or not this is a change where the long term benefits for
new and existing Python users outweigh the short term costs involved in the
wider ecosystem (including developers of other implementations, language
curriculum developers, and authors of other Python related educational
material) adjusting to the change.
...TBD...
Acknowledgements
================
Thanks go to Mark E. Haase for feedback on and contributions to earlier drafts
of this proposal. However, his clear and exhaustive explanation of the original
protocol design that modified the semantics of ``if-else`` conditional
expressions to use an underlying ``__then__``/``__else__`` protocol helped
convince me it was too complicated to keep, so this iteration contains neither
that version of the protocol, nor Mark's explanation of it.
References
==========
.. [1] PEP 335 rejection notification
(http://mail.python.org/pipermail/python-dev/2012-March/117510.html)
Copyright
=========
This document has been placed in the public domain under the terms of the
CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
Hello,
I would like to propose some reversed assignment operators.
Python already has some assignment operators:
a += c is the same as a = a + c
a -= c is the same as a = a - c
a *= c is the same as a = a * c
a /= c is the same as a = a / c
a //= c is the same as a = a // c
a %= c is the same as a = a % c
a **= c is the same as a = a ** c
What I would like to propose is the creation of the reverse:
a =+ c is the same as a = c + a
a =- c is the same as a = c - a
a =* c is the same as a = c * a
a =/ c is the same as a = c / a
a =// c is the same as a = c // a
a =% c is the same as a = c % a
a =** c is the same as a = c ** a
For addition (+= and =+), multiplication (*= and =*) and subtraction (-=
and =-) of numbers it would give the same result, but addition (+= and
=+) and multiplication (*= and =*) when used with strings return a
different result.
All the other operations (=/, =%, =** and =//) return a different result.
I think it is simple and easy to understand, therefore pythonic. :)
Best regards,
JM
(Hi, I'm a first-time poster. I was inspired by Raymond Hettinger's keynote
at PyCon CA to look at new PEPs and comment on them. Hopefully, I'm not
committing any faux-pas in my post!)
1) I don't think this proposal sufficiently handles falsy values.
What is the expected value of x in the following?
x = exists(0) else 1
Since the PEP specifically states that exists() is to check that the input
is not None, as a user, I would expect x == 0. However, from my
interpretation, it appears that the value 0 would still be directly
evaluated for truthiness and we would get x == 1.
I don't fully get what the purpose of __then__ and __else__ are meant to
be, but it seems like instead of this:
type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS)
you would want:
LHS if type(_lhs).__then__(_lhs) else RHS
Where __then__ returns a simple True/False result. (Maybe the name __then__
doesn't make sense in that case.)
2) My initial reaction was that `else` doesn't belong in an expression, but
I guess there's already precedent for that. (I actually wasn't aware of the
`x if y else z` expression until I read this PEP!)
I'm already not a fan of the overloading of else in the cases of for/else
and try/else. (Are there other uses? It's a hard thing to search on
Google...) Now, we're going to have `else`s that aren't anchored to other
statements. You've always known that an `else` belonged to the preceding
if/for/try at the same level of indentation. (Or in the same expression.)
Is there a chance of a missing colon silently changing the meaning of code?
if foo():
a = 1
else
bar()
(Probably not...)
Are two missing spaces too outrageous?
x = yiffoo() else bar()
I'm not 100% sure, but I think that a current parser would see that as a
syntax error very early, as opposed to having to wait for it to try to find
'yiffoo' at run-time.
3) Towards the end, you propose some very Perl-like syntax:
print(some_expensive_query()) if verbosity > 2
This seems completely unrelated to the rest of the PEP, and will likely
invite people to propose an `unless` operator if implemented. (Followed by
an `until` statement.) :)
While it does read more naturally like English, it seems superfluous in a
programming language.
4) The proposal shows how it fixes some common pain points:
value = missing(obj) else obj.field.of.interest
value = missing(obj) else obj["field"]["of"]["interest"]
But it doesn't address very similar ones:
missing(obj) else missing(obj.field) else missing(obj.field.of) else
obj.field.of.interest
obj.get('field', {}).get('of', {}).get('interest')
(The first example shows how it would be handled with the PEP in its
current state.)
Maybe these are too far out of scope, I'm not sure. They feel very similar
to me though.
I hope these are useful comments and not too nit-picky.
Thanks,
Ryan Fox
Hi,
async/await syntax is a very nice recent feature, but there is something
that I miss for coroutines defined with async def, as compared to
generators. Coroutines represent an interesting mental model that goes
beyond only asynchronous IO, so that I play with them in REPL often. But
there is no built-in function to actually run a coroutine, so that
typically I use something like:
>>> def run(coro):
... try:
... coro.send(None)
... except StopIteration as e:
... return e.value
>>> async def f():
... return 42
>>> run(f())
42
There is a simple yet useful function for interactive play with generators
- ``next``, but not for coroutines. There is an option to do:
>>> import asyncio
>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(f())
42
But this feels a bit redundant for an interactive play. I would propose to
add something like an above described ``run`` function to built-ins.
Yes, I know, there is a very high bar for adding a built-in function, but I
believe such a function will help to promote async/await to a wider
community (especially to novices).
--
Ivan
If recommend 'valued(foo)'. Without the final 'd' I think of "the value of
foo" rather than "does foo have a value?" Obviously, the value of foo is
just spelled 'foo' in Python, but it seems confusing.
'exists(foo)' is even more confusing since almost everyone will read it as
"is foo defined?" I know you can't do that with a call in Python, but you
can in lots of other languages.
On Nov 12, 2016 8:50 AM, "Mark E. Haase" <mehaase(a)gmail.com> wrote:
I like PEP-532. Given the opposition to non-Pythonic syntax like ?? (i.e.
PEP-505), Nick's proposal offers a Pythonic alternative that is protocol
based, more generalized, and uses built-ins and keywords to avoid
punctuation.
I agree with other posters that the terms "exists" and "missing" could lead
developers to think that it tests for NameError. Maybe "value(foo) else
bar"? I can't think of a better spelling for the inverse. Maybe the "if"
syntax described in the PEP is better: "foo.bar if value(foo)". In that
case, we wouldn't need an inverse to exists()/value()/whatever.
I also wanted to mention a couple of unmentioned benefits of this PEP:
1. It is easier to Google a name. E.g., Google "c# ??" and you'll get
nothing related to null coalescing in c#". ("C# question marks" does find
the right content, however.)
2. Dismay over the meaning of foo?.bar.baz is much clearer when expressed
as missing(foo) else foo.bar.baz -- it's very close to the ternary logic
you'd write if you didn't have a circuit breaking operator: None if foo is
None else foo.bar.baz.
On Sat, Nov 5, 2016 at 5:50 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote:
> Hi folks,
>
> As promised, here's a follow-up to the withdrawn PEP 531 that focuses
> on coming up with a common protocol driven solution to conditional
> evaluation of subexpressions that also addresses the element-wise
> comparison chaining problem that Guido noted when rejecting PEP 335.
>
> I quite like how it came out, but see the "Risks & Concerns" section
> for a discussion of an internal inconsistency it would introduce into
> the language if accepted as currently written, and the potentially
> far-reaching consequences actually resolving that inconsistency might
> have on the way people write their Python code (if the PEP was
> subsequently approved).
>
> Regards,
> Nick.
>
> =================================
> PEP: 532
> Title: A circuit breaking operator and protocol
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan <ncoghlan(a)gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 30-Oct-2016
> Python-Version: 3.7
> Post-History: 5-Nov-2016
>
> Abstract
> ========
>
> Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this
> PEP
> proposes the addition of a new protocol-driven circuit breaking operator to
> Python that allows the left operand to decide whether or not the expression
> should short circuit and return a result immediately, or else continue
> on with evaluation of the right operand::
>
> exists(foo) else bar
> missing(foo) else foo.bar()
>
> These two expressions can be read as:
>
> * "the expression result is 'foo' if it exists, otherwise it is 'bar'"
> * "the expression result is 'foo' if it is missing, otherwise it is
> 'foo.bar()'"
>
> Execution of these expressions relies on a new circuit breaking protocol
> that
> implicitly avoids repeated evaluation of the left operand while letting
> that operand fully control the result of the expression, regardless of
> whether
> it skips evaluation of the right operand or not::
>
> _lhs = LHS
> type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS)
>
> To properly support logical negation of circuit breakers, a new ``__not__``
> protocol method would also be introduced allowing objects to control
> the result of ``not obj`` expressions.
>
> As shown in the basic example above, the PEP further proposes the addition
> of
> builtin ``exists`` and ``missing`` circuit breakers that provide
> conditional
> branching based on whether or not an object is ``None``, but return the
> original object rather than the existence checking wrapper when the
> expression
> evaluation short circuits.
>
> In addition to being usable as simple boolean operators (e.g. as in
> ``assert all(exists, items)`` or ``if any(missing, items):``), these
> circuit
> breakers will allow existence checking fallback operations (aka
> None-coalescing
> operations) to be written as::
>
> value = exists(expr1) else exists(expr2) else expr3
>
> and existence checking precondition operations (aka None-propagating
> or None-severing operations) to be written as::
>
> value = missing(obj) else obj.field.of.interest
> value = missing(obj) else obj["field"]["of"]["interest"]
>
> A change to the definition of chained comparisons is also proposed, where
> the comparison chaining will be updated to use the circuit breaking
> operator
> rather than the logical disjunction (``and``) operator if the left hand
> comparison returns a circuit breaker as its result.
>
> While there are some practical complexities arising from the current
> handling
> of single-valued arrays in NumPy, this change should be sufficient to allow
> elementwise chained comparison operations for matrices, where the result
> is a matrix of boolean values, rather than tautologically returning
> ``True``
> or raising ``ValueError``.
>
>
> Relationship with other PEPs
> ============================
>
> This PEP is a direct successor to PEP 531, replacing the existence checking
> protocol and the new ``?then`` and ``?else`` syntactic operators defined
> there
> with a single protocol driven ``else`` operator and adjustments to the
> ``not``
> operator. The existence checking use cases are taken from that PEP.
>
> It is also a direct successor to PEP 335, which proposed the ability to
> overload the ``and`` and ``or`` operators directly, with the ability to
> overload the semantics of comparison chaining being one of the consequences
> of that change. The proposal in this PEP to instead handle the element-wise
> comparison use case by changing the semantic definition of comparison
> chaining
> is drawn from Guido's rejection of PEP 335.
>
> This PEP competes with the dedicated null-coalescing operator in PEP 505,
> proposing that improved support for null-coalescing operations be offered
> through a more general protocol-driven short circuiting operator and
> related
> builtins, rather than through a dedicated single-purpose operator.
>
> It doesn't compete with PEP 505's proposed shorthands for existence
> checking
> attribute access and subscripting, but instead offers an alternative
> underlying
> semantic framework for defining them:
>
> * ``EXPR?.attr`` would be syntactic sugar for ``missing(EXPR) else
> EXPR.attr``
> * ``EXPR?[key]`` would be syntactic sugar for ``missing(EXPR) else
> EXPR[key]``
>
> In both cases, the dedicated syntactic form could be optimised to avoid
> actually creating the circuit breaker instance.
>
>
> Specification
> =============
>
> The circuit breaking operator (``else``)
> ----------------------------------------
>
> Circuit breaking expressions would be written using ``else`` as a new
> binary
> operator, akin to the existing ``and`` and ``or`` logical operators::
>
> LHS else RHS
>
> Ignoring the hidden variable assignment, this is semantically equivalent
> to::
>
> _lhs = LHS
> type(_lhs).__then__(_lhs) if _lhs else type(_lhs).__else__(_lhs, RHS)
>
> The key difference relative to the existing ``or`` operator is that the
> value
> determining which branch of the conditional expression gets executed *also*
> gets a chance to postprocess the results of the expressions on each of the
> branches.
>
> As part of the short-circuiting behaviour, interpreter implementations
> are expected to access only the protocol method needed for the branch
> that is actually executed, but it is still recommended that circuit
> breaker authors that always return ``True`` or always return ``False`` from
> ``__bool__`` explicitly raise ``NotImplementedError`` with a suitable
> message from branch methods that are never expected to be executed (see the
> comparison chaining use case in the Rationale section below for an example
> of that).
>
> It is proposed that the ``else`` operator use a new precedence level that
> binds
> less tightly than the ``or`` operator by adjusting the relevant line in
> Python's grammar from the current::
>
> test: or_test ['if' or_test 'else' test] | lambdef
>
> to instead be::
>
> test: else_test ['if' or_test 'else' test] | lambdef
> else_test: or_test ['else' test]
>
> The definition of ``test_nocond`` would remain unchanged, so circuit
> breaking expressions would require parentheses when used in the ``if``
> clause of comprehensions and generator expressions just as conditional
> expressions themselves do.
>
> This grammar definition means precedence/associativity in the otherwise
> ambiguous case of ``expr1 if cond else expr2 else epxr3`` resolves as
> ``(expr1 if cond else expr2) else epxr3``.
>
> A guideline will also be added to PEP 8 to say "don't do that", as such a
> construct will be inherently confusing for readers, regardless of how the
> interpreter executes it.
>
>
> Overloading logical inversion (``not``)
> ---------------------------------------
>
> Any circuit breaker definition will have a logical inverse that is still a
> circuit breaker, but inverts the answer as to whether or not to short
> circuit
> the expression evaluation. For example, the ``exists`` and ``missing``
> circuit
> breakers proposed in this PEP are each other's logical inverse.
>
> A new protocol method, ``__not__(self)``, will be introduced to permit
> circuit
> breakers and other types to override ``not`` expressions to return their
> logical inverse rather than a coerced boolean result.
>
> To preserve the semantics of existing language optimisations, ``__not__``
> implementations will be obliged to respect the following invariant::
>
> assert not bool(obj) == bool(not obj)
>
>
> Chained comparisons
> -------------------
>
> A chained comparison like ``0 < x < 10`` written as::
>
> LEFT_BOUND left_op EXPR right_op RIGHT_BOUND
>
> is currently roughly semantically equivalent to::
>
> _expr = EXPR
> _lhs_result = LEFT_BOUND left_op _expr
> _expr_result = _lhs_result and (_expr right_op RIGHT_BOUND)
>
> This PEP proposes that this be changed to explicitly check if the left
> comparison returns a circuit breaker, and if so, use ``else`` rather than
> ``and`` to implement the comparison chaining::
>
> _expr = EXPR
> _lhs_result = LEFT_BOUND left_op _expr
> if hasattr(type(_lhs_result), "__then__"):
> _expr_result = _lhs_result else (_expr right_op RIGHT_BOUND)
> else:
> _expr_result = _lhs_result and (_expr right_op RIGHT_BOUND)
>
> This allows types like NumPy arrays to control the behaviour of chained
> comparisons by returning circuit breakers from comparison operations.
>
>
> Existence checking comparisons
> ------------------------------
>
> Two new builtins implementing the new protocol are proposed to encapsulate
> the
> notion of "existence checking": seeing if a value is ``None`` and either
> falling back to an alternative value (an operation known as
> "None-coalescing")
> or passing it through as the result of the overall expression (an operation
> known as "None-severing" or "None-propagating").
>
> These builtins would be defined as follows::
>
> class CircuitBreaker:
> """Base circuit breaker type (available as types.CircuitBreaker)"""
> def __init__(self, value, condition, inverse_type):
> self.value = value
> self._condition = condition
> self._inverse_type = inverse_type
> def __bool__(self):
> return self._condition
> def __not__(self):
> return self._inverse_type(self.value)
> def __then__(self):
> return self.value
> def __else__(self, other):
> if other is self:
> return self.value
> return other
>
> class exists(types.CircuitBreaker):
> """Circuit breaker for 'EXPR is not None' checks"""
> def __init__(self, value):
> super().__init__(value, value is not None, missing)
>
> class missing(types.CircuitBreaker):
> """Circuit breaker for 'EXPR is None' checks"""
> def __init__(self, value):
> super().__init__(value, value is None, exists)
>
> Aside from changing the definition of ``__bool__`` to be based on
> ``is not None`` rather than normal truth checking, the key characteristic
> of
> ``exists`` is that when it is used as a circuit breaker, it is *ephemeral*:
> when it is told that short circuiting has taken place, it returns the
> original
> value, rather than the existence checking wrapper.
>
> ``missing`` is defined as the logically inverted counterpart of ``exists``:
> ``not exists(obj)`` is semantically equivalent to ``missing(obj)``.
>
> The ``__else__`` implementations for both builtin circuit breakers are
> defined
> such that the wrapper will always be removed even if you explicitly pass
> the
> circuit breaker to both sides of the ``else`` expression::
>
> breaker = exists(foo)
> assert (breaker else breaker) is foo
> breaker = missing(foo)
> assert (breaker else breaker) is foo
>
>
> Other conditional constructs
> ----------------------------
>
> No changes are proposed to if statements, while statements, conditional
> expressions, comprehensions, or generator expressions, as the boolean
> clauses
> they contain are already used for control flow purposes.
>
> However, it's worth noting that while such proposals are outside the scope
> of
> this PEP, the circuit breaking protocol defined here would be sufficient to
> support constructs like::
>
> while exists(dynamic_query()) as result:
> ... # Code using result
>
> and:
>
> if exists(re.search(pattern, text)) as match:
> ... # Code using match
>
> Leaving the door open to such a future extension is the main reason for
> recommending that circuit breaker implementations handle the ``self is
> other``
> case in ``__else__`` implementations the same way as they handle the
> short-circuiting behaviour in ``__then__``.
>
>
> Style guide recommendations
> ---------------------------
>
> The following additions to PEP 8 are proposed in relation to the new
> features
> introduced by this PEP:
>
> * In the absence of other considerations, prefer the use of the builtin
> circuit breakers ``exists`` and ``missing`` over the corresponding
> conditional expressions
>
> * Do not combine conditional expressions (``if-else``) and circuit breaking
> expressions (the ``else`` operator) in a single expression - use one or
> the
> other depending on the situation, but not both.
>
>
> Rationale
> =========
>
> Adding a new operator
> ---------------------
>
> Similar to PEP 335, early drafts of this PEP focused on making the existing
> ``and`` and ``or`` operators less rigid in their interpretation, rather
> than
> proposing new operators. However, this proved to be problematic for a few
> reasons:
>
> * defining a shared protocol for both ``and`` and ``or`` was confusing, as
> ``__then__`` was the short-circuiting outcome for ``or``, while
> ``__else__``
> was the short-circuiting outcome for ``and``
> * the ``and`` and ``or`` operators have a long established and stable
> meaning,
> so readers would inevitably be surprised if their meaning now became
> dependent on the type of the left operand. Even new users would be
> confused
> by this change due to 25+ years of teaching material that assumes the
> current well-known semantics for these operators
> * Python interpreter implementations, including CPython, have taken
> advantage
> of the existing semantics of ``and`` and ``or`` when defining runtime and
> compile time optimisations, which would all need to be reviewed and
> potentially discarded if the semantics of those operations changed
>
> Proposing a single new operator instead resolves all of those issues -
> ``__then__`` always indicates short circuiting, ``__else__`` only indicates
> "short circuiting" if the circuit breaker itself is also passed in as the
> right operand, and the semantics of ``and`` and ``or`` remain entirely
> unchanged. While the semantics of the unary ``not`` operator do change, the
> invariant required of ``__not__`` implementations means that existing
> expression optimisations in boolean contexts will remain valid.
>
> As a result of that design simplification, the new protocol and operator
> would
> even allow us to expose ``operator.true`` and ``operator.false``
> as circuit breaker definitions if we chose to do so::
>
> class true(types.CircuitBreaker):
> """Circuit breaker for 'bool(EXPR)' checks"""
> def __init__(self, value):
> super().__init__(value, bool(value), when_false)
>
> class false(types.CircuitBreaker):
> """Circuit breaker for 'not bool(EXPR)' checks"""
> def __init__(self, value):
> super().__init__(value, not bool(value), when_true)
>
> Given those circuit breakers:
>
> * ``LHS or RHS`` would be roughly ``operator.true(LHS) else RHS``
> * ``LHS and RHS`` would be roughly ``operator.false(LHS) else RHS``
>
>
> Naming the operator and protocol
> --------------------------------
>
> The names "circuit breaking operator", "circuit breaking protocol" and
> "circuit breaker" are all inspired by the phrase "short circuiting
> operator":
> the general language design term for operators that only conditionally
> evaluate their right operand.
>
> The electrical analogy is that circuit breakers in Python detect and handle
> short circuits in expressions before they trigger any exceptions similar
> to the
> way that circuit breakers detect and handle short circuits in electrical
> systems before they damage any equipment or harm any humans.
>
> The Python level analogy is that just as a ``break`` statement lets you
> terminate a loop before it reaches its natural conclusion, a circuit
> breaking
> expression lets you terminate evaluation of the expression and produce a
> result
> immediately.
>
>
> Using an existing keyword
> -------------------------
>
> Using an existing keyword has the benefit of allowing the new expression to
> be introduced without a ``__future__`` statement.
>
> ``else`` is semantically appropriate for the proposed new protocol, and the
> only syntactic ambiguity introduced arises when the new operator is
> combined
> with the explicit ``if-else`` conditional expression syntax.
>
>
> Element-wise chained comparisons
> --------------------------------
>
> In ultimately rejecting PEP 335, Guido van Rossum noted [1_]:
>
> The NumPy folks brought up a somewhat separate issue: for them,
> the most common use case is chained comparisons (e.g. A < B < C).
>
> To understand this observation, we first need to look at how comparisons
> work
> with NumPy arrays::
>
> >>> import numpy as np
> >>> increasing = np.arange(5)
> >>> increasing
> array([0, 1, 2, 3, 4])
> >>> decreasing = np.arange(4, -1, -1)
> >>> decreasing
> array([4, 3, 2, 1, 0])
> >>> increasing < decreasing
> array([ True, True, False, False, False], dtype=bool)
>
> Here we see that NumPy array comparisons are element-wise by default,
> comparing
> each element in the lefthand array to the corresponding element in the
> righthand
> array, and producing a matrix of boolean results.
>
> If either side of the comparison is a scalar value, then it is broadcast
> across
> the array and compared to each individual element::
>
> >>> 0 < increasing
> array([False, True, True, True, True], dtype=bool)
> >>> increasing < 4
> array([ True, True, True, True, False], dtype=bool)
>
> However, this broadcasting idiom breaks down if we attempt to use chained
> comparisons::
>
> >>> 0 < increasing < 4
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ValueError: The truth value of an array with more than one element
> is ambiguous. Use a.any() or a.all()
>
> The problem is that internally, Python implicitly expands this chained
> comparison into the form::
>
> >>> 0 < increasing and increasing < 4
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ValueError: The truth value of an array with more than one element
> is ambiguous. Use a.any() or a.all()
>
> And NumPy only permits implicit coercion to a boolean value for
> single-element
> arrays where ``a.any()`` and ``a.all()`` can be assured of having the same
> result::
>
> >>> np.array([False]) and np.array([False])
> array([False], dtype=bool)
> >>> np.array([False]) and np.array([True])
> array([False], dtype=bool)
> >>> np.array([True]) and np.array([False])
> array([False], dtype=bool)
> >>> np.array([True]) and np.array([True])
> array([ True], dtype=bool)
>
> The proposal in this PEP would allow this situation to be changed by
> updating
> the definition of element-wise comparison operations in NumPy to return a
> dedicated subclass that implements the new circuit breaking protocol and
> also
> changes the result array's interpretation in a boolean context to always
> return ``False`` and hence never trigger the short-circuiting behaviour::
>
> class ComparisonResultArray(np.ndarray):
> def __bool__(self):
> return False
> def _raise_NotImplementedError(self):
> msg = ("Comparison array truth values are ambiguous outside "
> "chained comparisons. Use a.any() or a.all()")
> raise NotImplementedError(msg)
> def __not__(self):
> self._raise_NotImplementedError()
> def __then__(self):
> self._raise_NotImplementedError()
> def __else__(self, other):
> return np.logical_and(self, other.view(ComparisonResultArray))
>
> With this change, the chained comparison example above would be able to
> return::
>
> >>> 0 < increasing < 4
> ComparisonResultArray([ False, True, True, True, False], dtype=bool)
>
>
> Existence checking expressions
> ------------------------------
>
> An increasingly common requirement in modern software development is the
> need
> to work with "semi-structured data": data where the structure of the data
> is
> known in advance, but pieces of it may be missing at runtime, and the
> software
> manipulating that data is expected to degrade gracefully (e.g. by omitting
> results that depend on the missing data) rather than failing outright.
>
> Some particularly common cases where this issue arises are:
>
> * handling optional application configuration settings and function
> parameters
> * handling external service failures in distributed systems
> * handling data sets that include some partial records
>
> At the moment, writing such software in Python can be genuinely awkward, as
> your code ends up littered with expressions like:
>
> * ``value1 = expr1.field.of.interest if expr1 is not None else None``
> * ``value2 = expr2["field"]["of"]["interest"] if expr2 is not None else
> None``
> * ``value3 = expr3 if expr3 is not None else expr4 if expr4 is not
> None else expr5``
>
> PEP 531 goes into more detail on some of the challenges of working with
> this
> kind of data, particularly in data transformation pipelines where dealing
> with
> potentially missing content is the norm rather than the exception.
>
> The combined impact of the proposals in this PEP is to allow the above
> sample
> expressions to instead be written as:
>
> * ``value1 = missing(expr1) else expr1.field.of.interest``
> * ``value2 = missing(expr2) else expr2.["field"]["of"]["interest"]``
> * ``value3 = exists(expr3) else exists(expr4) else expr5``
>
> In these forms, significantly more of the text presented to the reader is
> immediately relevant to the question "What does this code do?", while the
> boilerplate code to handle missing data by passing it through to the output
> or falling back to an alternative input, has shrunk to two uses of the new
> ``missing`` builtin, and two uses of the new ``exists`` builtin.
>
> In the first two examples, the 31 character boilerplate suffix
> ``if exprN is not None else None`` (minimally 27 characters for a single
> letter
> variable name) has been replaced by a 19 character ``missing(expr1) else``
> prefix (minimally 15 characters with a single letter variable name),
> markedly
> improving the signal-to-pattern-noise ratio of the lines (especially if it
> encourages the use of more meaningful variable and field names rather than
> making them shorter purely for the sake of expression brevity). The
> additional
> syntactic sugar proposals in PEP 505 would further reduce this boilerplate
> to
> a single ``?`` character that also eliminated the repetition of the
> expession
> being checked for existence.
>
> In the last example, not only are two instances of the 21 character
> boilerplate,
> `` if exprN is not None`` (minimally 17 characters) replaced with the
> 8 character function call ``exists()``, but that function call is placed
> directly around the original expression, eliminating the need to duplicate
> it
> in the conditional existence check.
>
>
> Risks and concerns
> ==================
>
> This PEP has been designed specifically to address the risks and concerns
> raised when discussing PEPs 335, 505 and 531.
>
> * it defines a new operator and adjusts the definition of chained
> comparison
> rather than impacting the existing ``and`` and ``or`` operators
> * the changes to the ``not`` unary operator are defined in such a way that
> control flow optimisations based on the existing semantics remain valid
> * rather than the cryptic ``??``, it uses ``else`` as the operator keyword
> in
> exactly the same sense as it is already used in conditional expressions
> * it defines a general purpose short-circuiting binary operator that can
> even
> be used to express the existing semantics of ``and`` and ``or`` rather
> than
> focusing solely and inflexibly on existence checking
> * it names the proposed builtins in such a way that they provide a strong
> mnemonic hint as to when the expression containing them will
> short-circuit
> and skip evaluating the right operand
>
>
> Possible confusion with conditional expressions
> -----------------------------------------------
>
> The proposal in this PEP is essentially for an "implied ``if``" where if
> you
> omit the ``if`` clause from a conditional expression, you invoke the
> circuit
> breaking protocol instead. That is::
>
> exists(foo) else calculate_default()
>
> invokes the new protocol, but::
>
> foo.field.of.interest if exists(foo) else calculate_default()
>
> bypasses it entirely, *including* the non-short-circuiting ``__else__``
> method.
>
> This mostly wouldn't be a problem for the proposed ``types.CircuitBreaker``
> implementation (and hence the ``exists`` and ``missing`` builtins), as the
> only purpose the extended protocol serves in that case is to remove the
> wrapper in the short-circuiting case - the ``__else__`` method passes the
> right operand through unchanged.
>
> However, this discrepancy could potentially be eliminated entirely by also
> updating conditional expressions to use the circuit breaking protocol if
> the condition defines those methods. In that case, ``__then__`` would need
> to be updated to accept the left operand as a parameter, with
> short-circuiting
> indicated by passing in the circuit breaker itself::
>
> class CircuitBreaker:
> """Base circuit breaker type (available as types.CircuitBreaker)"""
> def __init__(self, value, condition, inverse_type):
> self.value = value
> self._condition = condition
> self._inverse_type = inverse_type
> def __bool__(self):
> return self._condition
> def __not__(self):
> return self._inverse_type(self.value)
> def __then__(self, other):
> if other is not self:
> return other
> return self.value # Short-circuit, remove the wrapper
> def __else__(self, other):
> if other is not self:
> return other
> return self.value # Short-circuit, remove the wrapper
>
> With this symmetric protocol, the definition of conditional expressions
> could be updated to also make the ``else`` clause optional::
>
> test: else_test ['if' or_test ['else' test]] | lambdef
> else_test: or_test ['else' test]
>
> (We would avoid the apparent simplification to ``else_test ('if'
> else_test)*``
> in order to make it easier to correctly preserve the semantics of normal
> conditional expressions)
>
> Given that expanded definition, the following statements would be
> functionally equivalent::
>
> foo = calculate_default() if missing(foo)
> foo = calculate_default() if foo is None else foo
>
> Just as the base proposal already makes the following equivalent::
>
> foo = exists(foo) else calculate_default()
> foo = foo if foo is not None else calculate_default()
>
> The ``if`` based circuit breaker form has the virtue of reading
> significantly
> better when used for conditional imperative commands like debug messages::
>
> print(some_expensive_query()) if verbosity > 2
>
> If we went down this path, then ``operator.true`` would need to be declared
> as the nominal implicit circuit breaker when the condition didn't define
> the
> circuit breaker protocol itself (so the above example would produce
> ``None``
> if the debugging message was printed, and ``False`` otherwise)
>
> The main objection to this expansion of the proposal is that it makes it a
> more intrusive change that may potentially affect the behaviour of existing
> code, while the main point in its favour is that allowing both ``if`` and
> ``else`` as circuit breaking operators and also supporting the circuit
> breaking
> protocol for normal conditional expressions would be significantly more
> self-consistent than special-casing a bare ``else`` as a stand-alone
> operator.
>
>
> Design Discussion
> =================
>
> Arbitrary sentinel objects
> --------------------------
>
> Unlike PEPs 505 and 531, this proposal readily handles custom sentinel
> objects::
>
> class defined(types.CircuitBreaker):
> MISSING = object()
> def __init__(self, value):
> super().__init__(self, value is not self.MISSING, undefined)
>
> class undefined(types.CircuitBreaker):
> def __init__(self, value):
> super().__init__(self, value is defined.MISSING, defined)
>
> # Using the sentinel to check whether or not an argument was supplied
> def my_func(arg=defined.MISSING):
> arg = defined(arg) else calculate_default()
>
>
> Implementation
> ==============
>
> As with PEP 505, actual implementation has been deferred pending
> in-principle
> interest in the idea of making these changes - aside from the possible
> syntactic ambiguity concerns covered by the grammer proposals above, the
> implementation isn't really the hard part of these proposals, the hard part
> is deciding whether or not this is a change where the long term benefits
> for
> new and existing Python users outweigh the short term costs involved in the
> wider ecosystem (including developers of other implementations, language
> curriculum developers, and authors of other Python related educational
> material) adjusting to the change.
>
> ...TBD...
>
>
> Acknowledgements
> ================
>
> Thanks go to Mark E. Haase for feedback on and contributions to earlier
> drafts
> of this proposal. However, his clear and exhaustive explanation of the
> original
> protocol design that modified the semantics of ``if-else`` conditional
> expressions to use an underlying ``__then__``/``__else__`` protocol helped
> convince me it was too complicated to keep, so this iteration contains
> neither
> that version of the protocol, nor Mark's explanation of it.
>
>
> References
> ==========
>
> .. [1] PEP 335 rejection notification
> (http://mail.python.org/pipermail/python-dev/2012-March/117510.html)
>
> Copyright
> =========
>
> This document has been placed in the public domain under the terms of the
> CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/
>
>
> --
> Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas(a)python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
_______________________________________________
Python-ideas mailing list
Python-ideas(a)python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Currently str(slice(10)) returns "slice(None, 10, None)"
If the start and step are None, consider not emitting them. Similarly
slice(None) is rendered slice(None, None, None).
When you're printing a lot of slices, it's a lot of extra noise.