Backtick expressions work exactly like lambdas, except that they are bound to the instance they are created in every time that class is used to create one. To illustrate, this “percent” property is bound to the instance, not to the class.
class Example:
percent = property(`self.v*self.v2/100`)
And a few more examples for clarity.
def example():
locals()['a'] = 1
expr = `a+1`
return expr() # error: one variable is required
Any variable names that exist when the backtick expression is created are bound to the expression, and the reference to the expression is stored within the expression. Names that do not exist when the expresssion is created must be passed in as parameters. Such names can also be passed in as keyword arguments. Backtick expressions are created when their scope is created.
Variable names that are declared but have not been assigned to will be considered to exist for the purposes of the backtick expression.
Directly calling a backtick expression as soon as it’s created is forbidden:
`v+1`(a)
But this is technically allowed but discouraged, like how := works:
(`v+1`)(a)
Use Cases
This can be used anywhere a lambda would feel “heavy” or long. Here are a few use cases where using a backtick expression would allow code to be significantly mote readable:
If/else chains that would be switch statements.
Creating decorators.
Passing in logging hooks.
Writing design-by-contract contracts. (See icontract on GitHub for an example of what DBC looks like in Python.)
Tests and assertions.
Additionally, the instance binding enables:
A shorthand way to create a class that wraps an API to a better or more uniform code interface. Previously you’d need to make defs and @property, now each wrapped property and method is a single, readable line of code.
Appendix
I propose syntax highlighters show a backtick expression on a different background color, a lighter shade of black for a dark theme; dirty white for a light thing.
I also propose the following attributes on the backtick expression.
__str__(): the string [parameter names separated by commas] => [the string of the backtick expression]
__repr__(): the original string of the backtick expression, surrounded by backticks.
I secondarily propose that backtick expressions are only bound to their instances when defined within a class when the following syntax is used:
def a = <expression>
—
Now, let’s bikeshed.
So here’s an interesting idea, not a proposal yet.
In C++20, a Concept is a list of Boolean expressions with a name that can be used in place of a type in a templated (ie type-generic) function.
from typing import Concept
Iterator = Concept(lambda o: hasattr(o, "__iter__", lambda o: iter(o) != NotImplemented)
# Concept inheritance
Iterable = Concept(lambda o: hasattr(o, "__next__"), Iterator)
You could use concepts to define many practical “real-world” duck types.
A concept is like an opt-in duck typing type assertion. Since it’s a part of type annotation syntax, assertions can be generated automatically by looking at assertions.
You would use a Concept like any other type in type annotations.
Marko, how do you think Concepts might integrate with icontract? (I’m imagining overriding an import hook to automatically add contracts to functions with concepts.) How frequently do you use duck typing at Parquery? How might Concepts affect how often you used duck typing?
>
> The thing that concerns me is that any such problem and solution seems
> to apply equally to any other kind of block. Why not allow excepts on fo
> loops, for example?
>
Very good point.
I think 'with' is special in that it typically contains the entirety of the
use of an object, and the type of objects one tends to use in a 'with' are
prone to throwing exceptions. Other statements like it don't intrinsically
encapsulate the usage of an object.
On Tue, Jan 22, 2019 at 1:23 PM Calvin Spealman <cspealma(a)redhat.com> wrote:
>
>
> On Tue, Jan 22, 2019 at 3:11 PM Paul Ferrell <pflarr(a)gmail.com> wrote:
>
>> I've found that almost any time I'm writing a 'with' block, it's doing
>> something that could throw an exception. As a result, each of those
>> 'with' blocks needs to be nested within a 'try' block. Due to the
>> nature of 'with', it is rarely (if ever) the case that the try block
>> contains anything other than the with block itself.
>>
>> As a result, I would like to propose that the syntax for 'with' blocks
>> be changed such that they can be accompanied by 'except', 'finally',
>> and/or 'else' blocks as per a standard 'try' block. These would handle
>> exceptions that occur in the 'with' block, including the execution of
>> the applicable __enter__ and __exit__ methods.
>>
>> Example:
>>
>> try:
>> with open(path) as myfile:
>> ... # Do stuff with file
>> except (OSError, IOError) as err:
>> logger.error("Failed to read/open file {}: {}".format(path, err)
>>
>> The above would turn into simply:
>>
>> with open(path) as myfile:
>> ... # Do stuff with file
>> except (OSError, IOError) as err:
>> logger.error(...)
>>
>>
> It definitely makes sense, both the problem and the proposed solution.
>
> The thing that concerns me is that any such problem and solution seems
> to apply equally to any other kind of block. Why not allow excepts on fo
> loops, for example?
>
>
>>
>> I think this is rather straightforward in meaning and easy to read,
>> and simplifies some unnecessary nesting. I see this as the natural
>> evolution of what 'with'
>> is all about - replacing necessary try-finally blocks with something
>> more elegant. We just didn't include the 'except' portion.
>>
>> I'm a bit hesitant to put this out there. I'm not worried about it
>> getting shot down - that's kind of the point here. I'm just pretty
>> strongly against to unnecessary syntactical additions to the language.
>> This though, I think I can except. It introduces no new concepts and
>> requires no special knowledge to use. There's no question about what
>> is going on when you read it.
>>
>> --
>> Paul Ferrell
>> pflarr(a)gmail.com
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas(a)python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
>
> CALVIN SPEALMAN
>
> SENIOR QUALITY ENGINEER
>
> cspealma(a)redhat.com M: +1.336.210.5107
> <https://red.ht/sig>
> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>
--
Paul Ferrell
pflarr(a)gmail.com
--
Paul Ferrell
pflarr(a)gmail.com
We started a discussion in https://github.com/python/typing/issues/600
about adding support for extra annotations in the typing module.
Since this is probably going to turn into a PEP I'm transferring the
discussion here to have more visibility.
The document below has been modified a bit from the one in GH to reflect
the feedback I got:
+ Added a small blurb about how ``Annotated`` should support being used as
an alias
Things that were raised but are not reflected in this document:
+ The dataclass example is confusing. I kept it for now because
dataclasses often come up in conversations about why we might want to
support annotations in the typing module. Maybe I should rework the
section.
+ `...` as a valid parameter for the first argument (if you want to add an
annotation but use the type inferred by your type checker). This is an
interesting idea, it's probably worth adding support for it if and only if
we decide to support in other places. (c.f.:
https://github.com/python/typing/issues/276)
Thanks,
Add support for external annotations in the typing module
==========================================================
We propose adding an ``Annotated`` type to the typing module to decorate
existing types with context-specific metadata. Specifically, a type ``T``
can be annotated with metadata ``x`` via the typehint ``Annotated[T, x]``.
This metadata can be used for either static analysis or at runtime. If a
library (or tool) encounters a typehint ``Annotated[T, x]`` and has no
special logic for metadata ``x``, it should ignore it and simply treat the
type as ``T``. Unlike the `no_type_check` functionality that current exists
in the ``typing`` module which completely disables typechecking annotations
on a function or a class, the ``Annotated`` type allows for both static
typechecking of ``T`` (e.g., via MyPy or Pyre, which can safely ignore
``x``) together with runtime access to ``x`` within a specific
application. We believe that the introduction of this type would address a
diverse set of use cases of interest to the broader Python community.
Motivating examples:
~~~~~~~~~~~~~~~~~~~~
reading binary data
+++++++++++++++++++
The ``struct`` module provides a way to read and write C structs directly
from their byte representation. It currently relies on a string
representation of the C type to read in values::
record = b'raymond \x32\x12\x08\x01\x08'
name, serialnum, school, gradelevel = unpack('<10sHHb', record)
The struct documentation [struct-examples]_ suggests using a named tuple to
unpack the values and make this a bit more tractable::
from collections import namedtuple
Student = namedtuple('Student', 'name serialnum school gradelevel')
Student._make(unpack('<10sHHb', record))
# Student(name=b'raymond ', serialnum=4658, school=264, gradelevel=8)
However, this recommendation is somewhat problematic; as we add more
fields, it's going to get increasingly tedious to match the properties in
the named tuple with the arguments in ``unpack``.
Instead, annotations can provide better interoperability with a type
checker or an IDE without adding any special logic outside of the
``struct`` module::
from typing import NamedTuple
UnsignedShort = Annotated[int, struct.ctype('H')]
SignedChar = Annotated[int, struct.ctype('b')]
@struct.packed
class Student(NamedTuple):
# MyPy typechecks 'name' field as 'str'
name: Annotated[str, struct.ctype("<10s")]
serialnum: UnsignedShort
school: SignedChar
gradelevel: SignedChar
# 'unpack' only uses the metadata within the type annotations
Student.unpack(record))
# Student(name=b'raymond ', serialnum=4658, school=264, gradelevel=8)
dataclasses
++++++++++++
Here's an example with dataclasses [dataclass]_ that is a problematic from
the typechecking standpoint::
from dataclasses import dataclass, field
@dataclass
class C:
myint: int = 0
# the field tells the @dataclass decorator that the default action in
the
# constructor of this class is to set "self.mylist = list()"
mylist: List[int] = field(default_factory=list)
Even though one might expect that ``mylist`` is a class attribute
accessible via ``C.mylist`` (like ``C.myint`` is) due to the assignment
syntax, that is not the case. Instead, the ``@dataclass`` decorator strips
out the assignment to this attribute, leading to an ``AttributeError`` upon
access::
C.myint # Ok: 0
C.mylist # AttributeError: type object 'C' has no attribute 'mylist'
This can lead to confusion for newcomers to the library who may not expect
this behavior. Furthermore, the typechecker needs to understand the
semantics of dataclasses and know to not treat the above example as an
assignment operation in (which translates to additional complexity).
It makes more sense to move the information contained in ``field`` to an
annotation::
@dataclass
class C:
myint: int = 0
mylist: Annotated[List[int], field(default_factory=list)]
# now, the AttributeError is more intuitive because there is no
assignment operator
C.mylist # AttributeError
# the constructor knows how to use the annotations to set the 'mylist'
attribute
c = C()
c.mylist # []
The main benefit of writing annotations like this is that it provides a way
for clients to gracefully degrade when they don't know what to do with the
extra annotations (by just ignoring them). If you used a typechecker that
didn't have any special handling for dataclasses and the ``field``
annotation, you would still be able to run checks as though the type were
simply::
class C:
myint: int = 0
mylist: List[int]
lowering barriers to developing new types
+++++++++++++++++++++++++++++++++++++++++
Typically when adding a new type, we need to upstream that type to the
typing module and change MyPy [MyPy]_, PyCharm [PyCharm]_, Pyre [Pyre]_,
pytype [pytype]_, etc. This is particularly important when working on
open-source code that makes use of our new types, seeing as the code would
not be immediately transportable to other developers' tools without
additional logic (this is a limitation of MyPy plugins [MyPy-plugins]_),
which allow for extending MyPy but would require a consumer of new
typehints to be using MyPy and have the same plugin installed). As a
result, there is a high cost to developing and trying out new types in a
codebase. Ideally, we should be able to introduce new types in a manner
that allows for graceful degradation when clients do not have a custom MyPy
plugin, which would lower the barrier to development and ensure some degree
of backward compatibility.
For example, suppose that we wanted to add support for tagged unions
[tagged-unions]_ to Python. One way to accomplish would be to annotate
``TypedDict`` in Python such that only one field is allowed to be set::
Currency = Annotated(
TypedDict('Currency', {'dollars': float, 'pounds': float}, total=False),
TaggedUnion,
)
This is a somewhat cumbersome syntax but it allows us to iterate on this
proof-of-concept and have people with non-patched IDEs work in a codebase
with tagged unions. We could easily test this proposal and iron out the
kinks before trying to upstream tagged union to `typing`, MyPy, etc.
Moreover, tools that do not have support for parsing the ``TaggedUnion``
annotation would still be able able to treat `Currency` as a ``TypedDict``,
which is still a close approximation (slightly less strict).
Details of proposed changes to ``typing``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
syntax
++++++
``Annotated`` is parameterized with a type and an arbitrary list of Python
values that represent the annotations. Here are the specific details of the
syntax:
* The first argument to ``Annotated`` must be a valid ``typing`` type or
``...`` (to use the infered type).
* Multiple type annotations are supported (Annotated supports variadic
arguments): ``Annotated[int, ValueRange(3, 10), ctype("char")]``
* ``Annotated`` must be called with at least two arguments
(``Annotated[int]`` is not valid)
* The order of the annotations is preserved and matters for equality
checks::
Annotated[int, ValueRange(3, 10), ctype("char")] != \
Annotated[int, ctype("char"), ValueRange(3, 10)]
* Nested ``Annotated`` types are flattened, with metadata ordered starting
with the innermost annotation::
Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] ==\
Annotated[int, ValueRange(3, 10), ctype("char")]
* Duplicated annotations are not removed: ``Annotated[int, ValueRange(3,
10)] != Annotated[int, ValueRange(3, 10), ValueRange(3, 10)]``
* ``Annotation`` can be used a higher order aliases::
Typevar T = ...
Vec = Annotated[List[Tuple[T, T]], MaxLen(10)]
# Vec[int] == `Annotated[List[Tuple[int, int]], MaxLen(10)]
consuming annotations
++++++++++++++++++++++
Ultimately, the responsibility of how to interpret the annotations (if at
all) is the responsibility of the tool or library encountering the
`Annotated` type. A tool or library encountering an `Annotated` type can
scan through the annotations to determine if they are of interest (e.g.,
using `isinstance`).
**Unknown annotations**
When a tool or a library does not support annotations or encounters an
unknown annotation it should just ignore it and treat annotated type as the
underlying type. For example, if we were to add an annotation that is not
an instance of `struct.ctype` to the annotation for name (e.g.,
`Annotated[str, 'foo', struct.ctype("<10s")]`), the unpack method should
ignore it.
**Namespacing annotations**
We do not need namespaces for annotations since the class used by the
annotations acts as a namespace.
**Multiple annotations**
It's up to the tool consuming the annotations to decide whether the
client is allowed to have several annotations on one type and how to merge
those annotations.
Since the ``Annotated`` type allows you to put several annotations of the
same (or different) type(s) on any node, the tools or libraries consuming
those annotations are in charge of dealing with potential duplicates. For
example, if you are doing value range analysis you might allow this::
T1 = Annotated[int, ValueRange(-10, 5)]
T2 = Annotated[T1, ValueRange(-20, 3)]
Flattening nested annotations, this translates to::
T2 = Annotated[int, ValueRange(-10, 5), ValueRange(-20, 3)]
An application consuming this type might choose to reduce these
annotations via an intersection of the ranges, in which case ``T2`` would
be treated equivalently to ``Annotated[int, ValueRange(-10, 3)]``.
An alternative application might reduce these via a union, in which case
``T2`` would be treated equivalently to ``Annotated[int, ValueRange(-20,
5)]``.
Other applications may decide to not support multiple annotations and
throw an exception.
References
===========
.. [struct-examples]
https://docs.python.org/3/library/struct.html#examples
.. [dataclass]
https://docs.python.org/3/library/dataclasses.html
.. [MyPy]
https://github.com/python/mypy
.. [MyPy-plugins]
https://mypy.readthedocs.io/en/latest/extending_mypy.html#extending-mypy-us…
.. [PyCharm]
https://www.jetbrains.com/pycharm/
.. [Pyre]
https://pyre-check.org/
.. [pytype]
https://github.com/google/pytype
.. [tagged-unions]
https://en.wikipedia.org/wiki/Tagged_union
without starting a should we ban tkinter discussion, i'd like to propose
that we add rounded corners buttons. that might make the aesthetic level go
up a bit more
poor me, if only py had some really nice native gui
Abdur-Rahmaan Janhangeer
http://www.pythonmembers.club
Mauritius
>
> One possible argument for making PASS the default, even if that means
>> implementation-dependent behaviour with NANs, is that in the absense of a
>> clear preference for FAIL or RETURN, at least PASS is backwards compatible.
>>
>> You might shoot yourself in the foot, but at least you know its the same
>> foot you shot yourself in using the previous version *wink*
>>
>
I've lost attribution chain. I think this is Steven, but it doesn't really
matter.
This statement is untrue, or at least only accidentally true at most. The
behavior of sorted() against partially ordered collections is unspecified.
The author of Timsort says exactly this.
If stastics.median() keeps the same implementation—or keeps it with a PASS
argument—it may or may not produce the same result in a later Python
versions. Timsort is great, but even that has been tweaked sightly over
time.
I guess the statement is true if "same foot" means "meaningless answer" not
some specific value. But that hardly feels like a defense of the behavior.
>
Bug #33084 reports that the statistics library calculates median and
other stats wrongly if the data contains NANs. Worse, the result depends
on the initial placement of the NAN:
py> from statistics import median
py> NAN = float('nan')
py> median([NAN, 1, 2, 3, 4])
2
py> median([1, 2, 3, 4, NAN])
3
See the bug report for more detail:
https://bugs.python.org/issue33084
The caller can always filter NANs out of their own data, but following
the lead of some other stats packages, I propose a standard way for the
statistics module to do so. I hope this will be uncontroversial (he
says, optimistically...) but just in case, here is some prior art:
(1) Nearly all R stats functions take a "na.rm" argument which defaults
to False; if True, NA and NAN values will be stripped.
(2) The scipy.stats.ttest_ind function takes a "nan_policy" argument
which specifies what to do if a NAN is seen in the data.
https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.…
(3) At least some Matlab functions, such as mean(), take an optional
flag that determines whether to ignore NANs or include them.
https://au.mathworks.com/help/matlab/ref/mean.html#bt5b82t-1-nanflag
I propose adding a "nan_policy" keyword-only parameter to the relevant
statistics functions (mean, median, variance etc), and defining the
following policies:
IGNORE: quietly ignore all NANs
FAIL: raise an exception if any NAN is seen in the data
PASS: pass NANs through unchanged (the default)
RETURN: return a NAN if any NAN is seen in the data
WARN: ignore all NANs but raise a warning if one is seen
PASS is equivalent to saying that you, the caller, have taken full
responsibility for filtering out NANs and there's no need for the
function to slow down processing by doing so again. Either that, or you
want the current implementation-dependent behaviour.
FAIL is equivalent to treating all NANs as "signalling NANs". The
presence of a NAN is an error.
RETURN is equivalent to "NAN poisoning" -- the presence of a NAN in a
calculation causes it to return a NAN, allowing NANs to propogate
through multiple calculations.
IGNORE and WARN are the same, except IGNORE is silent and WARN raises a
warning.
Questions:
- does anyone have an serious objections to this?
- what do you think of the names for the policies?
- are there any additional policies that you would like to see?
(if so, please give use-cases)
- are you happy with the default?
Bike-shed away!
--
Steve
On Sun, Jan 06, 2019 at 10:52:47PM -0500, David Mertz wrote:
> Playing with Tim's examples, this suggests that statistics.median() is
> simply outright WRONG. I can think of absolutely no way to characterize
> these as reasonable results:
>
> Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 09:50:42)
> In [4]: statistics.median([9, 9, 9, nan, 1, 2, 3, 4, 5])
> Out[4]: 1
> In [5]: statistics.median([9, 9, 9, nan, 1, 2, 3, 4])
> Out[5]: nan
The second is possibly correct if one thinks that the median of a list
containing NAN should return NAN -- but its only correct by accident,
not design.
As I wrote on the bug tracker:
"I agree that the current implementation-dependent behaviour when there
are NANs in the data is troublesome."
The only reason why I don't call it a bug is that median() makes no
promises about NANs at all, any more than it makes promises about the
median of a list of sets or any other values which don't define a total
order. help(median) says:
Return the median (middle value) of numeric data.
By definition, data containing Not A Number values isn't numeric :-)
I'm not opposed to documenting this better. Patches welcome :-)
There are at least three correct behaviours in the face of data
containing NANs: propogate a NAN result, fail fast with an exception, or
treat NANs as missing data that can be ignored. Only the caller can
decide which is the right policy for their data set.
Aside: the IEEE-754 standard provides both signalling and quiet NANs. It
is hard and unreliable to generate signalling float NANs in Python, but
we can do it with Decimal:
py> from statistics import median
py> from decimal import Decimal
py> median([1, 3, 4, Decimal("sNAN"), 2])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.5/statistics.py", line 349, in median
data = sorted(data)
decimal.InvalidOperation: [<class 'decimal.InvalidOperation'>]
In principle, one ought to be able to construct float signalling NANs
too, but unfortunately that's platform dependent:
https://mail.python.org/pipermail/python-dev/2018-November/155713.html
Back to the topic on hand: I agree that median() does "the wrong thing"
when NANs are involved, but there is no one "right thing" that we can do
in its place. People disagree as to whether NANs should propogate, or
raise, or be treated as missing data, and I see good arguments for all
three.
--
Steve
On Tue, Jan 08, 2019 at 04:25:17PM +0900, Stephen J. Turnbull wrote:
> Steven D'Aprano writes:
>
> > By definition, data containing Not A Number values isn't numeric :-)
>
> Unfortunately, that's just a joke, because in fact numeric functions
> produce NaNs.
I'm not sure if you're agreeing with me or disagreeing, so I'll assume
you're agreeing and move on :-)
> I agree that this can easily be resolved by documenting that it is the
> caller's responsibility to remove NaNs from numeric data, but I prefer
> your proposed flags.
>
> > The only reason why I don't call it a bug is that median() makes no
> > promises about NANs at all, any more than it makes promises about the
> > median of a list of sets or any other values which don't define a total
> > order.
>
> Pedantically, I would prefer that the promise that ordinal data
> (vs. specifically numerical) has a median be made explicit, as there
> are many cases where statistical data is ordinal.
I think that is reasonable.
Provided the data defines a total order, the median is well-defined when
there are an odd number of data points, or you can use median_low and
median_high regardless of the number of data points.
> This may be a moot
> point, as in most cases ordinal data is represented numerically in
> computation (Likert scales, for example, are rarely coded as "hate,
> "dislike", "indifferent", "like", "love", but instead as 1, 2, 3, 4,
> 5), and from the point of view of UI presentation, IntEnums do the
> right thing here (print as identifiers, sort as integers).
>
> Perhaps a better way to document this would be to suggest that ordinal
> data be represented using IntEnums? (Again to be pedantic, one might
> want OrderedEnums that can be compared but don't allow other
> arithmetic operations.)
That's a nice solution.
--
Steve (the other one)
I was writing some python code earlier, and I noticed that in a code that
looks somwhat like this one :
try:
i = int("string")
print("continued on")
j = int(9.0)
except ValueError as e:
print(e)
>>> "invalid literal for int() with base 10: 'string'"
this code will handle the exception, but the code in the try block will not
continue.
I propose to be able to use the continue keyword to continue the execution
of the try block even when an error is handled. The above could then be
changed to :
try:
i = int("string")
print("continued on")
j = int(9.0)
except ValueError as e:
print(e)
continue
>>> "invalid literal for int() with base 10: 'string'"
>>> "continued on"
