(Apologies if it's a duplicate. I originally posted to python-ideas(a)googlegroups.com)
I've been recently working on an internal library written
entirely in Python. The library while being under development
was as actively used by other teams. The product was under
pressure and hasty design decisions were made.
Certain interfaces were exposed before scope of the problem
was entirely clear.
Didn't take long before the code started to beg for refactoring.
But compatibility needed to be preserved. As a result a set of utilities,
which I'd like to share, came to life.
---
The warnings module is a robust foundation. Authors can warn
what should be avoided and users can choose how to act.
However in its current form it still requires utilities
to be useful. E.g. it's non-trivial to properly replace
a deprecated class.
I'd like to propose an extension for the warnings module
to address this problem.
The extensions consists of 4 decorators:
- @deprecated
- @obsolete
- @deprecated_arg
- @obsolete_arg
The @deprecated decorator marks an object to issue a warning
if it's used:
- Callable objects issue a warning upon a call
- Property attributes issue a warning upon an access
- Classes issue a warning upon instantiation and subclassing
(directly or via a metaclass)
The @obsolete decorator marks an object in the same way
as @deprecated does but forwards usage to the replacement:
- Callable object redirect the call
- Property attribute redirect the access (get / set / del)
- Class is replaced in a way that during both instantiation
and subclassing replacement is used
In case of classes extra care is taken to preserve validity
of existing isinstance and issubclass checks.
The @deprecated_arg and @obsolete_arg work with signatures
of callable objects. Upon a call either a warning is issued
or an argument mapping is performed.
Please take a look at the implementation and a few examples:
https://gist.github.com/Kentzo/53df97c7a54609d3febf5f8eb6b67118
Why I think it should be a part of stdlib:
- Library authors are reluctant to add dependencies
especially when it's for internal usage
- Ease of use will reduce compatibility problems
and ease migration burden since the soltion will be
readily available
- IDEs and static analyzers are more likely
to support it
---
Joshua Harlow shared a link to OpenStack's debtcollector:
https://docs.openstack.org/debtcollector/latest/reference/index.html
Certain aspects of my implementation are inspired by it.
Please let me know what you think about the idea in general
and implementation in particular. If that's something the community
is interested in, I'm willing to work on it.
Hi,
The idea is simple: restore the "next" built-in and the "StopIteration"
propagation behavior from Python 3.6.
I'm using Python 3.7 for a while (as it's the default in Arch Linux), and
there's one single backwards incompatible change from Python 3.6 that is
breaking the code of some packages from PyPI. The reason is always the
same: a "next" called inside a generator function was expected to propagate
the StopIteration, but that no longer happens.
As an example of something that had been made public (in PyPI), I've tried
to run:
from articlemeta.client import RestfulClient
journals = list(RestfulClient().journals(collection="ecu"))
This breaks with "RuntimeError: generator raised StopIteration". I already
warned the maintainers of that project, it probably will be fixed.
Another example (disclaimer: this time it's a package I've created): a
simple "pip install dose" only works on Python<3.7, since some
reStructuredText processing functions rely on the StopIteration propagation
from calls to "next". I needed this package last saturday for a Coding
Dojo, but I had to download the package from the repository and change some
stuff before starting the Dojo (during the time reserved for it).
I can fix this in the packages I'm maintaining by creating a new
"fix_python37_next_runtime_error" decorator to restore the old behavior on
every generator function that uses "next" in its body. But I can't do that
to all packages from other people, and having to change/monkeypatch
imported stuff in order to keep it working in this new Python version is
getting annoying already.
Perhaps adding a new kwarg to the "next" built-in to choose between a
"propagate" default or a "error" alternative would avoid this.
--
Danilo J. S. Bellini
---------------
"*It is not our business to set up prohibitions, but to arrive at
conventions.*" (R. Carnap)
It’s been almost a week since this “discussion” first started. Can we please stop this in the name of productive work on python-ideas? Frankly, you don’t need to reply just because you can point out something wrong with someone else’s argument. Post because it’s worthwhile to hear, not because you have something to say.
This mindless and combative attitude is a big reason why Guido was motivated to suspend himself.
Are all current Python builds and C extensions vulnerable to Spectre
variants {1, 2, *}?
There are now multiple threads:
"SEC: Spectre variant 2: GCC: -mindirect-branch=thunk
-mindirect-branch-register"
-
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread...
- https://mail.python.org/pipermail/python-ideas/2018-September/053473.html
- https://mail.python.org/pipermail/python-dev/2018-September/155199.html
Original thread (that I forwarded to security@):
"[Python-ideas] Executable space protection: NX bit,"
https://mail.python.org/pipermail/python-ideas/2018-September/053175.html
> ~ Do trampolines / nested functions in C extensions switch off the NX bit?
On Sunday, September 16, 2018, Nathaniel Smith <njs(a)pobox.com> wrote:
> On Wed, Sep 12, 2018, 12:29 Joni Orponen <j.orponen(a)4teamwork.ch> wrote:
>
>> On Wed, Sep 12, 2018 at 8:48 PM Wes Turner <wes.turner(a)gmail.com> wrote:
>>
>>> Should C extensions that compile all add
>>> `-mindirect-branch=thunk -mindirect-branch-register` [1] to mitigate the
>>> risk of Spectre variant 2 (which does indeed affect user space applications
>>> as well as kernels)?
>>>
>>
>> Are those available on GCC <= 4.2.0 as per PEP 513?
>>
>
> Pretty sure no manylinux1 compiler is ever going to get these mitigations.
>
> For manylinux2010 on x86-64, we can easily use a much newer compiler: RH
> maintains a recent compiler, currently gcc 7.3, or if that doesn't work for
> some reason then the conda folks have be apparently figured out how to
> build the equivalent from gcc upstream releases.
>
Are there different CFLAGS and/or gcc compatibility flags in conda builds
of Python and C extensions?
Where are those set in conda builds?
What's the best way to set CFLAGS in Python builds and C extensions?
export CFLAGS="-mindirect-branch=thunk -mindirect-branch-register"
./configure
make
?
Why are we supposed to use an old version of GCC that doesn't have the
retpoline patches that only mitigate Spectre variant 2?
>
> Unfortunately, the manylinux2010 infrastructure is not quite ready... I'm
> pretty sure it needs some volunteers to push it to the finish line, though
> unfortunately I haven't had enough time to keep track.
>
"PEP 571 -- The manylinux2010 Platform Tag"
https://www.python.org/dev/peps/pep-0571/
"Tracking issue for manylinux2010 rollout"
https://github.com/pypa/manylinux/issues/179
Are all current Python builds and C extensions vulnerable to Spectre
variants {1, 2, *}?
>
Should Python builds add `-mindirect-branch=thunk
-mindirect-branch-register` to CFLAGS?
Where would this be to be added in the build scripts with which
architectures?
/QSpectre is the MSVC build flag for Spectre Variant 1:
> The /Qspectre option is available in Visual Studio 2017 version 15.7 and
later.
https://docs.microsoft.com/en-us/cpp/build/reference/qspectre?view=vs-2017
security@ directed me to the issue tracker / lists,
so I'm forwarding this to python-dev and python-ideas, as well.
# Forwarded message
From: *Wes Turner* <wes.turner(a)gmail.com>
Date: Wednesday, September 12, 2018
Subject: SEC: Spectre variant 2: GCC: -mindirect-branch=thunk
-mindirect-branch-register
To: distutils-sig <distutils-sig(a)python.org>
Should C extensions that compile all add
`-mindirect-branch=thunk -mindirect-branch-register` [1] to mitigate the
risk of Spectre variant 2 (which does indeed affect user space applications
as well as kernels)?
[1] https://github.com/speed47/spectre-meltdown-checker/
issues/119#issuecomment-361432244
[2] https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)
[3] https://en.wikipedia.org/wiki/Speculative_Store_Bypass#
Speculative_execution_exploit_variants
On Wednesday, September 12, 2018, Wes Turner <wes.turner(a)gmail.com> wrote:
>
>> On Wednesday, September 12, 2018, Joni Orponen <j.orponen(a)4teamwork.ch>
>> wrote:
>>
>>> On Wed, Sep 12, 2018 at 8:48 PM Wes Turner <wes.turner(a)gmail.com> wrote:
>>>
>>>> Should C extensions that compile all add
>>>> `-mindirect-branch=thunk -mindirect-branch-register` [1] to mitigate
>>>> the risk of Spectre variant 2 (which does indeed affect user space
>>>> applications as well as kernels)?
>>>>
>>>
>>> Are those available on GCC <= 4.2.0 as per PEP 513?
>>>
>>
>> AFAIU, only
>> GCC 7.3 and 8 have the retpoline (indirect-branch=thunk) support enabled
>> by the `-mindirect-branch=thunk -mindirect-branch-register` CFLAGS.
>>
>
On Wednesday, September 12, 2018, Wes Turner <wes.turner(a)gmail.com> wrote:
> "What is a retpoline and how does it work?"
> https://stackoverflow.com/questions/48089426/what-is-a-
> retpoline-and-how-does-it-work
>
>
I thank Steve D'Aprano for pointing me to this real-life (although
perhaps extreme) code example
https://github.com/Tinche/aiofiles/blob/master/aiofiles/threadpool/__init...
<code>
def open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None,
closefd=True, opener=None, *, loop=None, executor=None):
return AiofilesContextManager(_open(file, mode=mode, buffering=buffering,
encoding=encoding, errors=errors,
newline=newline, closefd=closefd,
opener=opener, loop=loop,
executor=executor))
@asyncio.coroutine
def _open(file, mode='r', buffering=-1, encoding=None, errors=None,
newline=None,
closefd=True, opener=None, *, loop=None, executor=None):
"""Open an asyncio file."""
if loop is None:
loop = asyncio.get_event_loop()
cb = partial(sync_open, file, mode=mode, buffering=buffering,
encoding=encoding, errors=errors, newline=newline,
closefd=closefd, opener=opener)
f = yield from loop.run_in_executor(executor, cb)
return wrap(f, loop=loop, executor=executor)
</code>
Anders Hovmöller has proposed a Python syntax extension to improve
this code. It provides, for example
return wrap(f, *, loop, executor)
as a shorthand for
return wrap(f, loop=loop, executor=executor)
See: https://mail.python.org/pipermail/python-ideas/2018-September/053207.html
I'd like us, in this thread, to discuss OTHER possible ways of
improving this code. This could include refactoring, and the
introduction of tools. I'm particularly interested in gathering
alternatives, and at this time not much interesting in "knowing which
one is best".
--
Jonathan
This has been bouncing around in my head for a while regarding the
requisite keys() method on mappings:
How come the ** unpacking operator, a built-in language feature, relies on
a non-dunder to operate?
To me, I mean to say, requiring that classes implement keys() – a method
whose name is totally undistinguished – in order to conform to the mapping
protocol feels like a design running counter to Python's norm of using
dunders for everything "hidden". I am not sure if it feels dirty to anybody
else, however. Interestingly, the docs already say
<https://docs.python.org/3/reference/datamodel.html#object.__iter__>
that *[f]or
mappings, [__iter__()] should iterate over the keys of the container*, but
it of course is not enforced in any way at present.
So, then — how about enforcing it? Should __iter__(), for the reasons
above, replace the current purpose of keys() in mappings?
I'm not properly equipped at the moment to mess around with CPython
(sorry), but I assume at a minimum this would entail either replacing all
instances of PyMapping_Keys() with PyObject_GetIter() or alternatively
changing PyMapping_Keys() to call the latter.
Does it sound like a reasonable change overall?
Eli
Have y’all seen ptpython’s autocomplete and syntax highlighting features? Ptpython, usually used as a cli application, might be worth integrating into IDLE.
