On 2018-06-18 15:09, Victor Stinner wrote:
> 2) we implemented a lot of other optimizations which made calls faster
> without having to touch tp_call nor tp_fastcall.
And that's a problem because these optimizations typically only work for
specific classes. My PEP wants to replace those by something more
I'm sure we've already had this conversation, but my google-fu is failing me.
Can someone provide a link to a discussion explaining why the new ordering of dictionaries does not defeat the
hash-randomization non-ordering we added a few versions ago?
I have been working on a slightly different PEP to use a new type slot
tp_ccalloffset instead the base_function base class. You can see the
work in progress here:
By creating a new protocol that each class can implement, there is a
full decoupling between the features of a class and between the class
hierarchy (such coupling was complained about during the PEP 575
discussion). So I got convinced that this is a better approach.
It also has the advantage that changes can be made more gradually: this
PEP changes nothing at all on the Python side, it only changes the
CPython implementation. I still think that it would be a good idea to
refactor the class hierarchy, but that's now an independent issue.
Another advantage is that it's more general and easier for existing
classes to use the protocol (PEP 575 on the other hand requires
subclassing from base_function which may not be compatible with an
existing class hierarchy).
On 2018-06-17 14:50, Ronald Oussoren wrote:
> This looks interesting. Why did you add a tp_ccalloffset slot to the type with the actual information in instances instead of storing the information in a slot?
Think of built-in functions. Every built-in function is a different
callable and calls a different C function. So it must be stored in the
However, the case where all instances share a PyCCallDef is also
possible: all instances would then simply have the same PyCCallDef pointer.
Given the discussion of adopting an annual release cadence at the language
summit this year , this recent Jakarta (nee Java) EE announcement caught
JEE are switching to an annual update cadence for the base platform, with
quarterly updates for individual components.
Since we last seriously discussed potential release cadence changes back in
2012 (with PEPs 407 and 413 [2,3]), that means JEE joins GCC switching to
an annual release cadence from GCC 5 back in 2015 (see ), while
clang/LLVM have been doing twice-annual feature releases for several years
Most directly relevant to Python would be the Node.js maintenance lifecycle
initially developed in 2013, and evolved since then:
That's quite a fast lifecycle (even faster than we're considering - they do
a release every 6 months, with every 2nd release being supported for 3 1/2
years), but one of the keys to making it work in practice is
The gist of the "Native Abstractions for Node.js" project is that it aims
to provide a stable API & ABI for third party libraries to link against,
but *without* having to keep those interfaces stable in V8/Node.js itself.
In the Python world, the closest current equivalent would by SIP for PyQt
projects , which provides a shim layer that allows version independent
extension modules to target CPython's native stable ABI with the aid of a
single version specific dependency (so only the "sip" wheel itself needs to
be rebuilt for each new Python version, not every extension module that
depends on it).
So I expect a release cadence change would be a lot more viable now than it
would have been 6 years ago, but I also suspect actually getting there will
require a non-trivial amount of effort invested in generalising the SIP
model such that the stable ABI gets a *lot* easier for projects to
realistically target (including for cffi and Cython generated extension
 https://releases.llvm.org/ (Note: LLVM switched to twice-yearly X.0.0
releases in 2017, but were doing twice yearly X.Y releases for some time
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
On 2018-05-19 15:29, Nick Coghlan wrote:
> That's not how code reviews work, as their complexity is governed by the
> number of lines changed (added/removed/modified), not just the number of
> lines that are left at the end.
Of course, you are right. I didn't mean literally that only the end
result matters. But it should certainly be considered.
If you only do small incremental changes, complexity tends to build up
because choices which are locally optimal are not always globally
optimal. Sometimes you need to do some refactoring to revisit some of
that complexity. This is part of what PEP 575 does.
> That said, "deletes more lines than it
> adds" is typically a point strongly in favour of a particular change.
This certainly won't be true for my patch, because there is a lot of
code that I need to support for backwards compatibility (all the old
code for method_descriptor in particular).
Going back to the review of PEP 575, I see the following possible outcomes:
(A) Accept it as is (possibly with minor changes).
(B) Accept the general idea but split the details up in several PEPs
which can still be discussed individually.
(C) Accept a minimal variant of PEP 575, only changing existing classes
but not changing the class hierarchy.
(D) Accept some yet-to-be-written variant of PEP 575.
(E) Don't fix the use case that PEP 575 wants to address.
Petr Viktorin suggests (C). I am personally quite hesitant because that
only adds complexity and it wouldn't be the best choice for the future
maintainability of CPython. I also fear that this hypothetical PEP
variant would be rejected because of that reason. Of course, if there is
some general agreement that (C) is the way to go, then that is fine for me.
If people feel that PEP 575 is currently too complex, I think that (B)
is a very good compromise. The end result would be the same as what PEP
575 proposes. Instead of changing many things at once, we could handle
each class in a separate PEP. But the motivation of those mini-PEPs will
still be PEP 575. So, in order for this to make sense, the general idea
of PEP 575 needs to be accepted: adding a base_function base class and
making various existing classes subclasses of that.
I was working on a concurrency limiting code for asyncio, so the user
may submit as many tasks as one wants, but only a max number of tasks
will be submitted to the event loop at the same time.
However, I wanted that passing an awaitable would always return a task,
no matter if the task was currently scheduled or not. The goal is that
you could add done callbacks to it, decide to force schedule it, etc
I dug in the asyncio.Task code, and encountered:
def __init__(self, coro, *, loop=None):
I was surprised to see that instantiating a Task class has any side
effect at all, let alone 2, and one of them being to be immediately
scheduled for execution.
I couldn't find a clean way to do what I wanted: either you
loop.create_task() and you get a task but it runs, or you don't run
anything, but you don't get a nice task object to hold on to.
I tried several alternatives, like returning a future, and binding the
future awaiting to the submission of a task, but that was complicated
code that duplicated a lot of things.
I tried creating a custom task, but it was even harder, setting a custom
event policy, to provide a custom event loop with my own create_task()
accepting parameters. That's a lot to do just to provide a parameter to
Task, especially if you already use a custom event loop (e.g: uvloop). I
was expecting to have to create a task factory only, but task factories
can't get any additional parameters from create_task()).
Additionally I can't use ensure_future(), as it doesn't allow to pass
any parameter to the underlying Task, so if I want to accept any
awaitable in my signature, I need to provide my own custom ensure_future().
All those implementations access a lot of _private_api, and do other
shady things that linters hate; plus they are fragile at best. What's
more, Task being rewritten in C prevents things like setting self._coro,
so we can only inherit from the pure Python slow version.
In the end, I can't even await the lazy task, because it blocks the
Hence I have 2 distinct, but independent albeit related, proposals:
- Allow Task to be created but not scheduled for execution, and add a
parameter to ensure_future() and create_task() to control this. Awaiting
such a task would just do like asyncio.sleep(O) until it is scheduled
- Add an parameter to ensure_future() and create_task() named "kwargs"
that accept a mapping and will be passed as **kwargs to the underlying
I insist on the fact that the 2 proposals are independent, so please
don't reject both if you don't like one or the other. Passing a
parameter to the underlying custom Task is still of value even without
the unscheduled instantiation, and vice versa.
Also, if somebody has any idea on how to make a LazyTask that we can
await on without blocking everything, I'll take it.
The Jseries acknowlegement by using Jetty containers can get you a best resolution To python wheel asynchronism bugs
Envoyé à partir d’un Smarpthone Android avec GMX Mail.
Le 14/06/2018, 4:00 PM python-dev-request(a)python.org a écrit:
On 13 Jun 2018, at 15:42, Nick Coghlan <ncoghlan(a)gmail.com<mailto:email@example.com>> wrote:
On 13 June 2018 at 02:23, Guido van Rossum <guido(a)python.org<mailto:firstname.lastname@example.org>> wrote:
So, to summarize, we need something like six for C?
Yeah, pretty much - once we can get to the point where it's routine for folks to be building "abiX" or "abiXY" wheels (with the latter not actually being a defined compatibility tag yet, but having the meaning of "targets the stable ABI as first defined in CPython X.Y"), rather than feature release specific "cpXYm" ones, then a *lot* of the extension module maintenance pain otherwise arising from more frequent CPython releases should be avoided.
There'd still be a lot of other details to work out to turn the proposed release cadence change into a practical reality, but this is the key piece that I think is a primarily technical hurdle: simplifying the current "wheel-per-python-version-per-target-platform" community project build matrices to instead be "wheel-per-target-platform”.
This requires getting people to mostly stop using the non-stable ABI, and that could be a lot of work for projects that have existing C extensions that don’t use the stable ABI or cython/cffi/…
That said, the CPython API tends to be fairly stable over releases and even without using the stable ABI supporting faster CPython feature releases shouldn’t be too onerous, especially for projects with some kind of automation for creating release artefacts (such as a CI system).