Guido van Rossum schrieb am 02.02.22 um 00:21:
> I wonder if a solution during CPython's rocky alpha release cycle could be
> to default (either in Cython or in projects using it) to the "not quite as
> fast but not relying on a lot of internal APIs" mode, and to switch to
> Cython's faster mode only once (a) beta is entered and (b) Cython has been
> fixed to work with that beta?
This seems tempting – with the drawback that it would make Cython modules
less comparable between final and alpha/beta CPython releases. So users
would start reporting ghost performance regressions because it
(understandably) feels important to them that the slow-down they witness
needs to be resolved before the final release, and they just won't know
that this will happen automatically triggered by the version switch. :)
It sounds like you are speaking from experience here, so I won't argue.
Feels a bit like car manufacturers who switch their exhaust cleaners on and
off based on the test mode detection.
That would be more like detecting benchmarks and doing something different. In terms of the car manufacturing process, we're talking about testing next year's model before production has started up yet. If the new model uses more gas than last year's, that would be a problem that needs to be solved before production starts, but what we seem to have with Cython is more like the new model's doors don't open. :-)
It may be hard to imagine if you're working on Cython, which only exists because of performance needs, but there are other things that people want to test with the upcoming CPython release in addition to performance (are the seats comfortable? do the controls for the moonroof work?), and given the long dependency chains in modern apps and packages, people want to get started on those things early. Until numpy builds with Cython for CPython 3.11, nobody can start testing scikit-learn with CPython 3.11, and that's frustrating for the scikit-learn maintainers.
More importantly, though, we'd get less bug reports during the alpha/beta
cycle ourselves, because things may look like they work but can still stop
working when we switch back to fast mode.
True, true. Nobody's perfect.
I'd rather make it more obvious to users what their intentions are. And
there is already a way to do that – the Limited API. (and similarly, HPy)
Your grammar confuses me. Do you want users to be clearer in expressing their intentions?
For Cython, support for the Limited API is still work in progress, although
many things are in place already. Getting it to work completely would give
users a simple way to decide whether they want to opt in for a) speed, lots
of wheels and adaptations for each CPython version, or b) less performance,
less hassle.
But until that work is complete, we're stuck with the unlimited API, right? And by its own statements in a recent post here, HPy is still not ready for all use cases, so it's also still a pipe dream.
As it looks now, that switch can be done after the code generation, by
defining a simple C define in their build script. That also makes both
modes easily comparable. I think that is as good as it can get.
Do you have specific instructions for package developers here? I could imagine that the scikit-learn maintainer (sorry to pick on you guys :-) might not know where to start with this if until now they've always been able to rely on either numpy wheels or building everything from source with default settings.