capi-sig July 2018

capi-sig@python.org

12 participants
19 discussions

[capi-sig]Fwd: C API and performance
by Antoine Pitrou 31 Jul '18

31 Jul '18

Le 31/07/2018 à 18:25, Victor Stinner a écrit : > > 2018-07-31 18:03 GMT+02:00 Antoine Pitrou <solipsis(a)pitrou.net>: >>> For example, PyPy uses different memory allocators depending on the >>> scope and the lifetime of an object. I'm not sure that you can >>> implement such optimization if you are stuck with reference counting. >> >> But what does reference counting have to do with memory allocators >> exactly? > > I repeated what I heard from other developers and made my own guess. > But it seems like you disagree. I think that it's better that you ask > directly authors of other Python implementations like PyPy or Pyston > their mood on the C API and if they see any opportunity to make their > runtime faster by modifying the current C API. Why don't you go and ask them? You are the one making claims such as "changing the C API will allow making CPython 2x faster", and trying to back them with allusions about how PyPy got 5x faster than CPython, as if there was an obvious connection between the C API and PyPy's performance. I'm not interested in finding arguments in favour of your position. That's your job, not mine. >>> How can we make CPython 2x faster? Why everybody, except of PyPy, >>> failed to do that? >> >> Because PyPy spent years working full time on a JIT compiler. It's also >> written in (a dialect of) Python, which helps a lot with experimenting >> and building abstractions, compared to C or even C++. > > Ok. But PyPy doesn't support natively the C API, and cpyext is still > causing troubles to PyPy. So what? I'm sorry, but it seems you have a hard time sticking to a precise topic. Are you talking about PyPy's performance, or about how to make CPython faster, or are you talking about issues with cpyext? If you want to make cpyext's life better, I would recommend asking the cpyext authors for suggestions, instead of trying to second-guess their wishes. > If we modify CPython to make similar changes than PyPy, we will hit > the same issues. I don't understand what this sentence means, sorry. Regards Antoine.

1 0

[capi-sig]Re: Adding an official (minimal) Cython-like tool.
by Jeroen Demeyer 31 Jul '18

31 Jul '18

On 2018-07-31 17:59, Eric Snow wrote: > Regardless, it would help all parties involved > if there *were* an official tool that was part of CPython (i.e. in the > repo). Why does the tool need to be "official"? I mean, is Sphinx considered to be the "official" tool for Python documentation? I would say that it is de facto standard and that's perfectly fine.

1 0

[capi-sig]Make the stable ABI usable
by Victor Stinner 31 Jul '18

31 Jul '18

Hi Gustavo, Le lundi 30 juillet 2018, Stefan Behnel <python_capi(a)behnel.de> a écrit : > Gustavo Carneiro schrieb am 30.07.2018 um 16:18: >> What is the relation to PEP 384? https://www.python.org/dev/peps/pep-0384/ > > The "stable ABI" is a restricted API/ABI that breaks backwards > compatibility with all extension modules out there. If you write code > against this API, then you can make it work in newer CPython versions > without recompilation. This does not apply to any previously existing C > extension, which means that any changes to the non-stable parts of the > C-API would still break the world. > > Stefan Replacing macros accessing structures fields with function calls makes the generated machine code compatible with the stable ABI: http://pythoncapi.readthedocs.io/stable_abi.html And: http://pythoncapi.readthedocs.io/new_api.html#hide-implementation-details It means that C extensions using PyList_GET_ITEM() become compatible with new Python runtimes using the stable ABI, without having to modify their code. The PEP 384 requires to replace PyList_GET_ITEM() with PyList_GetItem(). I would like to continue the work on this PEP, to allow to use the stable ABI on more C extensions. Victor

6 14

[capi-sig]C API and performance
by Victor Stinner 31 Jul '18

31 Jul '18

Antoine Pitrou replied to my "Let's change to C API!" thread on python-dev. I reply here on a new thread on the capi-sig mailing list. 2018-07-31 18:03 GMT+02:00 Antoine Pitrou <solipsis(a)pitrou.net>: >> For example, PyPy uses different memory allocators depending on the >> scope and the lifetime of an object. I'm not sure that you can >> implement such optimization if you are stuck with reference counting. > > But what does reference counting have to do with memory allocators > exactly? I repeated what I heard from other developers and made my own guess. But it seems like you disagree. I think that it's better that you ask directly authors of other Python implementations like PyPy or Pyston their mood on the C API and if they see any opportunity to make their runtime faster by modifying the current C API. >> Do you think that it's doable to port numpy to Cython? It's made of >> 255K lines of C code. > > Numpy is a bit special as it exposes its own C API, so porting it > entirely to Cython would be difficult (how do you expose a C macro in > Cython?). Also, internally it has a lot of macro-generated code for > specialized loop implementations (metaprogramming in C :-)). > > I suppose some bits could be (re)written in Cython. Actually, the > numpy.random module is already a Cython module. numpy is one example of piece of code which is old and big, and it would take a lot of time to be converted to Cython, but nobody wants to do it. I expect that they are many other C extensions where the maintainer doesn't want to spend time on converting their code to Cython. I don't plan to modify hundreds of C extensions. I did that in the past to port modules to Python 3 and it was painful. I would like to design change small enough to get what I want, without having to spend years on porting manually all C extensions one by one. >> How can we make CPython 2x faster? Why everybody, except of PyPy, >> failed to do that? > > Because PyPy spent years working full time on a JIT compiler. It's also > written in (a dialect of) Python, which helps a lot with experimenting > and building abstractions, compared to C or even C++. Ok. But PyPy doesn't support natively the C API, and cpyext is still causing troubles to PyPy. If we modify CPython to make similar changes than PyPy, we will hit the same issues. Victor

1 0

[capi-sig]Re: Let's change to C API!
by Jeroen Demeyer 31 Jul '18

31 Jul '18

On 2018-07-31 15:36, Victor Stinner wrote: > Don't we already provide source compatibility with the current API? Yes we do. Like I said, Ronald's "level 2" is really the current documented C API which provides source compatibility.

1 0

[capi-sig]Re: Make the stable ABI usable
by Jeroen Demeyer 31 Jul '18

31 Jul '18

On 2018-07-31 15:06, Victor Stinner wrote: > But PyArg_ParseTuple(), the question is if we can rely on the fact > that the caller is supposed to hold a strong reference. The caller is supposed to hold a reference. See https://bugs.python.org/issue25750 and https://github.com/python/cpython/pull/6118 for a crash where this is not the case (and somebody please review it!).

1 0

[capi-sig]Re: Let's change to C API!
by Jeroen Demeyer 31 Jul '18

31 Jul '18

It seems that your proposal to change the C API really does two independent things: 1. Clean up some of the bad C API, such as PyDict_GetItem() and borrowed references. 2. Hide implementation details and make the ABI more stable. Personally, I agree with 1 but not so much with 2 (Stefan already argued at length why). PS: Please add PyEval_GetFuncName() to the bad C API. It returns the internal const char* inside a borrowed reference to a function __name__. Getting that to work with PEP 580 was "interesting"... Jeroen.

3 3

[capi-sig]Re: Let's change to C API!
by Jeroen Demeyer 31 Jul '18

31 Jul '18

On 2018-07-31 14:02, Ronald Oussoren wrote: > I haven’t had time yet to read through those pages, but I wonder if there shouldn’t be (at least) three levels of API: > > 1) More or less the current full API (with some cleanups), but with full access to internals > 2) Smaller API without access to type internals, but with PyTypeObject and supporting types > 3) The restricted API with a stable ABI (PEP 384) That's exactly the current situation. Your "level 1" is the undocumented API, "level 2" is the documented API and "level 3" is the PEP 384 API.

2 1

[capi-sig]Second argument of METH_NOARGS functions
by Jeroen Demeyer 30 Jul '18

30 Jul '18

I propose to drop the guarantee that the second (unused) argument of a METH_NOARGS function is always NULL. This would allow a minor simplification in PEP 580. For details, see - https://bugs.python.org/issue34280 - https://github.com/python/cpython/pull/8563

1 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

capi-sig July 2018