[capi-sig]Re: Open questions about borrowed references
On 2018-09-04 23:50, Victor Stinner wrote:
I would like to design a new C API without borrow references. So Py_TYPE() should go, replaced by something else.
I don't see the problem with Py_TYPE(). Consider it an accessor macro, just a nice way to access to ob_type field.
Le mer. 5 sept. 2018 à 00:12, Jeroen Demeyer <J.Demeyer@ugent.be> a écrit :
I'm trying to design an API which allows to remove the ob_type field from PyObject. I would to do that for many reasons. I wrote a whole website to explain why: https://pythoncapi.readthedocs.io/
In short:
- Leaking ob_field prevents to compile a C extension to the stable ABI
- Accessing directly ob_field prevents to change the structure and so experiment optimizations
- Accessing directly ob_field by derefering pointers expect that a pointer can be dereference which prevents to used tagged pointers
The current ("old") C API prevents to experiment many optimizations (in CPython or CPython forks), make cpyext of PyPy inefficient, and make it really hard to write a new Python implementation which supports the C API.
Victor
On Wed, Sep 5, 2018 at 8:18 AM Victor Stinner <vstinner@redhat.com> wrote:
I don't see how removing Py_TYPE() will make it easier to remove the ob_type field. I'd expect that it would make it even harder.
As someone who does use the C API from time to time, I have never thought of Py_TYPE (or other similar macros/functions) as returning a borrowed reference. To me, it's a C programming style that asks me to please use these macros/functions to access properties of "objects" because the API designer wants to be able to later change the struct underneath without rewriting code.
Programmers using the Python C API need a way to ask "what is the type of this object?" With the current API they can either use ob_type directly, or wrap it in Py_TYPE() which is (I guess?) the preferred style.
So if you want to remove ob_type, you can - as long as you keep Py_TYPE(). Phase 1, announce that ob_type will be removed in next version, or that it might be a tagged pointer and no longer usable as a C pointer. Phase 2, in the next version, rename ob_type and change Py_TYPE() to match. Existing code sitll runs, but compilation errors for any native extensions not using Py_TYPE(). Finally in phase 3 you can turn the type into a tagged pointer, handle, or whatever.
--
cheers,
Hugh Fisher
On Wed, Sep 5, 2018 at 1:15 AM Hugh Fisher <hugo.fisher@gmail.com> wrote:
Borrowed references impose the condition on the caller that they must ensure the borrowed-from object stays alive (and keeps the borrowed object alive), but have the advantage that they relieve the caller from having to take an explicit action that they are now done with the result of the call, making code like Py_TYPE(obj)->tp_name possible. If the goal is to remove all borrowed references, I don't see a way around changing every single callsite that leverages borrowed references.
(It may also offer a performance advantage to avoid all the refcounting, e.g. when iterating over a container.)
Le mer. 5 sept. 2018 à 01:15, Hugh Fisher <hugo.fisher@gmail.com> a écrit :
I have multiple goals for the new C API:
- hide implementation details to allow to experiment new optimizations like tagged pointers
- hide implementation details to allow to modify deeply C structures used internally in CPython
- don't access C structures to generate machine code which would be compatible with multiple Python versions: "stable ABI"
I listed some ideas of optimizations: https://pythoncapi.readthedocs.io/optimization_ideas.html
My plan doesn't include implementing these optimizations, but prepare the code to allow to experiment multiple kinds of optimizations.
Again, you rely on the exact behaviour of the current ("old") C API where PyObject* is really a pointer to a concrete PyObject which exist somewhere in memory. In my plan, a PyObject* can be something else (ex: tagged pointer where a number is directly stored inside the "pointer").
I already modifiy Py_TYPE() macro to emit a function call, _Py_TYPE_impl(). Currently, it's called the "Py_NEWCAPI_NO_STRUCT" API. => this API is incompatible with tagged pointer because it returns a borrowed reference
I also implemented an API without Py_TYPE() called the "Py_NEWCAPI" API... => no more borrored reference, ready for tagged pointer for example
... but removing Py_TYPE() simply broke all C extensions of CPython. So I reverted temporary this change and started this discussion.
It seems like I have to add a new Py_GetType() function which returns a strong reference (Py_INCREF) and start to modify C extension for use it instead of Py_TYPE().
Victor
[…]
Just curious, but why?
Now that CI/CD systems are getting mainstream having extensions that work with multiple python versions is less and less useful. For PyObjC I create binary wheels for Python 2.7, 3.4, 3.5, 3.6 and 3.7 and for 3 of them I create 2 variants of those wheels (on two different versions of macOS). The only cost compared to using a stable ABI is time: building a lot of wheels takes more time than building just a few. A lot of projects could just use Travis/CircleCI/… to generate wheels and upload those to PyPI (or even automate that part in their CI/CD pipeline)
IMHO the PEP 384 way of defining types is also less friendly to developers because you don’t get compiler warnings/errors when the prototype of functions you use for slots don’t match.
Ronald
On Wed, Sep 5, 2018 at 4:44 AM Ronald Oussoren via capi-sig <capi-sig@python.org> wrote:
That's a lot to ask of every single C extension unless the infrastructure is provided by the PSF and basically trivial to set up
as in, register with PyPI, provide the source, and get all of your wheels for free. And it doesn't help for PyPy or IronPython or Jython, all of which are hamstrung by not really being able to use C extensions, and also unlikely to make the list for most extensions. A stable ABI (and opaque PyObjects) would make it much easier to support alternative implementations. (Although, using a special build of extensions just for IronPython isn't something I ever really considered, and may need another look.)
Jeff
Why is that? Most CIs have a free tier that can be used by open-source projects, which is most if not all packages on PyPI. The hardest part is writing the scripting that collects build artefact from the CI/CD and upload them to PyPI.
BTW. See for example <https://github.com/joerick/cibuildwheel <https://github.com/joerick/cibuildwheel>> for a script that automates most of the wheel creation for Windows, macOS and Linux.
Would the stable ABI really help the other python implementations? I honestly don’t know, although its obvious that the stable ABI makes it easier to reach compatibility because there is less to emulate.
Ronald
A stable ABI by itself doesn't help. But a trimmed down ABI which does not hardcode implementation details is surely helpful. Many implementation details are too deeply hardcoded by now and basically impossible to remove: reference counting, for example; but e.g., the work Victor is doing on killing borrowed references is surely helpful for us.
Maybe for PyPy it's too late: if we want to be relevant in today's world, we need to emulate all this crap anyway. But who knowns, maybe in 10 years another alternative python implementation will benefit from this.
Just to add a data point:
The stable API was mainly meant platforms such as Windows which at the time did not provide free compilers to compile your own extensions.
Nowadays, I think this is no longer a big concern, since VC++ compilers are freely available. Since extension writers have to test whether their extensions compile against newer Python versions anyway, many will have the necessary infrastructure for the Python versions they support in place.
That said: Having a PSF hosted infrastructure for compiling extensions for many different platforms would certainly be a great thing to put in place.
I know that Ben Nuttall, for example, build something like this for the Raspberry Pi: it tries to compile all PyPI packages from source and then provides the binaries for download:
https://bennuttall.com/piwheels-building-a-faster-python-package-repository-...
On 06.09.2018 19:17, Antonio Cuni wrote:
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 06 2018)
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
Le ven. 7 sept. 2018 à 09:24, Ronald Oussoren <ronaldoussoren@mac.com> a écrit :
Maybe earlier if this efforts leads to an API that’s more friendly for PyPy.
Is there anything you can share about what API details are particularly problematic?
I'm in discussion with PyPy developers, and they reported different APIs which cause them troubles:
- in general, any API addition means more work for them
- anything which use borrowed references
- PyObject** should be banned: "PyObject** PySequence_Fast_ITEMS(ob)" has to go https://pythoncapi.readthedocs.io/bad_api.html#array-of-pointers-to-python-o...
- the finalizer API: I guess that it's an issue because of their different garbage collector
- almost all PyUnicode API functions have to go according to them. PyPy3 uses UTF-8 internally, CPython uses "compact string" (array of Py_UCS1, Py_UCS2 or Py_UCS4 depending on the string content). https://pythoncapi.readthedocs.io/bad_api.html#pypy-requests
I tried to list all PyPy requests on this "Bad C API" page.
I'm ok with removing PyObject** and most PyUnicode APIs, but I'm not sure about the finalizer API which seems to be a great enhancement of Python regarding to reference cycles.
Victor
I'm -1 on removing the PyUnicode APIs. We deliberately created a useful and very complete C API for Unicode.
The fact that PyPy chose to use a different internal representation is not a good reason to remove APIs and have CPython extension take the hit as a result. It would be better for PyPy rethink the internal representation or create a shim API which translates between the two worlds.
Note that UTF-8 is not a good internal representation for Unicode if you want fast indexing and slicing. This is why we are using fixed code units to represent the Unicode strings.
On 07.09.2018 10:22, Victor Stinner wrote:
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 07 2018)
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
This is something thats completely off-topic for this discussion, but I wonder if fast indexing and slicing are really necessary. Even with our current representation doing slicing correctly is hard due to combining characters and emoji. Any changes in this regard would probably require changes to the string API and/or additional utilities.
Ronald
On 07.09.2018 10:48, Ronald Oussoren wrote:
You have to differentiate between indexing code points (which is what Python provides) or graphemes (combined code points).
Python doesn't have good support for the latter at the moment, but then again: most text processing also doesn't necessarily need this, otherwise people would have stepped in to add better support. I had suggested doing this soon after Python 2.0 received Unicode, but there was no interest, so I dropped the idea again:
https://mail.python.org/pipermail/python-dev/2001-July/015938.html
Now, UTF-8 has the same issues at the code unit level, as combined code points have at the Unicode level, but the situation is much worse: you enter this minefield as soon as you leave ASCII, which is a much more common situation than having to deal with accidentally breaking combining code points with a slice in Unicode.
To address this, you need an index into UTF-8 and then you lose the perceived better memory efficiency of UTF-8 again. On top your code gets a lot more complex.
BTW: I do wonder why you think that fast indexing and slicing are not needed. Pretty much all text parsing relies on this heavily.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 07 2018)
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
Not necessarily, it is often possible to use higher-level API (str.startswith, str.split, …) or regular expressions.
But: I also wrote that there probably need to be new APIs to fully remove the need for fast indexing on code points.
Ronald
On 07.09.2018 11:58, Ronald Oussoren wrote:
Well, yes, but those APIs use indexing and slicing for creating their return values.
But: I also wrote that there probably need to be new APIs to fully remove the need for fast indexing on code points.
I don't see how this would work in general.
You can probably get by with a few APIs to deal with short strings, but for any longer chunks of text, you would not want to first convert this into multiple shorter strings for processing (say lines of text). Instead, you'd try to avoid any copying or creation of temporary objects and always index directly into the larger text, identify the parts you need and then only work on those slices.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 07 2018)
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
M.-A. Lemburg schrieb am 07.09.2018 um 12:58:
Not necessarily at the C level, though, which is where pretty much all of them are implemented. In a UTF-8 string, search could use memmem() and not rely on character by character traversal, because the byte sequence is self-synchronising. And the splitting then just copies out bytes.
What is a problem then, is obviously finding out the character *position* of a substring, if that's requested from Python code. That would either require a character index (which costs memory) or counting characters (which costs time), as you noted. Even regexes can run into that problem, when users request a group position, for example. Python code that does its own slicing will always be impacted.
So, as always, the end result depends on the application. If indexes are important per se, it's costly. Otherwise, UTF-8 strings may be faster or not make a difference at all.
Stefan
On Fri, Sep 7, 2018 at 10:32 AM M.-A. Lemburg <mal@egenix.com> wrote:
I don't agree; an API should not dictate which data structure and/or internal representation to use, else it quickly become impossible to innovate and experiment.
Yes, we can (and will) create a translation layer, but it has a performance and complexity cost.
I don't want to debate whether this is a good requirement or not, I'll let others to do that :) But note that PyPy *does* support fast indexing and slicing in its utf8 branch: the basic idea is to create (lazily) an index to map between logical index in the str and physical offesets in the array.
On Fri, Sep 7, 2018 at 9:24 AM Ronald Oussoren <ronaldoussoren@mac.com> wrote:
yes, but the current status is that we already implemented 99% of the current API; before we are going to kill it, we would need to wait until most/all current C modules are rewritten using the new API. Considering what happened for py3k, 10 years sounds like a conservative estimate :)
Is there anything you can share about what API details are particularly problematic?
I am writing a blog post which tries to explain why cpyext is hard, the early draft is here: https://bitbucket.org/pypy/extradoc/src/extradoc/blog/draft/2018-09-cpyext/c...
it doesn't contain (yet) specific examples of problematic API, but should give a rough understanding of what are the main challenges. The biggest challenge is reference counting: ideally, if we had a C API which is compatible with moving GC, it would make life much easier to us (and probably to Jython, IronPython, etc.), but I think that it's impractical.
On Thu, Sep 6, 2018 at 10:06 AM Ronald Oussoren <ronaldoussoren@mac.com> wrote:
It's mostly that every project needs to go through the configuration steps for each platform it wants to support, and have access to test runners for those platforms. The combinatorial explosion is painful. Scripts help, obviously, but "n_platforms" is always goings to be less than "n_platforms * m_python_implementations". Also, I don't think we should be dependent on the generosity of commercial providers.
For IronPython, at least, it doesn't map 1:1 with CPython versions. Maintaining a C API shims across multiple versions is a lot of work. If extensions were compiled against "stable ABI 4", and we could assume that stable ABI 4 would never change, that would simplify things. I'm sure it could do this today, but I don't think there's a way for an extension to report what version it's compiled against.
Right now I don't think it's possible to use a 3.6 extension on 3.7 without recompiling, but in the future, maybe CPython 4.2 could understand CPython 4.0 and 4.1 extensions by checking the API version. (I think this is how Android's API levels work, more or less.) IronPython (and PyPy, etc.) would then only have to worry about what ABI version it's targeting, not what CPython version that means.
It does potentially make it harder to make changes to CPython internals, if that sort of compatibility has to be preserved, but it might also allow for *more* changes if it limits how extensions can muck around with CPython's internal data structures.
- Jeff
On Wed, Sep 5, 2018 at 8:29 PM Victor Stinner <vstinner@redhat.com> wrote:
...
No I am not - that's the whole point of using Py_TYPE(). I pass in an object, I get back another object, which is the type of the argument. Under the current API it's always a pointer. If you change ints, or booleans, or any other values into tagged pointers and at the same time modify Py_TYPE() to handle tagged pointers correctly, all my code continues to work. If you change Py_TYPE() into a function instead of a macro, all my code will still work.
And this happens to be exactly what Apple did when they added tagged pointers to Objective-C. They told programmers not to access objectptr->isa directly, instead to use OBJECT_GETCLASS(objectptr). Then later they started implementing some objects as tagged pointers, and only code that hadn't been updated broke. OBJECT_GETCLASS was the "stable ABI"
I predict this will lead to more bugs, not less.
Firstly, the majority of C extension methods have no effect on object lifetimes. Py_TYPE() and other functions are used on either self, or one of the arguments. If the object doesn't disappear while the method is executing, any borrowed reference that is only used within that C method remains valid until the end of the method. The only way it can cause problems if the borrowed reference is stored somewhere else, but that is very rare.
With your proposal, every time I use Py_GetType() within a C method I have to remember to store that object reference somewhere and Py_DECREF it before returning. So no, it won't be just modifying extensions to use it instead of Py_TYPE(). You'll also have to modify the code to store the reference, and add more code to release it. More code means more chances to get it wrong.
Secondly and more generally, I'm not sure it is possible to maintain invariants, such as "all references are correctly ref counted", within the code that is responsible for maintaining and implementing the invariants themselves. An implementation has to be able to break the rules internally.
--
cheers,
Hugh FisherAll of these are unrelated to borrowed references. Conceptually every instance owns a reference to its type, regardless of how and where that reference is stored, and it should be possible to return a borrowed reference.
As others have mentioned before it is unclear if your plans w.r.t. creating a new API will help in this regard.
Ronald
Le mer. 5 sept. 2018 à 09:17, Ronald Oussoren <ronaldoussoren@mac.com> a écrit :
You are describing the current ("old") C API. I ask you to imagine a new C API where PyObject* is really an opaque thing. It's better to even see as an integer (Py_uintptr_t) rather than a pointer.
I would like to use "PyObject*" as an opaque thing because I would like to be able to implement tagged pointer for example. If you store an integer inside a "PyObject*", there is no PyObject structure anywhere. There is no linked PyTypeObject anywhere.
But all existing C extensions really want a "PyObject*" which behaves as it's a real Python object with an associated PyObject structure linked to a concrete PyTypeObject. So in some places, we need to produce temporary PyLongObject which represent the integer stored in the tagged pointer.
Borrowed references are a blocker issue for that. How do you know when the temporary object should go away?
Tagged pointer is one example. Another example would be a list of integers which stores numbers are C integers (ex: int32_t) rather than PyLongObject. PyList_GetItemRef() has to create a temporary PyLongObject.
Victor
On 09/05/18 12:21, Victor Stinner wrote:
The tagged pointer optimization can only be used for a limited set of basic types int, bool, small str/bytes etc. Right? Can you make these basic type objects immortal? Their refcount would be ignored, their dealloc would be a no-op. If you need a heap type for some reason it would need to be destroyed specially at interpreter shutdown.
I used tagged pointers with integers because it's an example simple to understand. But the issue is wider than just this very specific use case. IMHO we should not rely on borrowed references of Py_TYPE() and use strong references.
Victor Le mer. 5 sept. 2018 à 13:32, Petr Viktorin <encukou@gmail.com> a écrit :
On Wed, Sep 5, 2018 at 10:12 PM Victor Stinner <vstinner@redhat.com> wrote:
In case of Py_TYPE(), I still don't understand what's wrong about borrowed reference. Py_TYPE(ob) borrows reference from ob. So Py_DECREF(ob) may break the borrowed reference. It's straight enough.
Int or str may be embedded as tagged pointer. Py_TYPE(ob) should return int or str type when ob is it's instance. It's not bad. All tagged-value-able types should be immortal -- never freed.
-- INADA Naoki <songofacandy@gmail.com>
But that’s almost entirely unrelated to the issue of using borrowed references for Py_TYPE. Every Python object has a type, and hence owns a reference to a type object regardless of how that’s implemented.
Borrowed references have there problems, but do make the CPython API easier to use. I’m all for making the API safer to use, but IMHO eradicating the use of borrowed references is not necessary for that. There are APIs where moving to a variant without borrowed references would be better (in particular PyList_GetItem, which is a ticking bomb if you don’t INCREF the result before calling a CPython API)
There’s two related issues here:
*All* existing PyObjects contain a “PyObject” structure (although that’s hidden in PyObject_HEAD and should be easy to remove in all extensions that are properly ported to Python 3)
Some macros and function currently assume that 1) is true (such as Py_TYPE, Py_REFCNT, Py_INCREF, Py_DECREF). Those can be changed to do something else, without a need to introduce temporary objects.
The most problematic part of 2) is that Py_REFCNT is an lvalue and some code might exploit this to change the refcount of an object, but that’s arguably inherently bad code.
Both require changes to the stable ABI though.
BTW. Both the refcount macros and Py_TYPE are called a lot, unconditionally turning them into external functions could be quite costly performance wise.
Having a go at tagged points has been on my wishlist for a long time, but I haven’t managed do do anything about that yet beyond some thinking. The second example would be less interesting with tagged pointers (although elements would necessarily be the same size as a pointer, which can use more memory than the minimal storage).
Ronald
Le mer. 5 sept. 2018 à 13:36, Ronald Oussoren <ronaldoussoren@mac.com> a écrit :
Borrowed references have there problems, but do make the CPython API easier to use. I’m all for making the API safer to use, but IMHO eradicating the use of borrowed references is not necessary for that.
I don't really care about designing an API easy to use. Right now, I prefer to focus on correctness and design the ideal API. Once we know what is the ideal API and see the cost to reach it, we can start talking about tradeoff and milestones to reach the final goal.
There are APIs where moving to a variant without borrowed references would be better (in particular PyList_GetItem, which is a ticking bomb if you don’t INCREF the result before calling a CPython API)
I removed PyTuple_GetItem() from Py_NEWCAPI, replaced with PyTuple_GetItemRef(). I will do the same for PyList_GetItem() and other variants.
The most problematic part of 2) is that Py_REFCNT is an lvalue and some code might exploit this to change the refcount of an object, but that’s arguably inherently bad code.
I introduced _Py_SET_TYPE() and _Py_SET_REFCNT() for that. Maybe I should make them public, I don't know yet.
Both require changes to the stable ABI though.
The stable ABI is not usable yet, so I don't think that anyone is using it right now. My plan is to make it much easier to use it, and later maybe make it the default.
BTW. Both the refcount macros and Py_TYPE are called a lot, unconditionally turning them into external functions could be quite costly performance wise.
I would prefer to not guess anything about performance. First experiment the change, run benchmark, then decide about the tradeoff between performance and correctness.
I'm working on a fork of CPython. I will not touch CPython upstream until we have enough information to get wise decisions ;-)
Victor
I do because an easy to use C API is one of the reasons why Python is as popular as it is.
One thing that might help with correct C API usage is to teach static analysis tools about the API and correct usage thereof, see <https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/ <https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/>> for an experimenal example of this from the past. Doing this might also help in finding APIs that are hard to reason about, and finding alternatives for them. Apple added something similar to clang before they switched to automatic reference counting (ARC) in ObjC.
PyTuple_GetItem should be fairly safe, if the callers owns a reference to the tuple itself. It is technically possible that tuples change, but that’s bad behaviour (similar to how C code can change the contents of strings, which would break assumptions in other code).
In analysing this it is useful to be aware of where you are borrowing the reference from, and that promise is invalidated.
IMHO these shouldn’t be public. Setting the type can be done the same way as from python; Set the __class__ attribute of the instance, and setting the reference count is at best questionable. Both won’t work with tagged pointers.
There are some users, a fairly high-profile one is PyQt5 which has wheels the big tree platforms on PyPI that use the stable ABI.
:-).
P.S. I’m sorry that I sound fairly negative, that is not my intention. I’m all for improving the API, but it should be done for the right reasons and not just because changes *might* help with future developments and/or other implementations.
Ronald
Le jeu. 6 sept. 2018 à 13:30, Ronald Oussoren <ronaldoussoren@mac.com> a écrit :
PyTuple_GetItem should be fairly safe, if the callers owns a reference to the tuple itself. It is technically possible that tuples change, but that’s bad behaviour (similar to how C code can change the contents of strings, which would break assumptions in other code).
It seems like you completely missed my point about borrowed references. I rewrote the rationale to remove borrowed references using Antonio Cuni's email: https://pythoncapi.readthedocs.io/bad_api.html#problem-caused-by-borrowed-re...
In short, you are describing the exact implementation of CPython 3.7. I care of PyPy cpyext, other Python implementations, and also deep changes in CPython internals (things that I called experimental optimization ideas). In PyPy, there is no such thing as PyObject, so you cannot point to a PyObject, since it doesn't exist... Well, see the rationale.
Victor
On 9/5/2018 9:17 AM, Victor Stinner wrote:
I think Victor's approach is reasonable. It can be optimized once we know the issues. Or maybe it can't be, and reference counting is forever baked in the way it is now. It's hard to say without more experimenting, exactly of the type Victor is doing.
For some more detailed discussion about the problem with borrowed references, Larry's Gilectomy work is interesting. There's a discussion which begins by discussing weakrefs, but goes on to further discussion on weak references in general. It's worth reading in the context of this thread: https://mail.python.org/pipermail/python-dev/2016-October/146604.html
Eric
Le jeu. 6 sept. 2018 à 19:36, <eric+a-python-dev@trueblade.com> a écrit :
My hope is to be able to avoid reference counter internally, as PyPy does. But I don't think that today it's reasonable to try to get rid of reference counting for the public ("external") C API.
PyPy and Gilectomy are my two main motivations to fix the C API. CPython is not really my motivation here.
Victor
Victor Stinner schrieb am 05.09.2018 um 12:21:
You keep saying that, but I'm yet to see any proof for this broad assumption. I can see why macros that reach directly into data structures are a problem, e.g. PyTuple_GET_ITEM(), or functions like PyDict_GetItem() that return a borrowed reference to something that the caller does not control directly.
But that's not a problem of borrowed references per se. It's a problem with borrowed references that refer to something that is difficult to control for the user.
Macros like Py_TYPE() and Py_SIZE() are a good thing, because they make it easy to migrate APIs, not difficult.
Stefan
Le mer. 5 sept. 2018 à 19:27, Stefan Behnel <python_capi@behnel.de> a écrit :
Hum, I'm not sure that I explained correctly the issue that I see with borrowed references. I added the "Specialized list for small integers" optimization idea to explain the issue of function returning a borrowed reference to a temporary object:
https://pythoncapi.readthedocs.io/optimization_ideas.html#specialized-list-f...
I'm not sure that I understood your point. Do you mean that such optimization is compatible with PyTuple_GET_ITEM()?
Victor
On Wed, Sep 5, 2018 at 7:26 PM Stefan Behnel <python_capi@behnel.de> wrote:
I think that what Victor is trying to say is that borrowed references are a problem whenever you don't have a reference to borrow: they assume that a reified object already exists (and thus have a positive refcout), so that you just borrow it.
Tagged pointers are an example of this: since there is no concrete PyObject* to represent the integer, you cannot easily manipulate it.
PyPy has a similar problem with list strategies: if we have a list containing only integers, we store it as a compact C array of longs, and we create the W_IntObject only when we access an item (most of the time the W_IntObject is optimized away by the JIT, but this is another story).
But for cpyext, this is a problem: PyList_GetItem returns a borrowed reference, but we don't have any concrete PyObject* to return! The current solution is very bad: basically, the first time we do a PyList_GetItem we convert the *whole* list to a list of PyObject*, just to have something to return: https://bitbucket.org/pypy/pypy/src/b9bbd6c0933349cbdbfe2b884a68a16ad16c3a8a...
If PyList_GetItem returned a new reference, we could just allocate the PyObject* on the fly and destroy it when the user decref it. Basically, by putting borrowed references in the API, we are fixing in advance the data structure to use!
Le mer. 5 sept. 2018 à 00:12, Jeroen Demeyer <J.Demeyer@ugent.be> a écrit :
I'm trying to design an API which allows to remove the ob_type field from PyObject. I would to do that for many reasons. I wrote a whole website to explain why: https://pythoncapi.readthedocs.io/
In short:
- Leaking ob_field prevents to compile a C extension to the stable ABI
- Accessing directly ob_field prevents to change the structure and so experiment optimizations
- Accessing directly ob_field by derefering pointers expect that a pointer can be dereference which prevents to used tagged pointers
The current ("old") C API prevents to experiment many optimizations (in CPython or CPython forks), make cpyext of PyPy inefficient, and make it really hard to write a new Python implementation which supports the C API.
Victor
On Wed, Sep 5, 2018 at 8:18 AM Victor Stinner <vstinner@redhat.com> wrote:
I don't see how removing Py_TYPE() will make it easier to remove the ob_type field. I'd expect that it would make it even harder.
As someone who does use the C API from time to time, I have never thought of Py_TYPE (or other similar macros/functions) as returning a borrowed reference. To me, it's a C programming style that asks me to please use these macros/functions to access properties of "objects" because the API designer wants to be able to later change the struct underneath without rewriting code.
Programmers using the Python C API need a way to ask "what is the type of this object?" With the current API they can either use ob_type directly, or wrap it in Py_TYPE() which is (I guess?) the preferred style.
So if you want to remove ob_type, you can - as long as you keep Py_TYPE(). Phase 1, announce that ob_type will be removed in next version, or that it might be a tagged pointer and no longer usable as a C pointer. Phase 2, in the next version, rename ob_type and change Py_TYPE() to match. Existing code sitll runs, but compilation errors for any native extensions not using Py_TYPE(). Finally in phase 3 you can turn the type into a tagged pointer, handle, or whatever.
--
cheers,
Hugh FisherOn Wed, Sep 5, 2018 at 1:15 AM Hugh Fisher <hugo.fisher@gmail.com> wrote:
Borrowed references impose the condition on the caller that they must ensure the borrowed-from object stays alive (and keeps the borrowed object alive), but have the advantage that they relieve the caller from having to take an explicit action that they are now done with the result of the call, making code like Py_TYPE(obj)->tp_name possible. If the goal is to remove all borrowed references, I don't see a way around changing every single callsite that leverages borrowed references.
(It may also offer a performance advantage to avoid all the refcounting, e.g. when iterating over a container.)
Le mer. 5 sept. 2018 à 01:15, Hugh Fisher <hugo.fisher@gmail.com> a écrit :
I have multiple goals for the new C API:
- hide implementation details to allow to experiment new optimizations like tagged pointers
- hide implementation details to allow to modify deeply C structures used internally in CPython
- don't access C structures to generate machine code which would be compatible with multiple Python versions: "stable ABI"
I listed some ideas of optimizations: https://pythoncapi.readthedocs.io/optimization_ideas.html
My plan doesn't include implementing these optimizations, but prepare the code to allow to experiment multiple kinds of optimizations.
Again, you rely on the exact behaviour of the current ("old") C API where PyObject* is really a pointer to a concrete PyObject which exist somewhere in memory. In my plan, a PyObject* can be something else (ex: tagged pointer where a number is directly stored inside the "pointer").
I already modifiy Py_TYPE() macro to emit a function call, _Py_TYPE_impl(). Currently, it's called the "Py_NEWCAPI_NO_STRUCT" API. => this API is incompatible with tagged pointer because it returns a borrowed reference
I also implemented an API without Py_TYPE() called the "Py_NEWCAPI" API... => no more borrored reference, ready for tagged pointer for example
... but removing Py_TYPE() simply broke all C extensions of CPython. So I reverted temporary this change and started this discussion.
It seems like I have to add a new Py_GetType() function which returns a strong reference (Py_INCREF) and start to modify C extension for use it instead of Py_TYPE().
Victor
[…]
Just curious, but why?
Now that CI/CD systems are getting mainstream having extensions that work with multiple python versions is less and less useful. For PyObjC I create binary wheels for Python 2.7, 3.4, 3.5, 3.6 and 3.7 and for 3 of them I create 2 variants of those wheels (on two different versions of macOS). The only cost compared to using a stable ABI is time: building a lot of wheels takes more time than building just a few. A lot of projects could just use Travis/CircleCI/… to generate wheels and upload those to PyPI (or even automate that part in their CI/CD pipeline)
IMHO the PEP 384 way of defining types is also less friendly to developers because you don’t get compiler warnings/errors when the prototype of functions you use for slots don’t match.
Ronald
On Wed, Sep 5, 2018 at 4:44 AM Ronald Oussoren via capi-sig <capi-sig@python.org> wrote:
That's a lot to ask of every single C extension unless the infrastructure is provided by the PSF and basically trivial to set up
as in, register with PyPI, provide the source, and get all of your wheels for free. And it doesn't help for PyPy or IronPython or Jython, all of which are hamstrung by not really being able to use C extensions, and also unlikely to make the list for most extensions. A stable ABI (and opaque PyObjects) would make it much easier to support alternative implementations. (Although, using a special build of extensions just for IronPython isn't something I ever really considered, and may need another look.)
Jeff
Why is that? Most CIs have a free tier that can be used by open-source projects, which is most if not all packages on PyPI. The hardest part is writing the scripting that collects build artefact from the CI/CD and upload them to PyPI.
BTW. See for example <https://github.com/joerick/cibuildwheel <https://github.com/joerick/cibuildwheel>> for a script that automates most of the wheel creation for Windows, macOS and Linux.
Would the stable ABI really help the other python implementations? I honestly don’t know, although its obvious that the stable ABI makes it easier to reach compatibility because there is less to emulate.
Ronald
A stable ABI by itself doesn't help. But a trimmed down ABI which does not hardcode implementation details is surely helpful. Many implementation details are too deeply hardcoded by now and basically impossible to remove: reference counting, for example; but e.g., the work Victor is doing on killing borrowed references is surely helpful for us.
Maybe for PyPy it's too late: if we want to be relevant in today's world, we need to emulate all this crap anyway. But who knowns, maybe in 10 years another alternative python implementation will benefit from this.
Just to add a data point:
The stable API was mainly meant platforms such as Windows which at the time did not provide free compilers to compile your own extensions.
Nowadays, I think this is no longer a big concern, since VC++ compilers are freely available. Since extension writers have to test whether their extensions compile against newer Python versions anyway, many will have the necessary infrastructure for the Python versions they support in place.
That said: Having a PSF hosted infrastructure for compiling extensions for many different platforms would certainly be a great thing to put in place.
I know that Ben Nuttall, for example, build something like this for the Raspberry Pi: it tries to compile all PyPI packages from source and then provides the binaries for download:
https://bennuttall.com/piwheels-building-a-faster-python-package-repository-...
On 06.09.2018 19:17, Antonio Cuni wrote:
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 06 2018)
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
Le ven. 7 sept. 2018 à 09:24, Ronald Oussoren <ronaldoussoren@mac.com> a écrit :
Maybe earlier if this efforts leads to an API that’s more friendly for PyPy.
Is there anything you can share about what API details are particularly problematic?
I'm in discussion with PyPy developers, and they reported different APIs which cause them troubles:
- in general, any API addition means more work for them
- anything which use borrowed references
- PyObject** should be banned: "PyObject** PySequence_Fast_ITEMS(ob)" has to go https://pythoncapi.readthedocs.io/bad_api.html#array-of-pointers-to-python-o...
- the finalizer API: I guess that it's an issue because of their different garbage collector
- almost all PyUnicode API functions have to go according to them. PyPy3 uses UTF-8 internally, CPython uses "compact string" (array of Py_UCS1, Py_UCS2 or Py_UCS4 depending on the string content). https://pythoncapi.readthedocs.io/bad_api.html#pypy-requests
I tried to list all PyPy requests on this "Bad C API" page.
I'm ok with removing PyObject** and most PyUnicode APIs, but I'm not sure about the finalizer API which seems to be a great enhancement of Python regarding to reference cycles.
Victor
I'm -1 on removing the PyUnicode APIs. We deliberately created a useful and very complete C API for Unicode.
The fact that PyPy chose to use a different internal representation is not a good reason to remove APIs and have CPython extension take the hit as a result. It would be better for PyPy rethink the internal representation or create a shim API which translates between the two worlds.
Note that UTF-8 is not a good internal representation for Unicode if you want fast indexing and slicing. This is why we are using fixed code units to represent the Unicode strings.
On 07.09.2018 10:22, Victor Stinner wrote:
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 07 2018)
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
This is something thats completely off-topic for this discussion, but I wonder if fast indexing and slicing are really necessary. Even with our current representation doing slicing correctly is hard due to combining characters and emoji. Any changes in this regard would probably require changes to the string API and/or additional utilities.
Ronald
On 07.09.2018 10:48, Ronald Oussoren wrote:
You have to differentiate between indexing code points (which is what Python provides) or graphemes (combined code points).
Python doesn't have good support for the latter at the moment, but then again: most text processing also doesn't necessarily need this, otherwise people would have stepped in to add better support. I had suggested doing this soon after Python 2.0 received Unicode, but there was no interest, so I dropped the idea again:
https://mail.python.org/pipermail/python-dev/2001-July/015938.html
Now, UTF-8 has the same issues at the code unit level, as combined code points have at the Unicode level, but the situation is much worse: you enter this minefield as soon as you leave ASCII, which is a much more common situation than having to deal with accidentally breaking combining code points with a slice in Unicode.
To address this, you need an index into UTF-8 and then you lose the perceived better memory efficiency of UTF-8 again. On top your code gets a lot more complex.
BTW: I do wonder why you think that fast indexing and slicing are not needed. Pretty much all text parsing relies on this heavily.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 07 2018)
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
Not necessarily, it is often possible to use higher-level API (str.startswith, str.split, …) or regular expressions.
But: I also wrote that there probably need to be new APIs to fully remove the need for fast indexing on code points.
Ronald
On 07.09.2018 11:58, Ronald Oussoren wrote:
Well, yes, but those APIs use indexing and slicing for creating their return values.
But: I also wrote that there probably need to be new APIs to fully remove the need for fast indexing on code points.
I don't see how this would work in general.
You can probably get by with a few APIs to deal with short strings, but for any longer chunks of text, you would not want to first convert this into multiple shorter strings for processing (say lines of text). Instead, you'd try to avoid any copying or creation of temporary objects and always index directly into the larger text, identify the parts you need and then only work on those slices.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 07 2018)
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
M.-A. Lemburg schrieb am 07.09.2018 um 12:58:
Not necessarily at the C level, though, which is where pretty much all of them are implemented. In a UTF-8 string, search could use memmem() and not rely on character by character traversal, because the byte sequence is self-synchronising. And the splitting then just copies out bytes.
What is a problem then, is obviously finding out the character *position* of a substring, if that's requested from Python code. That would either require a character index (which costs memory) or counting characters (which costs time), as you noted. Even regexes can run into that problem, when users request a group position, for example. Python code that does its own slicing will always be impacted.
So, as always, the end result depends on the application. If indexes are important per se, it's costly. Otherwise, UTF-8 strings may be faster or not make a difference at all.
Stefan
On Fri, Sep 7, 2018 at 10:32 AM M.-A. Lemburg <mal@egenix.com> wrote:
I don't agree; an API should not dictate which data structure and/or internal representation to use, else it quickly become impossible to innovate and experiment.
Yes, we can (and will) create a translation layer, but it has a performance and complexity cost.
I don't want to debate whether this is a good requirement or not, I'll let others to do that :) But note that PyPy *does* support fast indexing and slicing in its utf8 branch: the basic idea is to create (lazily) an index to map between logical index in the str and physical offesets in the array.
On Fri, Sep 7, 2018 at 9:24 AM Ronald Oussoren <ronaldoussoren@mac.com> wrote:
yes, but the current status is that we already implemented 99% of the current API; before we are going to kill it, we would need to wait until most/all current C modules are rewritten using the new API. Considering what happened for py3k, 10 years sounds like a conservative estimate :)
Is there anything you can share about what API details are particularly problematic?
I am writing a blog post which tries to explain why cpyext is hard, the early draft is here: https://bitbucket.org/pypy/extradoc/src/extradoc/blog/draft/2018-09-cpyext/c...
it doesn't contain (yet) specific examples of problematic API, but should give a rough understanding of what are the main challenges. The biggest challenge is reference counting: ideally, if we had a C API which is compatible with moving GC, it would make life much easier to us (and probably to Jython, IronPython, etc.), but I think that it's impractical.
On Thu, Sep 6, 2018 at 10:06 AM Ronald Oussoren <ronaldoussoren@mac.com> wrote:
It's mostly that every project needs to go through the configuration steps for each platform it wants to support, and have access to test runners for those platforms. The combinatorial explosion is painful. Scripts help, obviously, but "n_platforms" is always goings to be less than "n_platforms * m_python_implementations". Also, I don't think we should be dependent on the generosity of commercial providers.
For IronPython, at least, it doesn't map 1:1 with CPython versions. Maintaining a C API shims across multiple versions is a lot of work. If extensions were compiled against "stable ABI 4", and we could assume that stable ABI 4 would never change, that would simplify things. I'm sure it could do this today, but I don't think there's a way for an extension to report what version it's compiled against.
Right now I don't think it's possible to use a 3.6 extension on 3.7 without recompiling, but in the future, maybe CPython 4.2 could understand CPython 4.0 and 4.1 extensions by checking the API version. (I think this is how Android's API levels work, more or less.) IronPython (and PyPy, etc.) would then only have to worry about what ABI version it's targeting, not what CPython version that means.
It does potentially make it harder to make changes to CPython internals, if that sort of compatibility has to be preserved, but it might also allow for *more* changes if it limits how extensions can muck around with CPython's internal data structures.
- Jeff
On Wed, Sep 5, 2018 at 8:29 PM Victor Stinner <vstinner@redhat.com> wrote:
...
No I am not - that's the whole point of using Py_TYPE(). I pass in an object, I get back another object, which is the type of the argument. Under the current API it's always a pointer. If you change ints, or booleans, or any other values into tagged pointers and at the same time modify Py_TYPE() to handle tagged pointers correctly, all my code continues to work. If you change Py_TYPE() into a function instead of a macro, all my code will still work.
And this happens to be exactly what Apple did when they added tagged pointers to Objective-C. They told programmers not to access objectptr->isa directly, instead to use OBJECT_GETCLASS(objectptr). Then later they started implementing some objects as tagged pointers, and only code that hadn't been updated broke. OBJECT_GETCLASS was the "stable ABI"
I predict this will lead to more bugs, not less.
Firstly, the majority of C extension methods have no effect on object lifetimes. Py_TYPE() and other functions are used on either self, or one of the arguments. If the object doesn't disappear while the method is executing, any borrowed reference that is only used within that C method remains valid until the end of the method. The only way it can cause problems if the borrowed reference is stored somewhere else, but that is very rare.
With your proposal, every time I use Py_GetType() within a C method I have to remember to store that object reference somewhere and Py_DECREF it before returning. So no, it won't be just modifying extensions to use it instead of Py_TYPE(). You'll also have to modify the code to store the reference, and add more code to release it. More code means more chances to get it wrong.
Secondly and more generally, I'm not sure it is possible to maintain invariants, such as "all references are correctly ref counted", within the code that is responsible for maintaining and implementing the invariants themselves. An implementation has to be able to break the rules internally.
--
cheers,
Hugh FisherAll of these are unrelated to borrowed references. Conceptually every instance owns a reference to its type, regardless of how and where that reference is stored, and it should be possible to return a borrowed reference.
As others have mentioned before it is unclear if your plans w.r.t. creating a new API will help in this regard.
Ronald
Le mer. 5 sept. 2018 à 09:17, Ronald Oussoren <ronaldoussoren@mac.com> a écrit :
You are describing the current ("old") C API. I ask you to imagine a new C API where PyObject* is really an opaque thing. It's better to even see as an integer (Py_uintptr_t) rather than a pointer.
I would like to use "PyObject*" as an opaque thing because I would like to be able to implement tagged pointer for example. If you store an integer inside a "PyObject*", there is no PyObject structure anywhere. There is no linked PyTypeObject anywhere.
But all existing C extensions really want a "PyObject*" which behaves as it's a real Python object with an associated PyObject structure linked to a concrete PyTypeObject. So in some places, we need to produce temporary PyLongObject which represent the integer stored in the tagged pointer.
Borrowed references are a blocker issue for that. How do you know when the temporary object should go away?
Tagged pointer is one example. Another example would be a list of integers which stores numbers are C integers (ex: int32_t) rather than PyLongObject. PyList_GetItemRef() has to create a temporary PyLongObject.
Victor
On 09/05/18 12:21, Victor Stinner wrote:
The tagged pointer optimization can only be used for a limited set of basic types int, bool, small str/bytes etc. Right? Can you make these basic type objects immortal? Their refcount would be ignored, their dealloc would be a no-op. If you need a heap type for some reason it would need to be destroyed specially at interpreter shutdown.
I used tagged pointers with integers because it's an example simple to understand. But the issue is wider than just this very specific use case. IMHO we should not rely on borrowed references of Py_TYPE() and use strong references.
Victor Le mer. 5 sept. 2018 à 13:32, Petr Viktorin <encukou@gmail.com> a écrit :
On Wed, Sep 5, 2018 at 10:12 PM Victor Stinner <vstinner@redhat.com> wrote:
In case of Py_TYPE(), I still don't understand what's wrong about borrowed reference. Py_TYPE(ob) borrows reference from ob. So Py_DECREF(ob) may break the borrowed reference. It's straight enough.
Int or str may be embedded as tagged pointer. Py_TYPE(ob) should return int or str type when ob is it's instance. It's not bad. All tagged-value-able types should be immortal -- never freed.
-- INADA Naoki <songofacandy@gmail.com>
But that’s almost entirely unrelated to the issue of using borrowed references for Py_TYPE. Every Python object has a type, and hence owns a reference to a type object regardless of how that’s implemented.
Borrowed references have there problems, but do make the CPython API easier to use. I’m all for making the API safer to use, but IMHO eradicating the use of borrowed references is not necessary for that. There are APIs where moving to a variant without borrowed references would be better (in particular PyList_GetItem, which is a ticking bomb if you don’t INCREF the result before calling a CPython API)
There’s two related issues here:
*All* existing PyObjects contain a “PyObject” structure (although that’s hidden in PyObject_HEAD and should be easy to remove in all extensions that are properly ported to Python 3)
Some macros and function currently assume that 1) is true (such as Py_TYPE, Py_REFCNT, Py_INCREF, Py_DECREF). Those can be changed to do something else, without a need to introduce temporary objects.
The most problematic part of 2) is that Py_REFCNT is an lvalue and some code might exploit this to change the refcount of an object, but that’s arguably inherently bad code.
Both require changes to the stable ABI though.
BTW. Both the refcount macros and Py_TYPE are called a lot, unconditionally turning them into external functions could be quite costly performance wise.
Having a go at tagged points has been on my wishlist for a long time, but I haven’t managed do do anything about that yet beyond some thinking. The second example would be less interesting with tagged pointers (although elements would necessarily be the same size as a pointer, which can use more memory than the minimal storage).
Ronald
Le mer. 5 sept. 2018 à 13:36, Ronald Oussoren <ronaldoussoren@mac.com> a écrit :
Borrowed references have there problems, but do make the CPython API easier to use. I’m all for making the API safer to use, but IMHO eradicating the use of borrowed references is not necessary for that.
I don't really care about designing an API easy to use. Right now, I prefer to focus on correctness and design the ideal API. Once we know what is the ideal API and see the cost to reach it, we can start talking about tradeoff and milestones to reach the final goal.
There are APIs where moving to a variant without borrowed references would be better (in particular PyList_GetItem, which is a ticking bomb if you don’t INCREF the result before calling a CPython API)
I removed PyTuple_GetItem() from Py_NEWCAPI, replaced with PyTuple_GetItemRef(). I will do the same for PyList_GetItem() and other variants.
The most problematic part of 2) is that Py_REFCNT is an lvalue and some code might exploit this to change the refcount of an object, but that’s arguably inherently bad code.
I introduced _Py_SET_TYPE() and _Py_SET_REFCNT() for that. Maybe I should make them public, I don't know yet.
Both require changes to the stable ABI though.
The stable ABI is not usable yet, so I don't think that anyone is using it right now. My plan is to make it much easier to use it, and later maybe make it the default.
BTW. Both the refcount macros and Py_TYPE are called a lot, unconditionally turning them into external functions could be quite costly performance wise.
I would prefer to not guess anything about performance. First experiment the change, run benchmark, then decide about the tradeoff between performance and correctness.
I'm working on a fork of CPython. I will not touch CPython upstream until we have enough information to get wise decisions ;-)
Victor
I do because an easy to use C API is one of the reasons why Python is as popular as it is.
One thing that might help with correct C API usage is to teach static analysis tools about the API and correct usage thereof, see <https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/ <https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/>> for an experimenal example of this from the past. Doing this might also help in finding APIs that are hard to reason about, and finding alternatives for them. Apple added something similar to clang before they switched to automatic reference counting (ARC) in ObjC.
PyTuple_GetItem should be fairly safe, if the callers owns a reference to the tuple itself. It is technically possible that tuples change, but that’s bad behaviour (similar to how C code can change the contents of strings, which would break assumptions in other code).
In analysing this it is useful to be aware of where you are borrowing the reference from, and that promise is invalidated.
IMHO these shouldn’t be public. Setting the type can be done the same way as from python; Set the __class__ attribute of the instance, and setting the reference count is at best questionable. Both won’t work with tagged pointers.
There are some users, a fairly high-profile one is PyQt5 which has wheels the big tree platforms on PyPI that use the stable ABI.
:-).
P.S. I’m sorry that I sound fairly negative, that is not my intention. I’m all for improving the API, but it should be done for the right reasons and not just because changes *might* help with future developments and/or other implementations.
Ronald
Le jeu. 6 sept. 2018 à 13:30, Ronald Oussoren <ronaldoussoren@mac.com> a écrit :
PyTuple_GetItem should be fairly safe, if the callers owns a reference to the tuple itself. It is technically possible that tuples change, but that’s bad behaviour (similar to how C code can change the contents of strings, which would break assumptions in other code).
It seems like you completely missed my point about borrowed references. I rewrote the rationale to remove borrowed references using Antonio Cuni's email: https://pythoncapi.readthedocs.io/bad_api.html#problem-caused-by-borrowed-re...
In short, you are describing the exact implementation of CPython 3.7. I care of PyPy cpyext, other Python implementations, and also deep changes in CPython internals (things that I called experimental optimization ideas). In PyPy, there is no such thing as PyObject, so you cannot point to a PyObject, since it doesn't exist... Well, see the rationale.
Victor
On 9/5/2018 9:17 AM, Victor Stinner wrote:
I think Victor's approach is reasonable. It can be optimized once we know the issues. Or maybe it can't be, and reference counting is forever baked in the way it is now. It's hard to say without more experimenting, exactly of the type Victor is doing.
For some more detailed discussion about the problem with borrowed references, Larry's Gilectomy work is interesting. There's a discussion which begins by discussing weakrefs, but goes on to further discussion on weak references in general. It's worth reading in the context of this thread: https://mail.python.org/pipermail/python-dev/2016-October/146604.html
Eric
Le jeu. 6 sept. 2018 à 19:36, <eric+a-python-dev@trueblade.com> a écrit :
My hope is to be able to avoid reference counter internally, as PyPy does. But I don't think that today it's reasonable to try to get rid of reference counting for the public ("external") C API.
PyPy and Gilectomy are my two main motivations to fix the C API. CPython is not really my motivation here.
Victor
Victor Stinner schrieb am 05.09.2018 um 12:21:
You keep saying that, but I'm yet to see any proof for this broad assumption. I can see why macros that reach directly into data structures are a problem, e.g. PyTuple_GET_ITEM(), or functions like PyDict_GetItem() that return a borrowed reference to something that the caller does not control directly.
But that's not a problem of borrowed references per se. It's a problem with borrowed references that refer to something that is difficult to control for the user.
Macros like Py_TYPE() and Py_SIZE() are a good thing, because they make it easy to migrate APIs, not difficult.
Stefan
Le mer. 5 sept. 2018 à 19:27, Stefan Behnel <python_capi@behnel.de> a écrit :
Hum, I'm not sure that I explained correctly the issue that I see with borrowed references. I added the "Specialized list for small integers" optimization idea to explain the issue of function returning a borrowed reference to a temporary object:
https://pythoncapi.readthedocs.io/optimization_ideas.html#specialized-list-f...
I'm not sure that I understood your point. Do you mean that such optimization is compatible with PyTuple_GET_ITEM()?
Victor
On Wed, Sep 5, 2018 at 7:26 PM Stefan Behnel <python_capi@behnel.de> wrote:
I think that what Victor is trying to say is that borrowed references are a problem whenever you don't have a reference to borrow: they assume that a reified object already exists (and thus have a positive refcout), so that you just borrow it.
Tagged pointers are an example of this: since there is no concrete PyObject* to represent the integer, you cannot easily manipulate it.
PyPy has a similar problem with list strategies: if we have a list containing only integers, we store it as a compact C array of longs, and we create the W_IntObject only when we access an item (most of the time the W_IntObject is optimized away by the JIT, but this is another story).
But for cpyext, this is a problem: PyList_GetItem returns a borrowed reference, but we don't have any concrete PyObject* to return! The current solution is very bad: basically, the first time we do a PyList_GetItem we convert the *whole* list to a list of PyObject*, just to have something to return: https://bitbucket.org/pypy/pypy/src/b9bbd6c0933349cbdbfe2b884a68a16ad16c3a8a...
If PyList_GetItem returned a new reference, we could just allocate the PyObject* on the fly and destroy it when the user decref it. Basically, by putting borrowed references in the API, we are fixing in advance the data structure to use!
participants (12)
-
Antonio Cuni -
eric+a-python-dev＠trueblade.com -
Hugh Fisher -
INADA Naoki -
Jeff Hardy -
Jeroen Demeyer -
M.-A. Lemburg -
Petr Viktorin -
Robert Bradshaw -
Ronald Oussoren -
Stefan Behnel -
Victor Stinner