[Python-Dev] Const access to CPython objects outside of GIL?

[Tim Peters]

[Radim Řehůřek <radim at rare-technologies.com>]

> one of our Python projects calls for pretty heavy, low-level optimizations.


> We went down the rabbit hole and determined that having access to
> PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict, unicode)
> on Python objects **outside of GIL** might be a good-enough solution. The
> Python objects in question are guaranteed to live and not be mutated
> externally in any way. They're "frozen" and read-only.
>
> Under what conditions is it OK to call these 3 functions on such objects?
>

>From the "initialization, Finalization, and Threads" section of the
Python/C API Reference Manual:

Therefore, the rule exists that only the thread that has acquired the GIL
> may operate on Python objects or call Python/C API functions.


Under protection of the GIL is the only documented - or intended - way to
do anything with Python objects, or to call virtually any Python/C API
function.



> More generally, what is the CPython 2.7/3.5 contract regarding (lack of)
> object mutation, and the need for reference counting and synchronization
> via GIL?
>

There is no intention to support GIL-free access to any Python objects.  So
that's the contract:  "All warranties are null & void if you do just about
anything while not holding the GIL".


> Which C API functions are safe to call on "const" objects?
>

None.  If you find some combinations of CPython versions, functions, and
objects, that happen to work, that's fine, but there's no guarantee they'll
continue to work, not even across maintenance releases.  Although they'll
_probably_ continue to work.

Obviously releasing GIL and then calling C API is hacky, but from initial
> experiments, it seems to work (see
> https://stackoverflow.com/questions/51351609/can-i-const-access-cpython-objects-without-gil).
> But I'm wondering if there's a more formal contract around this behaviour.
>

Nope!  You're on your own here.

Which doesn't mean you can't do it.  Just that if things blow up, you're
still on your own - since you're far outside what the documentation says is
required (which I suspect you knew before you asked ;-) ), the project
won't even accept a bug report.

If you want more help trying to guess which functions _may_ work outside
the law, that's fine too, but "python-dev" (this mailing list) is for
development _of_ Python itself, not for questions about _using_ Python.
The more general python-list would be more appropriate.  But, since you're
trying to use the C API in unsupported ways it wasn't intended to be used,
you'll likely have a hard time finding people with significant experience
doing the same thing.  Since it's not at all an intended use, there are no
"general principles" at work to ease the pain of staring at the CPython
source code to try to guess what might go wrong across all code paths.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180717/794a277b/attachment-0001.html>