[Python-Dev] Const access to CPython objects outside of GIL?

[Radim Řehůřek]

Thanks Tim. That's one unambiguous answer.

I did consider posting to python-list, but this seemed somewhat
python-devish. Any appetite for documenting which foundational functions
are const-safe in the sense I described? Perhaps we could help. What kind
of analysis, demand or momentum is needed for the Python dev team to
consider introducing such a contract?

To be honest, I did do some CPython source code staring already. And at
least for the functions we care about, it wasn't all that painful
(PyDict_GetItem being the trickiest). Doing this wholesale might be a
superhuman task, but I'm thinking a practical subset could be relatively
straightforward while still useful, 80/20.



On Tue, Jul 17, 2018 at 7:15 AM, Tim Peters <tim.peters at gmail.com> wrote:

> [Radim Řehůřek <radim at rare-technologies.com>]
>
>> one of our Python projects calls for pretty heavy, low-level
>> optimizations.
>
>
>> We went down the rabbit hole and determined that having access to
>> PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict, unicode)
>> on Python objects **outside of GIL** might be a good-enough solution. The
>> Python objects in question are guaranteed to live and not be mutated
>> externally in any way. They're "frozen" and read-only.
>>
>> Under what conditions is it OK to call these 3 functions on such objects?
>>
>
> From the "initialization, Finalization, and Threads" section of the
> Python/C API Reference Manual:
>
> Therefore, the rule exists that only the thread that has acquired the GIL
>> may operate on Python objects or call Python/C API functions.
>
>
> Under protection of the GIL is the only documented - or intended - way to
> do anything with Python objects, or to call virtually any Python/C API
> function.
>
>
>
>> More generally, what is the CPython 2.7/3.5 contract regarding (lack of)
>> object mutation, and the need for reference counting and synchronization
>> via GIL?
>>
>
> There is no intention to support GIL-free access to any Python objects.
> So that's the contract:  "All warranties are null & void if you do just
> about anything while not holding the GIL".
>
>
>> Which C API functions are safe to call on "const" objects?
>>
>
> None.  If you find some combinations of CPython versions, functions, and
> objects, that happen to work, that's fine, but there's no guarantee they'll
> continue to work, not even across maintenance releases.  Although they'll
> _probably_ continue to work.
>
> Obviously releasing GIL and then calling C API is hacky, but from initial
>> experiments, it seems to work (see https://stackoverflow.com/
>> questions/51351609/can-i-const-access-cpython-objects-without-gil). But
>> I'm wondering if there's a more formal contract around this behaviour.
>>
>
> Nope!  You're on your own here.
>
> Which doesn't mean you can't do it.  Just that if things blow up, you're
> still on your own - since you're far outside what the documentation says is
> required (which I suspect you knew before you asked ;-) ), the project
> won't even accept a bug report.
>
> If you want more help trying to guess which functions _may_ work outside
> the law, that's fine too, but "python-dev" (this mailing list) is for
> development _of_ Python itself, not for questions about _using_ Python.
> The more general python-list would be more appropriate.  But, since you're
> trying to use the C API in unsupported ways it wasn't intended to be used,
> you'll likely have a hard time finding people with significant experience
> doing the same thing.  Since it's not at all an intended use, there are no
> "general principles" at work to ease the pain of staring at the CPython
> source code to try to guess what might go wrong across all code paths.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180717/31d90981/attachment.html>