[Python-Dev] Const access to CPython objects outside of GIL?

[Victor Stinner]

2018-07-17 6:18 GMT+02:00 Radim Řehůřek <radim at rare-technologies.com>:
> one of our Python projects calls for pretty heavy, low-level optimizations.
>
> We went down the rabbit hole and determined that having access to
> PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict, unicode)
> on Python objects **outside of GIL** might be a good-enough solution. The
> Python objects in question are guaranteed to live and not be mutated
> externally in any way. They're "frozen" and read-only.

IMHO it's a great path to introduce very tricky race conditions in
multithreaded applications! :-)

At the C level, even immutable Python objects are mutable: str, tuple, etc.

IMHO you need a different approach to implement optimizations. For
example, use your objects which don't rely on the GIL to be
consistent. Sadly, I have no experience with that and so cannot
provide any example.

Python releases the GIL *often* and pass pointers to buffers to C
functions. For example, os.stat() writes into a memory block allocated
by Python. But it's a raw memory block, it's much simpler than a
Python object like a tuple.

Victor