[Python-Dev] Const access to CPython objects outside of GIL?

[Ivan Pozdeev]

On 17.07.2018 7:18, Radim Řehůřek wrote:
> Hi all,
>
> one of our Python projects calls for pretty heavy, low-level 
> optimizations.
>
> We went down the rabbit hole and determined that having access to 
> PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict, 
> unicode) on Python objects **outside of GIL** might be a good-enough 
> solution. The Python objects in question are guaranteed to live and 
> not be mutated externally in any way. They're "frozen" and read-only.
>

The standard practice if you need to access something outside of GIL is 
to get a private C object from it.
For immutable types, you can get a pointer to the underlying data if the 
internal representation is compatible with some C type (e.g. char* 
|PyBytes_AsString|(PyObject/ *o/) and FILE* 
|PyFile_AsFile|(PyObject/ *p/) (Py2 only -- in Py3, PyFile no longer 
wraps stdio FILE) ); otherwise, the C API can produce a copy (e.g. 
wchar_t* |PyUnicode_AsWideCharString|(PyObject/ *unicode/, 
Py_ssize_t/ *size/) ).

Though you can call whatever you want outside of GIL (it's not like we 
can prevent you), anything that's not officially guaranteed you're doing 
on your own risk. Even if it happens to work now, it can break in 
unpredictable ways at any point in the future.

> Under what conditions is it OK to call these 3 functions on such objects?
>
> More generally, what is the CPython 2.7/3.5 contract regarding (lack 
> of) object mutation, and the need for reference counting and 
> synchronization via GIL?
>
> Which C API functions are safe to call on "const" objects?
>
> Obviously releasing GIL and then calling C API is hacky, but from 
> initial experiments, it seems to work (see 
> https://stackoverflow.com/questions/51351609/can-i-const-access-cpython-objects-without-gil). 
> But I'm wondering if there's a more formal contract around this behaviour.
>
> Cheers,
> Radim
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru

-- 
Regards,
Ivan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180717/e23f5860/attachment-0001.html>