[Python-Dev] Const access to CPython objects outside of GIL?

[Tim Peters]

[Radim Řehůřek <radim at rare-technologies.com>]

> Thanks Tim. That's one unambiguous answer.


I'm glad that part came across ;-)


> I did consider posting to python-list, but this seemed somewhat
> python-devish.
>

I agree it's on the margin - I don't mind.

Any appetite for documenting which foundational functions are const-safe in
> the sense I described? Perhaps we could help. What kind of analysis,
> demand or momentum is needed for the Python dev team to consider
> introducing such a contract?
>

Most are volunteers scratching their own itches, and sometimes paid to
scratch their employers' itches.  This isn't my itch, so not me.  Perhaps
someone else - but from the messages so far it appears that most who
replied suspect this is an "XY problem":

    https://en.wikipedia.org/wiki/XY_problem

Python's been around for decades, and best I recall nobody has suggested
anything really akin to what you're asking for here.  There certainly are C
extensions that want GIL-free access to (possibly massive amounts of) data
visible from Python too.  `numpy` is the prime example of that.  As Antoine
hinted, people working on those came up with the "buffer protocol" instead
(documented in the Python/C API Reference Manual under the "Buffer
Protocol" section).

But nobody here knows anything about your app, so can't guess from here
whether the buffer protocol is of any interest to you.  The buffer protocol
is aimed at uniform layouts of (possibly massive amounts of) native
C-format data, not at GIL-free access to individual little Python objects.

To be honest, I did do some CPython source code staring already.


Which is the only way you can learn anything about whether breaking
fundamental rules can get you in trouble.


> And at least for the functions we care about, it wasn't all that painful
> (PyDict_GetItem being the trickiest). Doing this wholesale might be a
> superhuman task, but I'm thinking a practical subset could be relatively
> straightforward while still useful, 80/20.
>

If you want to pursue this, it's probably necessary to first specify the
set of API functions that need to change their requirements and promises,
and to specify exactly what they're supposed to require and promise instead.

Even then, you should expect resistance.  It's been, historically, hard
enough to avoid thread bugs in CPython's implementation even with the
blanket "no access without the GIL, period" current requirement.  Adding
some number of "but these functions don't care about the GIL, in some
circumstances they can't verify obtain, let alone enforce" exceptions would
complicate an area that's already delicate.

CPython's implementation not only requires that only one thread holds the
GIL, it also works to verity that at runtime, and raises a fatal exception
if it detects that it's even slightly confused about which thread currently
holds the GIL.  Adding weaker but unverifiable preconditions to some
functions isn't attractive on the face of it.

Note:  the kind of people who find the GIL extremely intrusive tend instead
to work on ways to eliminate the GIL entirely.  They typically give up
after a few years of intense pain ;-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180717/ddf82182/attachment-0001.html>