[Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

[MRAB]

On 2016-10-10 18:50, Nathaniel Smith wrote:
> On Mon, Oct 10, 2016 at 10:03 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> > On 10 October 2016 at 17:49, MRAB <python at mrabarnett.plus.com> wrote:
> >> If you lookup something in a dict, it'll be a borrowed reference.
> >>
> >> If the dict is globals() and there's no GIL, another thread could delete the
> >> item while your code had the borrowed reference.
> >>
> >> It looks like there might be a lot that will need to changed post gilectomy!
> >
> > It seems to me that the whole concept of a borrowed reference may be
> > risky in a post-GIL world. There may be occasional cases where it's
> > possible to prove that borrowing is safe, but I suspect they'll be
> > pretty rare.
>
> I don't think it's that bad...
>
> In a post-GIL world, the problem cases we're talking about like
> deleting items from containers all require some sort of locking, even
> before we start thinking about borrowed references. If two threads
> start mucking around resizing the internals of a list or dict at the
> same time, then you are unlikely to go to space today. IIRC to handle
> this gilectomy adds per-object mutexes that you have to hold whenever
> you're mucking around with that object's internals.
>
> If we say that borrowing reference from a dict is one of the things
> that counts as mucking about with that dict, and thus requires you to
> hold the dict lock for as long as you hold the borrowed reference,
> then all should be well.
>
> I assume Larry is way ahead of us on this, which is why he's suddenly
> confusing us all by emphasizing that when you borrow a reference you
> should know who you're borrowing it from, so you know which lock to
> hold.
>
Instead of locking the object, could we keep the GIL, but have it 
normally released?

A thread could then still call a function such as PyWeakref_GetObject() 
that returns a borrowed reference, but only if it's holding the GIL. It 
would be able to INCREF the reference before releasing the GIL again.

There could still be a new function that returned a strong reference.