[Python-Dev] PyWeakref_GetObject() borrows its reference from... whom?

[MRAB]

On 2016-10-10 22:14, Larry Hastings wrote:
>
> On 10/10/2016 09:36 PM, Chris Angelico wrote:
>> Hmm. Here's a naughty, and maybe dangerous, theory. Obtain a "memory
>> deallocation lock". While it is held (by any thread - it's a guard,
>> more than a lock), Py_DECREF will not actually deallocate memory -
>> objects can fall to zero references without being wiped. Once the
>> lock/guard is freed/cleared, anything that had fallen to zero is now
>> deallocated. This probably would mean stuffing them onto a list of
>> "doomed objects", and upon release of the guard, any doomed objects
>> that still have no refs would get deallocated.
>
> If this worked, this would actually be really easy with my current
> "buffered reference counting" approach.  I literally already have a list
> of doomed objects, which a separate thread queues up for each thread to
> process later.
>
> (This was a necessary part of "buffered reference counting", in which
> reference count changes are stored in a transaction log and later
> committed by a single thread.  Since there's only one thread making
> reference count changes, there's no contention, so it doesn't have to
> use atomic incr/decr, which is a big performance win.)
>
> But I don't think this fixes the problem.  Consider:
>
>  1. Thread A calls Q = PyList_GetItem(L, 0), which returns a borrowed
>     reference.  Thread A then gets suspended, before it has a chance to
>     Py_INCREF(Q).
>  2. Thread B does L.clear(), the reference count of Q goes to 0, Q is
>     added to the doomed objects list.
>  3. Thread A gets to run again.  It does Py_INCREF(Q); Q's refcount is
>     now back up to 1, as if Q had been resurrected.
>  4. At some point in the future the "memory deallocation lock" is
>     released and Q is deleted.
>
> If you say "well, just look at the reference count, and if it's 1 throw
> it off the doomed objects list", keep in mind that I no longer have
> accurate real-time reference counts.  These hacks where we play games
> with the reference count are mostly removed in my branch.
>
> Also, I don't know when it would ever be safe to release the "memory
> deallocation lock".  Just because it's safe for your thread doesn't mean
> it's safe for another thread.  And if you do it on a thread-by-thread
> basis, in the above example it might be safe from thread B's perspective
> to release its "memory deallocation lock", but as illustrated that can
> have an effect on thread A.
>
The deallocation lock could be a counter. Doomed objects would be 
collectable when it's zero.

> I appreciate the brainstorming but I'm not currently sanguine about this
> idea,
>