[Python-3000] patch: bytes object PyBUF_LOCKDATA read-only and immutable support

[Guido van Rossum]

On 9/11/07, Travis E. Oliphant <oliphant at enthought.com> wrote:
> I'm not sure I understand the difference between a classic read lock and
> the exclusive write lock concept.   Does the classic read-lock just
> prevent writing to the memory area.  In my mind that is a read-only
> memory buffer and the buffer interface would complain if a writeable
> buffer was requested.

There are different notions of reading and writing.  Sometimes an
object it naturally read-only (e.g. a PyString). In that case
requesting SIMPLE access should pass but requesting WRITABLE or
LOCKDATA access should fail. (I think the other flags are orthogonal
to these, right?). Any number of concurrent SIMPLE accesses can
coexist since the clients promise they will only read.

OTOH suppose we have an object that is naturally writable (e.g. e
PyBytes). I understood that in this case any number of SIMPLE or
WRITABLE requests would be allowed to be outstanding simultaneously,
and any of these would simply prevent the buffer from moving (fixing
the object's size). But this doesn't sound like it is how you meant it
-- you seem to say that once any SIMPLE (readonly) requests are
outstanding, WRITABLE requests should fail. And I suppose that only
one WRITABLE request ought to be allowed at a time. But then I don't
know what the difference between WRITABLE and LOCKDATA would be.

I guess I would be inclined to propose separate flags for indicating
the operation that the caller will attempt (read or write) and the
level of locking (lock the buffer's address or also prevent anyone
else from writing). Then a "classic read lock" would request read
access while locking out writers (bsddb would use this); a "classic
write lock" would request write access while locking out writers (your
scratch area example would use this); others who don't really care if
the data changes underneath them as long as it doesn't move (e.g.
traditional I/O) could request read access without locking. I'm not
sure if there's a use case to be made for write access without
locking, but I wouldn't rule it out -- possibly when two threads share
a memory area they might have their own protocol for locking it and
might just both want to be able to write to (parts of) it.

What do you think? Another way to look at this would be to consider
these 4 cases:

basic read access (I can read, others can read or write)
locked read access (I can read, others can only read)
basic write access (I can read and write, others can read or write)
exclusive write access (I can read and write, no others can read or write)

Except that accessing the object from Python (e.g. iteration or
indexing) never gets locked out. (Or perhaps it should be? That can
also be done.)

Also, it remains to be seen whether basic read access should be
granted when someone has exclusive write access (see below).

> Actually, writeable is an accepted variant of 'writable' (but it doesn't
> show up in many spell-check dictionaries).  No, it is not too late to
> change it.  Or just define WRITEABLE as WRITABLE.   NumPy uses
> "WRITEABLE" simply because I like that spelling better.

Google found 1.4M occurrences of writeable vs. 3.9M occurrences of
writable. I guess you represent a strong minority. :-) I'd still like
to see it changed. We can leave WRITEABLE as an alias for WRITABLE for
those who are used to seeing it that way in NumPy.

> I'm anxious for feedback and help with the locking mechanism, because I
> do not have all use cases in mind.  I have never thought about a lock
> that prevents reading.  In my mind, this would be handled by the object
> itself.  It could refuse buffer requests if it's data had been locked or
> it could not.

Well, the scratch area scenario you describe makes it iffy to read
anything out of the original object since you wouldn't know whether
you were reading before, during or after the write back from the
scratch area to the object's buffer. The question is, do we really
care. If we adopted my 4 access modes above, we could say that basic
read access will still be granted when someone has exclusive write
access if we don't care, OR we could say that basic reads are locked
out by exclusive write access. (And then there's the separate issue of
whether python-level access counts as basic read access or doesn't
count at all -- though the moer I think about it, I think it should be
treated the smne as basic read access.)

> On the other hand, there could be two concepts of locking that a
> consumer could request from an object
>
> 1) Lock so that no other reads or writes are possible until the lock is
> released.
> 2) Lock so that only reads are possible.
>
> I had only thought of #2 for the current buffer interface.

#1 maps to locked read OR exclusive write access in the strict variant.
#2 maps to locked read in my scheme.

(Gotta go -- ttyl.)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)