[Python-3000] Py3k-buffer branch merged back to py3k branch

[Travis E. Oliphant]

Neal Norwitz wrote:
> On 8/18/07, Travis E. Oliphant <oliphant.travis at ieee.org> wrote:
>> In preparation for the sprints, I have converted all Python objects to
>> use the new buffer protocol PEP and implemented most of the C-API.  This
>> work took place in the py3k-buffer branch which now passes all the tests
>> that py3k does.
>>
>> So, I merged the changes back to the py3k branch in hopes that others
>> can continue working on what I've done.  The merge took place after
>> fully syncing the py3k-buffer branch with the current trunk.
>>
>> Left to do:
>>
>> 1) Finish the MemoryViewObject (getitem/setitem needs work).
>> 2) Finish the struct module changes (I've started, but have not checked
>>         the changes in).
>> 3) Add tests
> 
> Also need to add doc.  I noticed not all the new APIs mentioned the
> meaning of the return value.  Do all the new functions which return
> int only return 0 on success and -1 on failure.  Or do any return a
> size.  I'm thinking of possible issues with Py_ssize_t vs int
> mismatches.  I saw a couple which might have been a problem.  See
> below.

Yes,  IIRC that is correct.

> 
>> Possible problems:
>>
>> It seems that whenever a PyExc_BufferError is raised, problems (like
>> segfaults) occur.  I tried to add a new error object by copying how
>> Python did it for other errors, but it's likely that I didn't do it right.
> 
> I think I fixed this.  Needed to add PRE_INIT and POST_INIT for the
> new exception.  This fixed the problem reported by Christian Heimes in
> this thread.

Thanks very much.

> 
> I checked in revision 57193 which was a code review.  I pointed out
> all the places I thought there were problems.  Since some of this code
> is tricky, I expect there will be more issues.  This code really,
> really needs tests.

If Chris (the guy who will be at the sprint) does not write tests, I 
will, but it will probably be after about Aug. 27.

> 
> I added a comment about a memory leak.  Below is the stack trace of
> where the memory was allocated.  I added a comment (in release buffer)
> where I thought it could be freed, but I'm not sure that's the right
> place.

There should be no memory to free there.  The get and release buffer 
mechanism doesn't allocate or free any memory (there was a hack in 
arrayobject which I just fixed).

Now, perhaps there are some reference counting issues, but the mechanism 
doesn't really play with reference counts either.  I will be around 
after August 27th to test the code more (it will help to finish 
implementing the MemoryView Object -- i.e. get its tolist function 
working, and so forth).

> 
> When I ran the test suite test_xmlrpc failed.  I'm not sure if this
> was from your checkin, my checkin, or something else.
> 

This was definitely happening prior to my checking in.

> n
> --
> Memory leaked when allocated from:
>  array_buffer_getbuf (arraymodule.c:1775)
>  buffer_getbuf (bufferobject.c:28)
>  bytes_init (bytesobject.c:807)
>  type_call (typeobject.c:429)

Hmm.  I'm not sure what memory is being leaked unless there are 
reference counting issues I'm not seeing.

In bytes_init for example, that line number is a static memory 
allocation?  How is static memory being leaked?

The arraymodule.c malloc call should be gone now as the possible strings 
needed are now in the source code itself.