[Numpy-discussion] Re: Bytes Object and Metadata

[Perry Greenfield]

On Mar 28, 2005, at 4:30 AM, Travis Oliphant wrote:

> Scott,
>
> Thank you for your detailed explanations.  This is starting to make 
> more sense to me.  It is obvious that you understand what we are 
> trying to do, and I pretty much agree with you in how you think it 
> should be done.  I think you do a great job of explaining things.
> I agree we should come up with a set of names for the interface to 
> arrayobjects.  I'm even convinced that offset should be an optional 
> part of the interface (implied 0 if it's not there).
>

Just to add my two cents, I don't think I ever thought it was necessary 
to bundle the metadata with the memory object for the reasons Scott 
outlined. It isn't needed functionally, and there are cases where the 
same memory may be used in different contexts (as is done with our 
record arrays).

Numarray, when it uses the buffer object, always gets a fresh pointer 
for the buffer object for every data access. But Scott is right that 
that pointer is good so long as there isn't a chance for something else 
to change it. In practice, I don't think that ever happens with the 
buffers that numarray happens to use, but it's still a flaw of the 
current buffer object that there is no way to ensure it won't change.

I'm not sure how the support for large data sets should be handled. I 
generally think that it will be very awkward to handle these until 
Python does as well. Speaking of which...

I had been in occasional contact with Martin von Loewis about his work 
to update Python to handle 64-bit addressing. We weren't planning to 
handle this in nummarray (nor Numeric3, right Travis or do I have that 
wrong?) until Python did. A few months ago Martin said he was mostly 
done. I had a chance to talk to him at Pycon about where that work 
stood. Unfortunately, it is not turning out to be as easy as he hoped. 
This is too bad. I have a feeling that this work is going to stall 
without help on our (numpy community) part to help make the changes or 
drum beating to make it a higher priority. At the moment the Numeric3 
effort should be the most important focus, but I think that after that, 
this should become a high priority.

Perry