Re: [Numpy-discussion] Trying out Numeric3

--- Travis Oliphant <oliphant@ee.byu.edu> wrote:
- the PEP for a good "buffer" object (this has been called by others a "byte" array which might be a good name. Essentially, it needs to be a light-weight object around a chunk of memory -- i.e. a way to allocate memory through Python. We would like to standardize on a set of meta information that could be used to "understand" this memory as a numeric array. Then, other objects which used this buffer as a memory block would just have to expose the meta information in order to make seamless the transfer of data from one application to another. We need to be vocal about the value of the buffer object. This PEP is one way to do that. There are some people who think buffer objects were a "bad idea." This is primarily because of a fatal flaw in some objects that both expose a memory pointer through the buffer protocol AND allow the object's memory to be reallocated (using realloc) --- Numeric does not do this. This problem could actually be easily fixed by a good Python memory allocator that returns a simple memory object. If people who wanted memory went through it's C-API (instead of using malloc and realloc), much of the problems would be alleviated. This is what the new "byte" object should be. I think it also wise to expect the "byte" object to have an attribute called "meta" that would just be a dictionary of "other information" you might want to pass to something using the buffer protocol.
Hi Travis. I'm curious if you find PEP-296 sufficient: http://www.python.org/peps/pep-0296.html It is marked as "withdrawn by the author", but that is not really true. A more accurate statement would be "the author spent his allotted time defending and revising the PEP on the Python mailing list and was not left with sufficient time to finish the implementation on his corporate dollar". :-) It's a good PEP, and while my company uses Python quite extensively, after two weeks I had to get back to more direct goals... Regardless, I think PEP-296 meets your needs (and several other groups in the Python community), and it might save someone the time recreating a new PEP from scratch. More importantly, it might save someone some of the time required to defend and argue the PEP on the Python mailing list. When the discussion cleared, Guido was very positive toward the PEP - I just never got it implemented... The "meta" attribute would be a small change. It's possible to do that with composition or inheritance instead, but that's really a just matter of taste. When I wrote the PEP, I had high hopes of creating a Python only "ndarray" class out of bytes and the struct module, so it was definitely targeted at needs similar to what I believe yours to be. Obviously you should do what is best for you, but I would be pleased if my wasted effort was revived and completed to actually be useful. Cheers, -Scott

Hi Travis.
I'm curious if you find PEP-296 sufficient:
http://www.python.org/peps/pep-0296.html
It is marked as "withdrawn by the author", but that is not really true. A more accurate statement would be "the author spent his allotted time defending and revising the PEP on the Python mailing list and was not left with sufficient time to finish the implementation on his corporate dollar". :-) It's a good PEP, and while my company uses Python quite extensively, after two weeks I had to get back to more direct goals...
Great to hear from you Scott. Yes, I looked at this PEP (though I haven't studied it sufficiently to say if it's perfect for our needs or not), but it is very close. I did not know what "withdrawn by author" meant, thanks for clarifying. How would somebody change the status of that and re-open the PEP? I think it is a great place to start. Also, numarray has a memory object implemented that is a good start on the implementation. So, this wouldn't be a huge job at this point.
Regardless, I think PEP-296 meets your needs (and several other groups in the Python community), and it might save someone the time recreating a new PEP from scratch. More importantly, it might save someone some of the time required to defend and argue the PEP on the Python mailing list. When the discussion cleared, Guido was very positive toward the PEP - I just never got it implemented...
Good to hear.
The "meta" attribute would be a small change. It's possible to do that with composition or inheritance instead, but that's really a just matter of taste.
I don't think I fully understand what you mean by "composition" --- like a mixin class? or how inheritance solves the problem on a C-API level? I'm mainly thinking of Extension modules that want to use each others' memory on a C-level. That would be the main use of the meta information.
When I wrote the PEP, I had high hopes of creating a Python only "ndarray" class out of bytes and the struct module, so it was definitely targeted at needs similar to what I believe yours to be. Obviously you should do what is best for you, but I would be pleased if my wasted effort was revived and completed to actually be useful.
Numarray essentially did this. I think we still need a C-type object for arrays. But, it's great to hear you still believe in the byte object. I wasn't sure. -Travis

Scott Gilbert wrote:
Hi Travis.
I'm curious if you find PEP-296 sufficient:
http://www.python.org/peps/pep-0296.html
It is marked as "withdrawn by the author", but that is not really true. A more accurate statement would be "the author spent his allotted time defending and revising the PEP on the Python mailing list and was not left with sufficient time to finish the implementation on his corporate dollar". :-) It's a good PEP, and while my company uses Python quite extensively, after two weeks I had to get back to more direct goals...
I read the PEP again, and agree with Scott that it is quite good and would fit what we need quite well. I say let's resurrect it and push it forward. Scott, do you have any left-over code you could contribute? -Travis
participants (2)
-
Scott Gilbert
-
Travis Oliphant