Mailman 3 March 2005 - NumPy-Discussion

Numeric3 CVS compiles now
by Travis Oliphant 28 Mar '05

28 Mar '05

To all who were waiting: I've finished adding the methods to the array object so that Numeric3 in CVS now compiles (at least for me on Linux). I will be away for at least a day so it is a good time to play... -Travis

4 8

Re: [Numpy-discussion] Re: Trying out Numeric3
by Scott Gilbert 28 Mar '05

28 Mar '05

--- Michiel Jan Laurens de Hoon <mdehoon(a)ims.u-tokyo.ac.jp> wrote: > >>> > >> Do 4 gigabyte 1D numerical python arrays occur in practice? > > Why? There needs to be a good reason to break compatibility. Who needs > this? > I (and others I work with) routinely deal with 1D datasets that are multiple gigabytes in length. Working with terabyte datasets is on my near horizon. For lots of reasons, I don't/can't use Python/Numeric for very much of this sort of thing, but it would be nice it I could. The "32 bits is enough for anyone" design has bitten me with lots of tools (not just Python). The Python core will fix it's int/intp problem eventually, I can't see why Numeric3 wouldn't avoid the problem now. As a concrete case that I'm sure has been done, consider memory mapped file arrays. 64 bit platforms can mmap huge files without using huge amounts of of real memory. I try to remain a lurker on this list, but since I've already broken my silence, let me add a few other notes and then I'll go back to being silent... I'll try to sort them by priority. Pickling performance is important to us at my work. We use pickling to pass data across Unix pipes, through shared memory, across sockets on Gig-E, etc... Typically we'll have a dictionary containing some metadata, and a few large chunks (1-100 MBytes would be common) of Numeric array data. We'd like to transfer 100s of these per second. Currently, we pickle the array into a string in memory, then pickle the string across the conduit (pipe or socket or shared memory). For some reason, pickling a Numeric array directly to the file object is slower than the two stage process... If the new Numeric3 didn't break too much compatibility with the original Numeric but pickled much faster, we'd probably be in a hurry to upgrade based on this feature alone. The new pickling protocol that allows a generator to be used to copy small chunks at a time instead of an entire binary string copy could potentially save the cost of duplicating a 100 MByte array into a 100 MByte string. The reason we use pickling like we do is to pass data between processes. Almost all of our work machines have multiple processors (typically 4). A lot of times, the multi-process design is cleaner and less buggy, but there are also times when we'd prefer to use multiple threads in a single process. It's unfortunate that the GIL prohibits too much real concurrency with multiple threads. It would be nice if the ufuncs and other numerical algorithms released the GIL when possible. I know the limitations of the Python buffer protocol add significant headache in this area, but it's something to think about. We have a wide group of smart engineering folks using Python/Numeric, but most of them are not computer scientists or software engineers. Meaning they spend all day writing software, but know just enough about programming to solve their problems, and almost none of them have any knowledge about the internals of Python or Numeric. Complicated rules about whether something returns a scalar-versus-array, or a copy-versus-view add frustration and hard to find bugs. This has been beaten up on this list quite a bit, and there is probably too much momentum behind the case by case strategy that is now in place, but please count my vote for always getting an array copy (copy on write) from subscripting unless you explicitly ask for a view, and always returning a rank-0 array instead of a scalar. I agree with the other guy who pointed out that arrays are mutable and that likewise, rank-0 arrays should be mutable. I know it's unlikely to happen, but it would also be nice to see the Python parser change slightly to treat a[] as a[()]. Then the mutability of rank-0 could fit elegantly with the rank-(n > 1) arrays. It's a syntax error now, so there wouldn't be a backwards compatibility issue. We commonly use data types that aren't in Numeric. The most prevalent example at my work is complex-short. It looks like I can wrap the new "Void" type to handle this to some extent. Will indexing (subscripting) a class derived from a Numeric3 array return the derived class? class Derived(Numeric3.ArrayType): pass d = Derived(shape=(200, 200, 2), typecode='s') if isinstance(d[0], Derived): print "This is what I mean" I don't really expect Numeric3 to add all of the possible oddball types, but I think it's important to remember that other types are out there (fixed point for DSP, mu-law for audio, 16 bit floats for graphics, IBMs decimal64 decimal128 types, double-double and quad-double for increased precision, quaternions of standard types, ....). It's one thing to treat these like "record arrays", it's another thing for them to have overloaded arithmetic operators. Since Numeric3 can't support every type under the sun, it would be nice if when the final version goes into the Python core that the C-API and Python library functions used "duck typing" so that other array implementations could work to whatever extent possible. In other words, it would be better if users were not required to derive from the Numeric3 type in order to create new kinds of arrays that can be used with sufficiently generic Numeric3 routines. Simply having the required attributes (shape, strides, itemsize, ...) of a Numeric3 array should be enough to be treated like a Numeric3 array. This last one is definitely pie-in-the-sky, but I thought I'd mention it. Since the 64 bit Alphas are expensive and pretty much on the way out of production, we've stepped back to 32 bit versions of x86/Linux. The Linux boxes are cheaper, faster, and smaller, but not 64 bit. It would be really great using Numeric to directly manipulate huge (greater than 2**32 byte length) files on a 32 bit platform. This would require a smarter paging scheme than simply mmapping the whole thing, and I don't think any of the Python Array packages has proposed a good solution for this... I realize it adds considerable complexity to switch from a single buffer object pointing to the entire block of data to having multiple buffers pinning down pieces of the data at a time, but the result would be pretty useful. I realize this is a lot of commentary from someone who doesn't contribute much of anything back to the Numeric/Numarray/SciPy community. If you got this far, thanks for your time reading it. I appreciate the work you're doing. Cheers, -Scott

7 6

Re: [Numpy-discussion] Re: Trying out Numeric3
by Scott Gilbert 26 Mar '05

26 Mar '05

--- Stephen Walton <stephen.walton(a)csun.edu> wrote: > Scott Gilbert wrote: > > >It's nice because it fits with the rest of the array semantics and > creates > >a consistant system: > > > > Array3D = zeros((1, 1, 1)) > > > > Array2D = Array3D[0] > > Array1D = Array2D[0] > > Array0D = Array1D[0] > > > > > Hmm...in both Numeric3 and numarray, the last line creates a Python > scalar. Array2D and Array1D by contrast are not only arrays, but they > are views of Array3D. I should have been clear that I wasn't describing what any of the array packages do today. The views thing is a different can of worms (I think they should be copy-on-write copies by default and views only when explicitly asked for). > Is what you're saying is that you want Array0D to > be a rank-0 array after the above? > Yes. I think it fits the pattern and is consistant. There are also cases where it is useful. Array operations form a nice little calculus. Returning non-mutable scalars in place of rank-0 arrays is like having (2 - 1 == 1) while (2 - 1 - 1 == a donut). 0 looks like a donut, but it's a different food group. :-) > > > Array0D[()] = 1 > > > Of course, this generates an error at present: "TypeError: object does > not support item assignment" since it is a Python int. Moreover, it > isn't a view, so that Array0D doesn't change after the assignment to > Array3D. Is this also slated to be changed/fixed using rank 0 arrays? > Would Array0D.shape be () in that case? > Array0D.shape would be an empty tuple () in that case. I can't say what either Numeric3 or Numarray will do. The last time I read the Numeric3 PEP, it looked like there were going to be several special types of scalars.

1 0

Re: [Numpy-discussion] Re: Trying out Numeric3
by Scott Gilbert 26 Mar '05

26 Mar '05

--- Stephen Walton <stephen.walton(a)csun.edu> wrote: > Travis Oliphant wrote: > > > Well, rank-0 arrays are and forever will be mutable. But, Python > > scalars (and the new Array-like Scalars) are not mutable. > > This is a really minor point, and only slightly relevant to the > discussion, and perhaps I'm just revealing my Python ignorance again, > but: what does it mean for a scalar to be mutable? I can understand > that one wants a[0]=7 to be allowed when a is a rank-0 array, and I also > understand that str[k]='b' where str is a string is not allowed because > strings are immutable. But if I type "b=7" followed by "b=3", do I > really care whether the 3 gets stuck in the same memory location > previously occupied by the 7 (mutable) or the symbol b points to a new > location containing a 3 (immutable)? What are some circumstances where > this might matter? > It's nice because it fits with the rest of the array semantics and creates a consistant system: Array3D = zeros((1, 1, 1)) Array2D = Array3D[0] Array1D = Array2D[0] Array0D = Array1D[0] That each is mutable is shown by: Array3D[0, 0, 0] = 1 Array2D[0, 0] = 1 Array1D[0] = 1 Array0D[] = 1 # whoops! Unfortunately that last one, while it follows the pattern, doesn't work for Python's parser so you're stuck with: Array0D[()] = 1 This becomes useful when you start writing generic routines that want to work with *any* dimensional arrays: zero_all_elements(ArrayND) Python's immutable scalar types could not change in this case. More complicated examples are more interesting, but a simple implementation of the above would be: def zero_all_elements(any): any[...] = 0 Cheers, -Scott

2 1

Re: [Numpy-discussion] Trying out Numeric3
by Scott Gilbert 25 Mar '05

25 Mar '05

--- Travis Oliphant <oliphant(a)ee.byu.edu> wrote: > How would somebody change the status of that > and re-open the PEP? I believe all it would take is a note to the python-dev mailing list by the new champion who was willing to implement and defend it. The text is public domain, so there's no copyright silliness if you need to make changes. I'm curious to see how this flies as this has always been one of their pet peave topics. Talking about buffer objects/protocols in general draws ire from some and dead silence from the rest. :-) > > Also, numarray has a memory object implemented that is a good start on > the implementation. So, this wouldn't be a huge job at this point. > The memory object is a very good start. I don't know if it tries to be usable when the GIL is released, or if it handles the slice semantics the same way. I think doing pickling really well is a non-trivial issue - at least if this object is going into the core of Python. Implementing the new pickling protocol is not terribly difficult, and any object can do it, but that only solves the space half of the problem. The new pickling protocol allows one to serialize large data without making one large copy of the binary data as a string, but one still has to make a lot of little copies of the data a piece at time. The multitude of little parts cost time allocating and memcpy-ing just to be written to a file and discarded. It would be great if the Python core libraries (cPickle) could be "taught" about the new type and serialize directly from the memory that is already there without creating any new string copies. > > > The "meta" attribute would be a small change. It's possible > > to do that with composition or inheritance instead, but that's > > really a just matter of taste. > > > I don't think I fully understand what you mean by "composition" > --- like a mixin class? or how inheritance solves the problem > on a C-API level? > > I'm mainly thinking of Extension modules that want to use each others' > memory on a C-level. That would be the main use of the meta information. > It would be a lot like putting a similar meta dictionary on the builtin "list" object. Many people wouldn't use it and would consider it a tiny wart just taking up space, while others would use it pretty differently from the way Numeric3 did and store completely different keys. The result would be that Numeric3 would have to check for the keys that it wanted in the meta dictionary. Since I think you're going to allow folks to pass in their own buffer objects to some of the array constructors (mmap for instance), the underlying Numeric3 code can't really assume that the "meta" attribute is there on all buffer objects. If you wanted to annotate all buffers that were passed inside of Numeric, something like the following would work with "memory", and "mmap" alike: # Composition of a memory buffer and meta data class NumericStorage(object): def __init__(self, buf, **meta): self.buf = buf self.meta = meta.copy() Of course at the C-Level it could just be a lightweight struct with two PyObject pointers. If you really wanted to add a meta attribute to the new generic memory object, you could do: # Inheritance to add metadata to a memory buffer class NumericBytes(memory): def __init__(self, *args, **kwds): memory.__init__(self, *args, **kwds) self.meta = {} It's a minor pain, but obviously inheritance like this can be done at the C level too... I don't know what particular meta data you plan to store with the buffer itself, and I'm going to resist the urge to guess. You probably have some very good use cases. What are you planning? If you have a list of meta keys that many if not all users would agree on, then it would be worth considering just building them efficiently into the proposed type and not wasting the overhead of a dictionary. That would also standardize their usage to some extent. As I said before, this is all just a matter of taste. I appologize for using so much text to try and explain what I meant. When all is said and done, I think whether the C API code is required to check for keys in the meta dictionary or attributes of the object itself, it's probably a pretty similar task. It would be PyDict_GetItem(...) versus PyObject_GetAttr(...). > > > When I wrote the PEP, I had high hopes of creating a > > Python only "ndarray" class out of bytes and the struct > > module > > Numarray essentially did this. I think we still need a C-type object > for arrays. > Yup. I understand and appreciate your attention to performance. For small arrays, it's tough to argue that a C implementation won't win. At the time, all I really needed was something to store and casually inspect/manipulate my odd ball data (large arrays of complex short) without converting to a larger representation. We have something very similar to weave.inline that I used when it came time to go fast. > > I read the PEP again, and agree with Scott that it > is quite good and would fit what we need quite well. > > I say let's resurrect it and push it forward. > Very cool. I hope it does what you need and makes it into the core. With your enthusiasm, I wish I had time to finish or at least help with the implementation. Unfortunately, I'm more swamped at work now than I was when I dropped the ball on this the first time. > > Scott, do you have any left-over code you could contribute? > I'll try and find what I had, but I probably don't have too much that you'll find much more valuable than the memory object from Numarray. I remember I went through a bit of pain to implement the "new style classes" correctly, but the pickling stuff in the core of the Python library is where the real challenge is, and I never got going on the TeX docs or unit tests that would be necessary for acceptance. Cheers, -Scott

4 7

Re: [Numpy-discussion] Trying out Numeric3
by Scott Gilbert 25 Mar '05

25 Mar '05

--- Travis Oliphant <oliphant(a)ee.byu.edu> wrote: > > - the PEP for a good "buffer" object (this has been called by others > a "byte" array which might be a good name. Essentially, it needs to be > a light-weight object around a chunk of memory -- i.e. a way to allocate > memory through Python. We would like to standardize on a set of meta > information that could be used to "understand" this memory as a numeric > array. Then, other objects which used this buffer as a memory block > would just have to expose the meta information in order to make seamless > the transfer of data from one application to another. We need to be > vocal about the value of the buffer object. This PEP is one way to do > that. There are some people who think buffer objects were a "bad > idea." This is primarily because of a fatal flaw in some objects that > both expose a memory pointer through the buffer protocol AND allow the > object's memory to be reallocated (using realloc) --- Numeric does not > do this. This problem could actually be easily fixed by a good > Python memory allocator that returns a simple memory object. If people > who wanted memory went through it's C-API (instead of using malloc and > realloc), much of the problems would be alleviated. This is what the > new "byte" object should be. I think it also wise to expect the "byte" > object to have an attribute called "meta" that would just be a > dictionary of "other information" you might want to pass to something > using the buffer protocol. > Hi Travis. I'm curious if you find PEP-296 sufficient: http://www.python.org/peps/pep-0296.html It is marked as "withdrawn by the author", but that is not really true. A more accurate statement would be "the author spent his allotted time defending and revising the PEP on the Python mailing list and was not left with sufficient time to finish the implementation on his corporate dollar". :-) It's a good PEP, and while my company uses Python quite extensively, after two weeks I had to get back to more direct goals... Regardless, I think PEP-296 meets your needs (and several other groups in the Python community), and it might save someone the time recreating a new PEP from scratch. More importantly, it might save someone some of the time required to defend and argue the PEP on the Python mailing list. When the discussion cleared, Guido was very positive toward the PEP - I just never got it implemented... The "meta" attribute would be a small change. It's possible to do that with composition or inheritance instead, but that's really a just matter of taste. When I wrote the PEP, I had high hopes of creating a Python only "ndarray" class out of bytes and the struct module, so it was definitely targeted at needs similar to what I believe yours to be. Obviously you should do what is best for you, but I would be pleased if my wasted effort was revived and completed to actually be useful. Cheers, -Scott

2 2

Re: Trying out Numeric3
by Michiel Jan Laurens de Hoon 23 Mar '05

23 Mar '05

Travis Oliphant wrote: > Michiel Jan Laurens de Hoon wrote: >> Another warning was that PyArrayObject's "dimensions" doesn't seem to >> be an int array any more. > > Yes. To allow for dimensions that are bigger than 32-bits, dimensions > and strides are (intp *). intp is a signed integer with sizeof(intp) == > sizeof(void *). On 32-bit systems, the warning will not cause > problems. We could worry about fixing it by typedefing intp to int > (instead of the current long for 32-bit systems). > Do 4 gigabyte 1D numerical python arrays occur in practice? If I understand correctly, the current implementation gives dimensions a different pointer type on different platforms. This will break extension modules on platforms other than 32-bits, as the extension module expects dimensions to be a pointer to int. --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon

6 10

Current state of performance?
by Russell E. Owen 23 Mar '05

23 Mar '05

I'm curious as to the current state of numarray vs. Numeric performance. My code is a mix at the moment: - Numeric: coordinate conversion code that was written before numarray was very solid and makes heavy use of small matrices. - numarray: some image processing stuff that uses PyFits (which uses numarray). I'd like to settle on one package. At one time numarray was at a clear disadvantage for small arrays, but was wondering if that was still true. Any advice? -- Russell

2 1

Numeric3 compilation errors on OS X
by John Gleeson 21 Mar '05

21 Mar '05

I get the following errors when building the extensions for Numeric3 on OS X 10.3.8 (with PatherPythonFix installed): Src/arrayobject.c:2538: error: conflicting types for `_swap_axes' Src/arrayobject.c:1170: error: previous declaration of `_swap_axes' This is for arrayobject.c v. 1.61. Any ideas? thanks, John

2 1

Re: [Numpy-discussion] casting in numarray
by Joachim Boomberschloss 21 Mar '05

21 Mar '05

Thanks, that's exactly what I needed! --- Thomas Grill <gr(a)grrrr.org> wrote: > Hi Joachim, > this is what i do in my Python extension of the Pure > Data realtime > modular system. You have to create a Python buffer > object pointing to > your memory location and then create a numarray from > that. It's quite easy. > See the code in > http://cvs.sourceforge.net/viewcvs.py/pure-data/externals/grill/py/source/ > files pybuffer.h and pybuffer.cpp > > best greetings, > Thomas > > Joachim Boomberschloss schrieb: > > >Hi, > > > >I'm using numarray for an audio-related application > as > >a buffer in an audio-processing pipeline. I would > like > >to be able to allocate the buffer in advance and > later > >regard it as a buffer of 8bit or 16bit samples as > >appropriate, but in numarray, casting always > produces > >a new array, which I don't want. How difficult > should > >it be to make it possible to create an array using > an > >exsisting pre-allocated buffer to act as an > interface > >to that buffer? Also, if others consider it useful, > is > >there anyone willing to guide me through the code > in > >doing so? > > > >Thanks, > > > >Joe > > > > > > > >__________________________________ > >Do you Yahoo!? > >Yahoo! Small Business - Try our new resources site! > >http://smallbusiness.yahoo.com/resources/ > > > > > >------------------------------------------------------- > >SF email is sponsored by - The IT Product Guide > >Read honest & candid reviews on hundreds of IT > Products from real users. > >Discover which products truly live up to the hype. > Start reading now. > >http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click > >_______________________________________________ > >Numpy-discussion mailing list > >Numpy-discussion(a)lists.sourceforge.net > >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > > > -- > --->----->->----->-- > Thomas Grill > gr(a)grrrr.org > +43 699 19715543 > > __________________________________ Do you Yahoo!? Yahoo! Small Business - Try our new resources site! http://smallbusiness.yahoo.com/resources/

1 0