Many problems are best solved with irregular array structures. These are aggregations not having a rectangular shape. To motivate, here's one example, http://lambda-the-ultimate.org/files/HammingNumbersDeclarative.7z - from http://lambda-the-ultimate.org/node/608#comment-5746 Irregularity here changes an O(N^3) solution to O(N). (The file format is a 7zip archive with a MathReader file inside, readable in Windows or Unix with free software.) These cases also arise in simulations where physical geometry determines array shape. Here memory consumption is the minimization goal that makes irregularity desirable. The access function will return NaN or zero for out-of-bounds requests. There is no need to consume memory storing NaNs and zeros. Please advise how much support numpy/Scipy has for these structures, if any, including future plans. If support exists, could you kindly supply a Scipy declaration matching the first example. Thank you very much.
rw679aq02@sneakemail.com wrote:
Many problems are best solved with irregular array structures. These are aggregations not having a rectangular shape. To motivate, here's one example,
http://lambda-the-ultimate.org/files/HammingNumbersDeclarative.7z - from http://lambda-the-ultimate.org/node/608#comment-5746
Irregularity here changes an O(N^3) solution to O(N). (The file format is a 7zip archive with a MathReader file inside, readable in Windows or Unix with free software.)
These cases also arise in simulations where physical geometry determines array shape. Here memory consumption is the minimization goal that makes irregularity desirable. The access function will return NaN or zero for out-of-bounds requests. There is no need to consume memory storing NaNs and zeros
Please advise how much support numpy/Scipy has for these structures, if any, including future plans. If support exists, could you kindly supply a Scipy declaration matching the first example.
SciPy has sparse matrix support (scipy.sparse) with several storage formats You can also construct irregular arrays using arrays of objects or just lists of lists. -Travis
Travis, A sparse matrix is a different animal serving a different purpose, i.e., solution of linear systems. Those storage formats are geared for that application: upper diagonal, block diagonal, stripwise, etc. To be more specific: here tight numerical arrays are presumably discussed. Python and other languages could define an "irregular list of irregular lists" or "aggregation of objects" configuration. Probably Lisp would be better for that. But it is not my driving interest. My interest is packed storage minimizing memory consumption and access time, with bonus points for integration with numerical recipes and element-wise operations. Again, actual demonstration would be appreciated. I selected an example with minimal deviation from a regular array to simplify things. The shape is essentially a cube with a planar cut across one corner. The Mathematica code shows it is very easy to define in that language. (I am not sure whether it is tightly packed but it shows O(N) performance graphs.)
Forgive me if I missed any replies. Since I have seen none, I will rephrase the request. Please demonstrate an irregular array in numpy with time complexity measurement. The shape does not matter so long as it is non-rectangular and includes a complexity measurement. A sparse matrix is conceptually rectangular, so it does not fit the request at all. The question is whether numpy has such support; if not, is it planned. The question is not about the general-purpose Python language. (Although Python demonstrations are of interest if they include the time complexity measure from first principles or experiment.)
rw679aq02@sneakemail.com wrote:
Forgive me if I missed any replies. Since I have seen none, I will rephrase the request.
Please demonstrate an irregular array in numpy with time complexity measurement. The shape does not matter so long as it is non-rectangular and includes a complexity measurement. A sparse matrix is conceptually rectangular, so it does not fit the request at all.
The question is whether numpy has such support; if not, is it planned.
No, and no. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
The question is whether numpy has such support; if not, is it planned.
No, and no.
Thank you for answering, and I am sorry to hear that. I will be dropping my membership on the scipy-numpy email list shortly. Many systems handle rectangular arrays quite well already, and are more fully developed. It is a common fallacy rectangles are the only shapes one ever needs. Physical geometry is only rarely rectangular, and solution of actual physical problems, and even number-theoretical problem, is a far larger problem domain.
On 9/4/06, rw679aq02@sneakemail.com <rw679aq02@sneakemail.com> wrote:
The question is whether numpy has such support; if not, is it planned.
No, and no.
Thank you for answering, and I am sorry to hear that.
I will be dropping my membership on the scipy-numpy email list shortly. Many systems handle rectangular arrays quite well already, and are more fully developed.
It is a common fallacy rectangles are the only shapes one ever needs. Physical geometry is only rarely rectangular, and solution of actual physical problems, and even number-theoretical problem, is a far larger problem domain.
Thanks for blessing us with your exalted presence and superior intellect. You will be missed.
On Mon, 4 Sep 2006 17:35:04 -0600 "Charles R Harris" <charlesr.harris@gmail.com> wrote:
On 9/4/06, rw679aq02@sneakemail.com <rw679aq02@sneakemail.com> wrote:
The question is whether numpy has such support; if not, is it planned.
No, and no.
Thank you for answering, and I am sorry to hear that.
I will be dropping my membership on the scipy-numpy email list shortly. Many systems handle rectangular arrays quite well already, and are more fully developed.
It is a common fallacy rectangles are the only shapes one ever needs. Physical geometry is only rarely rectangular, and solution of actual physical problems, and even number-theoretical problem, is a far larger problem domain.
Thanks for blessing us with your exalted presence and superior intellect. You will be missed.
Well now, that's just snarky. rw679aq02 (if that is indeed your real name!), the reason that numpy will not support irregular "arrays" anytime soon comes down to multiple reasons: 1) It would require a complete rework; better to make a new package. Irregular arrays would require an entirely different approach than regular arrays. 2) While most things are not rectangular, the equations that describe them are, for the most part. Finite-element methods, for instance, use a triangulation of the physical object, and the equations can then be cast as very large set of array equations. 3) I would guess that problems that could be described by irregular arrays could be better recast with a different data structure. There's a saying that you can write Fortran in any language; however, that doesn't mean you should! 4) No one else has asked for them, and the developers don't need them (this is how open source works: scratching one's own itch) 5) If, instead, you're looking for efficient memory representations of of sparse matrices (most elements being 0), then there are various ways to do this. I'm not familiar with them, but scipy has packages to handle sparse matrices. A lot of work happens in this field (those finite-element methods tend to make sparse matrices). Note that you could simulate an irregular array (with a maximum size in the dimensions) using the masked arrays provided by NumPy. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
Exalted presences and superior intellects aside, the point is not hard to get: Motivational examples are everywhere. Think about gridding physical problems expressed in cylindrical or spherical coordinates. The natural slices are not rectangles. You can use rectangular storage but only with O(n^3) waste. More abstract solution spaces of math and physics do not usually lend themselves to rectangular treatments. (I understand finite element techniques and am not referring to those.) Again, rectangular storage is possible only with O(n^d) waste, where commonly d>3. Granted one may overcome these issues with software development effort; that insight begs the question. I am looking for teaching software that already does so. I agree that rectangular storage is easiest for software programmers and hence common. It is not easiest for solving all problems. Students should explore solutiuon spaces in a proper setting. So I just asked what numpy could do in this regard. Now I have the plain answer, and am grateful for it.
Think about gridding physical problems expressed in cylindrical or spherical coordinates. The natural slices are not rectangles. You can use rectangular storage but only with O(n^3) waste.
I don't get this argument. Are you slicing your spherical coordinates with a cartesian coordinate system? That's surely a waste. Many times bases can be chosen so that you even in an array doesn't waste space. I can't see the point attacking numpy with this "irregular" generalisation. Numpy is a tool very good for it's purpose, we don't really want to trash it with features for solving unspecified general problems. Instead, specify a real world problem, and perhaps someone might have a good tip, otherwise look for other tools. Pytables for example can efficiently store varying length arrays. Maybe start on developing an variable length array yourself that integrates perfectly with Numpy where applicable? //Torgil On 9/5/06, rw679aq02@sneakemail.com <rw679aq02@sneakemail.com> wrote:
Exalted presences and superior intellects aside, the point is not hard to get: Motivational examples are everywhere.
Think about gridding physical problems expressed in cylindrical or spherical coordinates. The natural slices are not rectangles. You can use rectangular storage but only with O(n^3) waste.
More abstract solution spaces of math and physics do not usually lend themselves to rectangular treatments. (I understand finite element techniques and am not referring to those.) Again, rectangular storage is possible only with O(n^d) waste, where commonly d>3.
Granted one may overcome these issues with software development effort; that insight begs the question. I am looking for teaching software that already does so.
I agree that rectangular storage is easiest for software programmers and hence common. It is not easiest for solving all problems. Students should explore solutiuon spaces in a proper setting. So I just asked what numpy could do in this regard. Now I have the plain answer, and am grateful for it.
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
participants (6)
-
Charles R Harris -
David M. Cooke -
Robert Kern -
rw679aq02＠sneakemail.com -
Torgil Svensson -
Travis Oliphant