From amullhau@ix.netcom.com Tue Jul 1 00:13:11 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Mon, 30 Jun 1997 19:13:11 -0400
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199706301443.KAA23238@dante.mh.lucent.com>
Message-ID: <3.0.1.32.19970630191311.006d258c@popd.netcruiser>
At 10:45 AM 6/30/97 -0400, Aaron Watters wrote:
>
>Take should have an inverse, and in my case I am fiddling during
>spare moments on a win95 box, and I don't have time right now
>to learn how to build other peoples stuff there. I suspect old S
>hackers might run into something like this in NumPy and say "huh?"
>and go back to S.
Not at all. take is taken from APL, and most S programmers of any
stature at all are fluent in APL. The reason for this is that S is
sort of an 'unconscious' APL derivative - and there is a lot you can
learn from APL. Recently, S has become much more widely used, and
so you _will_ find a lot of newbie S users running around who don't
know much about APL - but that's OK, most of them will not be coming
across Python yet, either.
S is a very demanding language to understand since it is such a wide
sprawling mass of all sorts of stuff. It's a little like PL/I in that
sense but S was never really designed, and so it turns out to be more
or less a superset of APL, Lisp, and pretty much Numerical Python, too.
Scientifically oriented interpreters have been around for quite a while
now - the first generation was APL in 1963. S started in the late 60's,
and this is well to keep in mind - there is a _ton_ of experience in
these communities, especially with crystalized rigid language design as
in APL and the amoebic evolution of S.
What most programmers literate in scientific interpreters will do when
they come to Numerical Python is:
1. Nice fast second generation interpreter, fine object orientation,
good extensibility, decent embeddability. Object orientation OK. As
usual, TK is integrated.
2. Graphics not tightly integrated - Yet.
3. Pressure of users will probably revise unfortunate limitations on indexing,
eliminating the need for some of the functions - (take is an example).
4. Hope they don't fall for the GC nonsense. (Been there, done that with
APL and S). Doubt they understand VM yet, but maybe the Linux community
will spread that gospel.
5. Threading is an unresolved issue - but it doesn't seem to get in the way
of using multithreaded libraries (e.g. Sun Performance Library, etc.).
6. Default argument processing is consistent, interesting, but also annoying.
However, small idioms suffice to produce the desired behaviors, so no
problem here.
7. Event processing probably will turn out OK, but a custom main loop for
fast response time is probably necessary. Looks like some people already
are working on this.
8. Everything is a reference - and the reference counting is done well.
This helps reduce memory waste of large objects by increasing it for
small objects. Probably a loser for memory bandwidth, but that can be
treated easily for large objects.
9. Installation is peculiar, but this is easily remedied, too.
10. IT'S FREE, COMES WITH SOURCE, AND _NO LICENSE MANAGER_!
Give you a hint, Aaron. Numerical Python is almost identical to about 90%
of the S programming language, is frequently tens or hundreds of times faster
than the equivalent S code, costs nothing, and your sysadmin might even
like it. Sure. Lots of S programmers will just walk away. But don't hold
your breath.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hyoon@nyptsrv1.etsd.ml.com Tue Jul 1 00:17:10 1997
From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant)
Date: Mon, 30 Jun 1997 19:17:10 -0400
Subject: [MATRIX-SIG] Newbie Q.
Message-ID: <199706302317.TAA22517@ptq3.etsd>
Hello,
I am new to NumPy and there has to be a easy way to do this than I can think of.
>>> a
array([1, 2, 3])
>>> b
array([2, 4, 6])
>>> c
array([[1, 2, 3],
[2, 4, 6]])
>>> concatenate((c,a)) # Here's my problem they are different row shape.
Traceback (innermost last):
File "<stdin>", line 1, in ?
File "/opt/gnu/lib/python1.4/NumPy/Numeric.py", line 58, in concatenate
return multiarray.concatenate(a)
ValueError: arrays must have same number of dimensions
I have attempting to append (vertical concatenation) a row of 1xn to mxn
Numerical arrary, but failing miserably since they are not the same dimention.
I can use loop or flatten the mxn out and concatenate vertically and to reshape,
but that sounds extremely silly. In my past in Gauss:
d = c | a
would have done the job.
I must be doing something wrong, but can't figure out form the Tutorial.
In addition, in Gauss
Horizontal would be e = a~b
Transposing c would have been
t = c'
dot product would be
d = a .* b'
This to me faster than writing transpose(c) or concatenate((a,b)1), etc...
Is there shorter method in Python? I guess I could write a emacs key. Does
anyone have one already?
While I am at it.
Does PyGist do Bar, pie, and combo graphs? Is there a good site with examples?
Plus, does X-axis always have to be Numeric? Should I attempt to use something
else for these simple graphs?
Any help with above is greatly appreciated!
Thank you much,
Hoon,
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Tue Jul 1 01:31:04 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Mon, 30 Jun 1997 20:31:04 -0400
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199706301843.OAA03287@dante.mh.lucent.com>
Message-ID: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser>
At 02:45 PM 6/30/97 -0400, Aaron Watters wrote:
> I propose that arraytype[arraytype] automatically be interpreted
>as (1) gather as a value (2) scatter as an l-value. I apologize if
>this is obviously dumb, incompatible with existing usage, or
>already proposed (I'm new, remember, don't hurt me!) :).
This proposal is OK with me and is essentially what a lot of people
wish Python did - and not just the numerical arrays. (No matter though-
the numerical arrays are probably general enough).
However, there are some variations that make sense and ought to be considered.
Can you index a multi-dimensional array with a one-dimensional array? In
other words does arraytype[range(10)] refer to the first 10 elements of
the array? I think this would be doable using other more verbose methods,
but this is a simple way to spell this.
Can you index a one-dimensional array with a multi-dimensional array? In
languages which provide this, this usually returns a value with the same
shape as the matrix and the values corresponding to the value in the vector
with that index. This is really nice for image processing - for example
palette remapping. You can get this by applying a function, but that is
usually a lot slower.
What about in between? I don't know of a language that permits these kinds
of shenanigins so I assume we can drop this possibility - undoubtedly now
someone will point out some language where this is a powerful tool...
If array0 is d-dimensional and arrayk is 1-dimensional for 1<=k<=d, does
array0[array1, ..., arrayd] refer to the elements corresponding to the
Cartesian product of array1 ... arrayd? It should. What is the shape
of the result when some of the index arrays have length 1? Should there be
an additional argument to control this? (In S, x[1,1:5] is 1-dimensional with
length 5 and x[1,1:5,drop=F] is 2-dimensional with shape (1, 5). - The reason
you probably want this is that it can be really annoying to reshape the
result which has been collapsed if it was an intermediate expression.)
There are some other approaches to multidimensional indexing and it does
not do to pick one without some consideration of the edge cases, etc..
Presumably the way to go is to make it possible to play with versions of this
so a concensus can emerge. This looks quite feasible in Python.
By the way, once you have powerful indexing, you will not need all the
functions you used to; but there are at least two which will emerge as
really useful now although only marginally useful previously.
1. A function which returns the permutation which would put the elements of
a 1-d array in order. In APL this is called 'upgrade' and in S it is called
'order'. A quick illustration of the use of this is computing the ranks
of elements in a vector - I'll suppose we call the function order.
def rank(x) :
return order(order(x))
2. A function which returns the indices into a table where given keys are
located. In APL this is called 'dyadic iota' and in S it is called 'match'.
Strictly speaking you can implement this in terms of the order function
described above - an O((n+m)log(n+m)) solution for searching m keys in a
table of length n - but usually dyadic iota is implemented in terms of
hashing which is much faster. Note that you want to locate stuff sprinkled
around in arrays with this - it isn't clear to me at this point that Python
dicts do/do not provide an efficient, perhaps more pythonistic alternative.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Tue Jul 1 01:46:02 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Mon, 30 Jun 1997 20:46:02 -0400
Subject: [MATRIX-SIG] Newbie Q.
In-Reply-To: <199706302317.TAA22517@ptq3.etsd>
Message-ID: <3.0.1.32.19970630204602.006d8dec@popd.netcruiser>
At 07:17 PM 6/30/97 -0400, Hoon Yoon - IPT Quant wrote:
>Hello,
> I am new to NumPy and there has to be a easy way to do this than I can
think of.
>>>> a
>array([1, 2, 3])
>>>> b
>array([2, 4, 6])
>>>> c
>array([[1, 2, 3],
> [2, 4, 6]])
>>>> concatenate((c,a))
This works for me:
concatenate([c, reshape(a, [-1, len(a)])])
The point is that the a that you have has only one dimension. Some
languages will handle this for you, but NumPy doesn't.
>In addition, in Gauss
>Horizontal would be e = a~b
>Transposing c would have been
>t = c'
>dot product would be
>d = a .* b'
>This to me faster than writing transpose(c) or concatenate((a,b)1), etc...
>Is there shorter method in Python? I guess I could write a emacs key. Does
>anyone have one already?
A lot of different scientific interpreters are around. In Matlab, S, APL,
Maple, etc. all these functions have different spellings, and they all
look like nonsense to people coming from the other languages - quick -
what would %o% mean?
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Tue Jul 1 09:25:02 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 1 Jul 1997 10:25:02 +0200
Subject: [MATRIX-SIG] Newbie Q.
In-Reply-To: <199706302317.TAA22517@ptq3.etsd> (hyoon@nyptsrv1.etsd.ml.com)
Message-ID: <199707010825.KAA31356@lmspc1.ibs.fr>
> I am new to NumPy and there has to be a easy way to do this than I can think of.
> >>> a
> array([1, 2, 3])
> >>> b
> array([2, 4, 6])
> >>> c
> array([[1, 2, 3],
> [2, 4, 6]])
> >>> concatenate((c,a)) # Here's my problem they are different row shape.
> Traceback (innermost last):
> File "<stdin>", line 1, in ?
> File "/opt/gnu/lib/python1.4/NumPy/Numeric.py", line 58, in concatenate
> return multiarray.concatenate(a)
> ValueError: arrays must have same number of dimensions
You must add the missing dimension to a before concatenating. The
easiest solution is:
>>> concatenate((c, a[NewAxis,:]))
array([[1, 2, 3],
[2, 4, 6],
[1, 2, 3]])
> but that sounds extremely silly. In my past in Gauss:
> d = c | a
> would have done the job.
I don't know much about Gauss. Does it treat all arrays as
two-dimensional, like Matlab? In that case there is never a dimension
mismatch.
> In addition, in Gauss
> Horizontal would be e = a~b
> Transposing c would have been
> t = c'
> dot product would be
> d = a .* b'
> This to me faster than writing transpose(c) or concatenate((a,b)1), etc...
Gauss is a special array language, whereas Python is a general-purpose
language with arrays. If Python had a short notation for all its
domains of application, it would be a complete mess. So it follows the
"spell-everything-out" approach (like Mathematica).
> Is there shorter method in Python? I guess I could write a emacs key. Does
> anyone have one already?
Functions are objects in Python, so you can create shorthands by assignment:
TR = transpose
CC = concatenate
A = array
CC(TR(a), A([2, 3, 4]))
You can then put all your favourite abbreviations into a module and
import from it. But don't expect everyone else to understand your
private notation!
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Tue Jul 1 09:58:43 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 1 Jul 1997 10:58:43 +0200
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199706301843.OAA03287@dante.mh.lucent.com>
(arw@dante.mh.lucent.com)
Message-ID: <199707010858.KAA31442@lmspc1.ibs.fr>
> I propose that arraytype[arraytype] automatically be interpreted
> as (1) gather as a value (2) scatter as an l-value. I apologize if
> this is obviously dumb, incompatible with existing usage, or
> already proposed (I'm new, remember, don't hurt me!) :).
Probably it has already been proposed, since anything imaginable has
already been proposed. ;-)
Let's make this a bit more specific:
array[sequence [, sequence [, sequence ...]]
would be the most general form, with any sequence of integers allowed.
But if we allow tuples, we have to decide on the interpretation of
array[(2, 3)]
For compatibility this will have to remain equivalent to array[2, 3].
(In fact we have little choice there; this equivalence is enforced by
the interpreter.) So allowing tuples creates trouble. On the other
hand, not allowing tuples creates a very unpythonic restriction. So
there's the first problem to be resolved.
Second problem: do we allow arrays of rank > 1 as indices? If yes,
what are the exact semantics? One interpretation would make
a[b]
an array of the same shape as b with a[b][i, j, k,...] = a[b[i, j, k,...]].
This gets a bit complicated when a has rank > 1, but it's still doable
and probably useful.
Third problem: an index sequence can contain the same index more than
once. Fine for extracting data from an array, but what is supposed
to happen if this occurs on the left-hand side? For example, what is
a[[0, 1, 0]] = [1, 2, 3]
supposed to do?
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Tue Jul 1 10:52:46 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Tue, 01 Jul 1997 05:52:46 -0400
Subject: [MATRIX-SIG] Newbie Q.
References: <199707010825.KAA31356@lmspc1.ibs.fr>
Message-ID: <33B8D36E.1988@ix.netcom.com>
Konrad Hinsen wrote:
> I don't know much about Gauss. Does it treat all arrays as
> two-dimensional, like Matlab? In that case there is never a dimension
> mismatch.
Matlab no longer treats all arrays as two-dimensional.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From P.S.Craig@durham.ac.uk Tue Jul 1 13:35:06 1997
From: P.S.Craig@durham.ac.uk (P.S.Craig@durham.ac.uk)
Date: Tue, 1 Jul 1997 13:35:06 +0100 (BST)
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199707010858.KAA31442@lmspc1.ibs.fr> from "Konrad Hinsen" at Jul 1, 97 10:58:43 am
Message-ID: <9478.199707011235@laplace>
onrad Hinsen writes:
>
> > I propose that arraytype[arraytype] automatically be interpreted
> > as (1) gather as a value (2) scatter as an l-value. I apologize if
> > this is obviously dumb, incompatible with existing usage, or
> > already proposed (I'm new, remember, don't hurt me!) :).
>
> Probably it has already been proposed, since anything imaginable has
> already been proposed. ;-)
>
> Let's make this a bit more specific:
>
> array[sequence [, sequence [, sequence ...]]
>
> would be the most general form, with any sequence of integers allowed.
> But if we allow tuples, we have to decide on the interpretation of
>
> array[(2, 3)]
>
> For compatibility this will have to remain equivalent to array[2, 3].
> (In fact we have little choice there; this equivalence is enforced by
> the interpreter.) So allowing tuples creates trouble. On the other
> hand, not allowing tuples creates a very unpythonic restriction. So
> there's the first problem to be resolved.
I think this does expose a flaw in the way in which multiple indexing
is handled in python. That we cannot distinguish a[(2,3)] from a[2,3]
is unfortunate. However, the potential ambiguity only arises if we
allow rank 1 arrays to be indices for arrays of higher rank. If we
don't, I can't see a problem. This is a convenience which S (and
S-PLus) allow. However, in Numerical Python we can write ravel(a)[b]
which is not (efficiently) available in S. It should be OK as an
lvalue since ravel(a) does not copy a's data but creates a new
reference. If this is too cumbersome, surely we can find some
shorthand for the ravelling operation.
>
> Second problem: do we allow arrays of rank > 1 as indices? If yes,
> what are the exact semantics? One interpretation would make
>
> a[b]
>
> an array of the same shape as b with a[b][i, j, k,...] = a[b[i, j, k,...]].
> This gets a bit complicated when a has rank > 1, but it's still doable
> and probably useful.
How about this? Suppose that b.shape[-1]==len(a.shape) Then
a[b].shape will be b.shape[:-1] and, for example, a[b][i,j,k] =
a[b[i,j,k,:]] as you propose (of course the latter may be written
a[b[i,j,k]] because ofthe way Numerical Python works. This extends
what happens in S to rank(b)>2. I think it's also consistent with what
I said earlier.
I think it may even be possible to go beyond this to the situation
where b.shape[-1]<len(a.shape). Then a[b].shape will be the
concatenation of b.shape[:-1] and a.shape[b.shape[-1]:]. and again
a[b][i,j,k] = a[b[i,j,k]]
In fact, we might be able to go yet further to, for example,
a[:,b]. This would have shape the concatentaion of a.shape[0],
b.shape[:-1] and a.shape[1+b.shape[-1]:] and a[:,b][i,j,k] would be
the same as a[i][b[j,k]]
I'd better stop, but I think the possibilities are tremendous. Of
course, someone will probably spot an ambiguity somewhere!
>
> Third problem: an index sequence can contain the same index more than
> once. Fine for extracting data from an array, but what is supposed
> to happen if this occurs on the left-hand side? For example, what is
>
> a[[0, 1, 0]] = [1, 2, 3]
>
> supposed to do?
> --
In S-Plus, a[0] becomes 3. In other words, some form of left to right
evaluation should be used? We would have to be very careful to avoid
ambiguity.
New problem: Can we allow an expression such as
a[b][c] = d
In other words, c is applied as an index to the array a[b]. I think
that any situation like this can be written as
a[bc] = d
where bc is suitable determined from b, c and the shape of a. The idea
is to avoid computation of a[b] which is likely to require copying of
data.
At this point, I'm willing to volunteer to do some of the work
involved in making all this happen as I wish to use python as the base
for developing ideas that go beyond what we have in S and
S-Plus. Anyone got any suggestions as to how we should organise
things?
Peter Craig
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Tue Jul 1 14:07:10 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 1 Jul 1997 15:07:10 +0200
Subject: [MATRIX-SIG] Data filtering
Message-ID: <199707011307.PAA32490@lmspc1.ibs.fr>
I'll soon need to do some data filtering, and before writing my own
I'd like to ask whether someone has already done it (in Python, of
course). What I need is a low-pass filter, preferably one that can
treat long data streams in smaller segments.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hyoon@nyptsrv1.etsd.ml.com Tue Jul 1 14:06:29 1997
From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant)
Date: Tue, 1 Jul 1997 09:06:29 -0400
Subject: [MATRIX-SIG] Re: Newbie Q.
Message-ID: <199707011306.JAA22948@ptq3.etsd>
To Konrad, Andrew, and others,
Thank you much for help. As some of you suspected, Gauss is 2D only.
The main reason for the usage is the fact that Gauss and Ox are speed
demons, when compared to Splus and MatLab. For further info look at:
http://www.siu.edu/~econ/matlab.ps
http://www.ku-eichstaett.de/docs/WWF/STA/download.htm
These are papers written by a Professor interested in Matrix Langs.
I confirmed that Splus at least is about 10 to 30 times slower than
Gauss in most operations. Never did it with Mat Lab, however, according to
the papers, don't bother trying. Splus and Mat Lab is easier and has prettier
graphics than Gauss or Ox. Splus in particular has column headers associated
with it's matrix, which is shadowed. The headers stays off calcs, but available
for choosing (like dictionary indexes). This one feature I wish Python had. Well,
at least, Python will let me do this, if I write the functions on top of math
functions.
FYI: OX is multidimentional and object oriented, but not Gauss (2D).
In some tests Gauss and Ox will even outperform, decently optimized
C code. Ox for DOS & Unix is free for academics and both of them are
built for Econometrics. I do not consider them in a same catagory as
Python, since they have no GUI and no generalities. Gauss's
graphics are better than Python's in some ways right now. I would love
to see somekind of wrapper for either in Python, since both of them will
be comming out with a DLL for Win and Unix soon.
**************
One additional Q. What is the best general purpose graphics addition in
Python. I am talking about bar/ histogram/ line/ scatter/ pie/ & may be
even 3D surface kind of thing. PyGist is very good, I love it, but I
had some problem dealing with the kind of minimal business graphics I had
above. It will do line/ scatter/ 3D contour better than things I could
buy, but not for simple Bar or Stacked Bar. Not to mention I could not
use strings as x labels. Also, BLT extention to Tkinter seems to not quite
kept up with Tkinter changes. My SA had problem installing it.
Thanks for the info!
Hoon,
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Tue Jul 1 15:27:27 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Tue, 1 Jul 1997 10:27:27 -0400
Subject: [MATRIX-SIG] reverse of take?
Message-ID: <199707011424.KAA03491@dante.mh.lucent.com>
> From: Andrew P. Mullhaupt <amullhau@ix.netcom.com>
> At 02:45 PM 6/30/97 -0400, Aaron Watters wrote:
>
> > I propose that arraytype[arraytype] automatically be interpreted
> >as (1) gather as a value (2) scatter as an l-value. I apologize if
> >this is obviously dumb, incompatible with existing usage, or
> >already proposed (I'm new, remember, don't hurt me!) :).
>
> This proposal is OK with me and is essentially what a lot of people
> wish Python did - and not just the numerical arrays. (No matter though-
> the numerical arrays are probably general enough).
In other contexts (dictionaries at least) it makes sense that indexing
always has the same single-mapping interpretation. For example in
kjbuckets
Dict1[Set1]
retrieves the map for Set1 from Dict1 (ie, Set1 is a left member
of Dict1). The alternate interpretation "all maps for elts of Set1
from Dict1" can be got from a different operation, composition:
(Set1 * Dict1).values()
I wouldn't have it any other way :). But for arrays it seems that
Array1[Array2]
naturally interprets the rows of Array2 as indices of Array1 and
returns the elements of Array1 (which might be arrays too, I suppose)
for those indices. I suppose
List[Tuple]
might possibly return the elements of List at members of Tuple
but I suspect that this is the kind of sneakiness that Guido would
prefer to avoid in the core. High performance specialized
extensions such as Numeric can afford greater sneakiness tho
for important operations.
>From these discussions, the biggest technical problems I see
for scatter/gather-as-indexing are
A[(2,3)] = x
is ambiguous, as Konrad points out. I say just make tuples
a special incompatible case... maybe only support the scatter/gather
things for arrays as indices... oh well.
Also Peter Craig notes a problem with:
A[A2][A3] = x
[Note: to python this parses as value A[A2] indexed assigned at A3
to x. The intuitive reading is change A at indexes A2 reindexed at
indices A3 to x.]
In this notation the updates should propagate all the way back to A
as Peter Craig points out. This is a Big Problem AFAIK, since I'm
not sure how it could be resolved (not understanding the internal
representation). This suggests to me that maybe having the existing
"take" function and a dual "set" function might be a better idea once
again instead of jumping all the way to generalized indices. If a
"scatter/gather" index notation cannot be made fully general I suspect
it would cause a lot of confusion. I do think that it is important
to have a dual for "take" however it looks notationally.
-- Aaron Watters
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From motteler@laura.llnl.gov Tue Jul 1 15:41:19 1997
From: motteler@laura.llnl.gov (Zane C. Motteler)
Date: Tue, 1 Jul 1997 07:41:19 -0700 (PDT)
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser>
Message-ID: <Pine.HPP.3.95.970701073416.29412A-100000@laura.llnl.gov>
On Mon, 30 Jun 1997, Andrew P. Mullhaupt wrote:
>
>Can you index a multi-dimensional array with a one-dimensional array? In
>other words does arraytype[range(10)] refer to the first 10 elements of
>the array? I think this would be doable using other more verbose methods,
>but this is a simple way to spell this.
>
>Can you index a one-dimensional array with a multi-dimensional array? In
>languages which provide this, this usually returns a value with the same
>shape as the matrix and the values corresponding to the value in the vector
>with that index. This is really nice for image processing - for example
>palette remapping. You can get this by applying a function, but that is
>usually a lot slower.
Both cases are ambiguous in NumPy. NumPy arrays are not necessarily
stored contiguously (transpose merely exchanges some dimensions and
strides, but doesn't move any elements). This is obviously done for
speed. Secondly, to do these two things, even if arrays are contiguous,
means that users have to know whether arrays are stored in row-major or
column-major order, which is antithetical to the idea of a high-level
language.
>
>If array0 is d-dimensional and arrayk is 1-dimensional for 1<=k<=d, does
>array0[array1, ..., arrayd] refer to the elements corresponding to the
>Cartesian product of array1 ... arrayd? It should. What is the shape
>of the result when some of the index arrays have length 1? Should there be
>an additional argument to control this? (In S, x[1,1:5] is 1-dimensional with
>length 5 and x[1,1:5,drop=F] is 2-dimensional with shape (1, 5). - The reason
>you probably want this is that it can be really annoying to reshape the
>result which has been collapsed if it was an intermediate expression.)
Getting too esoteric here. I'm not even too sure how useful this would
be. It really demands intimate knowledge of the internal structure of
arrays.
>
>1. A function which returns the permutation which would put the elements of
> a 1-d array in order. In APL this is called 'upgrade' and in S it is called
> 'order'. A quick illustration of the use of this is computing the ranks
> of elements in a vector - I'll suppose we call the function order.
>
> def rank(x) :
> return order(order(x))
My arrayfnsmodule has such a function, called index_sort.
Zane
-----------------------------------------------------------------------------
Zane C. Motteler, Ph. D. ___________ALSO:_________________
Computer Scientist | Professor Emeritus of Computer|
Lawrence Livermore National Laboratory | Science and Engineering |
P O Box 808, L-472 | California Polytechnic State |
Livermore, CA 94551-9900 | University, San Luis Obispo |
510/423-2143 ---------------------------------
FAX 510/423-9969
zcm@llnl.gov or motteler@laura.llnl.gov or zmottel@calpoly.edu
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Tue Jul 1 16:02:31 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 1 Jul 1997 17:02:31 +0200
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser>
(amullhau@ix.netcom.com)
Message-ID: <199707011502.RAA00029@lmspc1.ibs.fr>
> Can you index a multi-dimensional array with a one-dimensional array? In
> other words does arraytype[range(10)] refer to the first 10 elements of
> the array? I think this would be doable using other more verbose methods,
> but this is a simple way to spell this.
This issue is already settled. Obviously, array[range(10)] should be
the same as array[:10], right? And that expression is equivalent to
array[:10, ...].
To get the first ten elements, use ravel(array)[:10].
> Can you index a one-dimensional array with a multi-dimensional array? In
> languages which provide this, this usually returns a value with the same
> shape as the matrix and the values corresponding to the value in the vector
> with that index. This is really nice for image processing - for example
> palette remapping. You can get this by applying a function, but that is
> usually a lot slower.
I see no reason why this shouldn't be allowed in NumPy.
> 1. A function which returns the permutation which would put the elements of
> a 1-d array in order. In APL this is called 'upgrade' and in S it is called
> 'order'. A quick illustration of the use of this is computing the ranks
And in NumPy it's called argsort!
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Tue Jul 1 16:18:18 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 1 Jul 1997 17:18:18 +0200
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199707011424.KAA03491@dante.mh.lucent.com>
(arw@dante.mh.lucent.com)
Message-ID: <199707011518.RAA00072@lmspc1.ibs.fr>
> Also Peter Craig notes a problem with:
>
> A[A2][A3] = x
>
> [Note: to python this parses as value A[A2] indexed assigned at A3
> to x. The intuitive reading is change A at indexes A2 reindexed at
> indices A3 to x.]
>
> In this notation the updates should propagate all the way back to A
> as Peter Craig points out. This is a Big Problem AFAIK, since I'm
They do with the current NumPy indexing, which returns not a copy
of the array, but a reference to the same data space with different
pointers and offsets. But that would be hard to keep up with general
list indices!
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Tue Jul 1 16:15:46 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 1 Jul 1997 17:15:46 +0200
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <9478.199707011235@laplace> (P.S.Craig@durham.ac.uk)
Message-ID: <199707011515.RAA00060@lmspc1.ibs.fr>
> I think this does expose a flaw in the way in which multiple indexing
> is handled in python. That we cannot distinguish a[(2,3)] from a[2,3]
Right, but there wasn't much choice. To keep the existing interface
to both classes and C types, an index had to be limited to a single
Python object. And using tuples for multiple indexing is also useful
for code that has to build an index expression from pieces.
> is unfortunate. However, the potential ambiguity only arises if we
> allow rank 1 arrays to be indices for arrays of higher rank. If we
That is already allowed and well defined! However, it's not what
you seem to think it is. The definition is that the index is applied
to the first dimension of the array. This makes arrays (of any dimension)
conform to the sequence protocol, and makes them very interchangeable
with nested lists.
> In fact, we might be able to go yet further to, for example,
> a[:,b]. This would have shape the concatentaion of a.shape[0],
> b.shape[:-1] and a.shape[1+b.shape[-1]:] and a[:,b][i,j,k] would be
> the same as a[i][b[j,k]]
And be the first step to the perlization (or J-ization) of NumPy:
combinations of funny symbols having a meaning not deducible from
the meanings of their pieces. Besides, Guido will probably refuse
any syntax change of that kind!
> In S-Plus, a[0] becomes 3. In other words, some form of left to right
> evaluation should be used? We would have to be very careful to avoid
> ambiguity.
Especially for higher-dimensional cases!
> New problem: Can we allow an expression such as
>
> a[b][c] = d
Syntactically yes.
> S-Plus. Anyone got any suggestions as to how we should organise
> things?
First we have to agree on what we want, and that could be difficult...
Then do an implementation in a separate module, with function syntax,
just for testing. In parallel, check with Jim (are you following all
this?) about the status of the NumPy source code - no point in
modifying code that is still changing. Finally, put it all together
and be happy...
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From P.S.Craig@durham.ac.uk Tue Jul 1 16:48:34 1997
From: P.S.Craig@durham.ac.uk (P.S.Craig@durham.ac.uk)
Date: Tue, 1 Jul 1997 16:48:34 +0100 (BST)
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199707011515.RAA00060@lmspc1.ibs.fr> from "Konrad Hinsen" at Jul 1, 97 05:15:46 pm
Message-ID: <9838.199707011549@laplace>
Konrad Hinsen writes:
>
From amullhau@ix.netcom.com Tue Jul 1 18:14:46 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Tue, 01 Jul 1997 13:14:46 -0400
Subject: [MATRIX-SIG] Re: Newbie Q.
In-Reply-To: <199707011306.JAA22948@ptq3.etsd>
Message-ID: <3.0.1.32.19970701131446.0069cf34@popd.netcruiser>
At 09:06 AM 7/1/97 -0400, Hoon Yoon - IPT Quant wrote:
>To Konrad, Andrew, and others,
>
> Splus in particular has column headers associated
> with it's matrix, which is shadowed.
I'm not sure what you mean by shadowed. But the mechanism by which S-plus
has 'column headers' is to add an attribute called dimnames to the array.
This attribute is a list containing a character vector for each dimension.
It would be very easy to obtain this functionality in python by subclassing
arrays. So you'll probably see this one soon. The only issue is whether we
allow you to use the dimnames for indexing, as in S.
Later,
Andrew Mulhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Tue Jul 1 19:20:56 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Tue, 01 Jul 1997 14:20:56 -0400
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199707011502.RAA00029@lmspc1.ibs.fr>
References: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser>
Message-ID: <3.0.1.32.19970701142056.006dab44@popd.netcruiser>
At 05:02 PM 7/1/97 +0200, Konrad Hinsen wrote:
>> Can you index a multi-dimensional array with a one-dimensional array? In
>> other words does arraytype[range(10)] refer to the first 10 elements of
>> the array? I think this would be doable using other more verbose methods,
>> but this is a simple way to spell this.
>
>This issue is already settled. Obviously, array[range(10)] should be
>the same as array[:10], right? And that expression is equivalent to
>array[:10, ...].
I know everyone wants this, I just didn't see it in Aaron's proposal, so I
asked. Since he's come recently.
>> Can you index a one-dimensional array with a multi-dimensional array?
>
>I see no reason why this shouldn't be allowed in NumPy.
Me either. But I didn't see it in Aaron's proposal, so I asked.
>> 1. A function which returns the permutation which would put the elements of
>> a 1-d array in order. In APL this is called 'upgrade' and in S it is
called
>> 'order'. A quick illustration of the use of this is computing the ranks
>
>And in NumPy it's called argsort!
But wait - what about index_sort? :-)
It turns out that this function is quite important, and so is his friend
"dyadic iota". The sort for argsort/index_sort usually has to be really good
in case of many duplicated values, as well as reversed values, etc.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From P.S.Craig@durham.ac.uk Tue Jul 1 23:56:47 1997
From: P.S.Craig@durham.ac.uk (P.S.Craig@durham.ac.uk)
Date: Tue, 1 Jul 1997 23:56:47 +0100 (BST)
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited
Message-ID: <10497.199707012256@laplace>
Hi all,
I have decided to stick my neck out and try to bring a little order to
the discussion about extending NumPy's indexing that has been going on
over the last week or so. There are clearly a number of schools of
thought about what should be done, ranging from Aaron Watter's minimal
suggestion that we have a put function corresponding to the existing
take function to Andrew Mullhaupt's and my desires for fairly exotic
indexing semantics based on ideas found in S and APL. I would like to
suggest the following program of development:
a) implement a give function which is the converse of the current take
function. It's still not entirely clear what this means. I would
tentatively propose:
give(a, b, seq, axis=0)
where a and b are NumPy arrays and seq is an integer value sequence
such that take(a, seq, axis) is an array of the same shape as
b. Formally, having checked the arguments, the function would do
the equivalent of
axisoffset = (slice(None,None,None),)*axis
for i in range(len(seq)):
a[axisoffset+(seq[i],)] = b[axisoffset+(i,)]
This has the nicely symmetric property that having done
give(a,b,seq,axis) then b and take(a, seq, axis) are arrays of the
same shape with the same contents.
I wonder how much efficiency gain there would be with a C implementation?
b) For most, if not all, of the indexing ideas that have been thrown
about, we want to be able to efficiently index more than one axis
simultaneously. So we appear to want versions of take and give that
operate on more than one axis simultaneously.
A natural extension to the current take function would be
multitake(a, (seq1,seq2,...,seqm), axes=range(m))
where m is the number of axes being simultaneously indexed. The
effect of assigning the result to b would be the same as
b = a
seqs = (seq1,seq2,...,seqm)
for i in range(m):
b = take(b, seqs[i], axes[i])
but could be much more efficiently implemented.
multigive is more challenging to define. In the spirit of my
proposal for the give function, this might be
multigive(a, b, (seq1,seq2,...,seqm), axes=range(m))
where multitake(a,(seq1,seq2,...,seqm), axes) would be an array of
the same shape as b
c) The simplest extension to current NumPy indexing is to allow an
arbitrary integer valued sequence as well as integers or slices as
an index. Thus, for example,
a[:,seq1,:,seq2,seq3]
would be the same as
multitake(a,(seq1,seq2,seq3),(1,3,4))
and
a[:,seq1:,seq2,seq3] = b
would be the same as
multigive(a,b,(seq1,seq2,seq3),(1,3,4))
Note that, according to the proposal so far,
a[seq1][seq2] = b
would be syntactically correct but would fail to alter a as a[seq1]
would have a copy of the apropriate part of a's data and would not
be a reference to a's data.
Mixing of slice and sequence indexing seems fairly straightforward
to implement.
d) It is interesting to note that if the take (and presumably also
multitake) function returned a reference object instead of copying
data to a new array, then the give (and multigive) function would
be trivial to implement.
At present,indexing in NumPy using slices is done by reference and
the resulting object is a highly efficient array implementation
because the memory location for a[i1,i2,...,ij ,...,in] is the
memory location for a[i1,i2,...,0,...,in] + ij *sj where sj is the
stride length for axis j. In other words, the reference object is
no different from a directly created array except for the value of
sj.
The difficulty with a reference implementation for indexing by
arbitrary sequences is that the resulting array will be less
efficient than the basic NumPy array. The implementation is fairly
straightforward and would have the advantage that a statement such
as
a[[1,3,4]][2]=1
would actually have the desired effect of changing a[4].
The other difficulty with a reference implementation for take is
that the default behaviour would have to remain copying. Otherwise,
existing code could well break.
Actually, once the basics of a reference implementation of take
have been sorted, multitake is a trivial function to write
efficiently by repeated use of take.
e) More exotic indexing ideas, such as those found in S and advocated
by Andrew Mullhaupt and me, would be the subject of individual or
collective experimentation using python classes for the time
being. I think that the ideas in a)-c) and possibly also d) would
provide a good basis for playing with more powerful index
notation. If a concensus emerged, we could then take the process
further.
Now, some of this is fairly straightforward. A start on code for a) is
Zane Mosteller's array_set function. However, this probably needs some
work. b) can either be tackled directly once a) has been completed or
can be tackled by doing d). c) is trivial once b) is done.
To my mind, the big question is whether we think d) is a good idea. If
it is, it really means having two different array objects, one of
which is Jim Hugunin's current object and the other of which
implements a more general reference array type. The latter is not
difficult to implement, I think. The problem is that the two
implementations cannot really be separated if we want efficiency. I
don't think we can implement as a separate module. Unfortunately, my
impression from Jim H's last message was that he does not have any
time to give to NumPy for the time being. So we would have to make
changes to NumPy code and hope that we could keep the changes up to
date if NumPy evolved. Of course, if we could get the changes accepted
into NumPy then all would be well.
I'm sorry to have been so long-winded. If anyone is still reading at
this point and is still interested, please comment. Otherwise, I might
just go ahead by myself anyway sometime in the next couple of months.
Exhaustedly yours,
Peter Craig
#--------------------------------------------------------------------#
| E-mail: P.S.Craig@durham.ac.uk Telephone: +44-91-3742376 (Work) |
| Fax: +44-91-3747388 +44-91-3860448 (Home) |
| |
| WWW: http://fourier.dur.ac.uk:8000/stats/psc.html |
| |
| Snail: Peter Craig, Dept. of Math. Sciences, Univ. of Durham, |
| South Road, Durham DH1 3LE, England |
#--------------------------------------------------------------------#
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Tue Jul 1 20:32:39 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Tue, 1 Jul 1997 15:32:39 -0400
Subject: [MATRIX-SIG] reverse of take?
Message-ID: <199707011929.PAA06248@dante.mh.lucent.com>
> a[i, j, k, hint=flavor_woo_woo_17]
I'm reeling about not following this discussion, but I suspect
the best approach is for Jim H. to "just do it". I personally
begin to think that excess overloading of indexing is not
advisable, possibly confusing. I vote for "take()" as is with dual
"put()" or whatever. Of course, my experience with numeric
matrix tools such as these is infinitesimal, so my opinion
may be of limited merit. If we can get by without asking Guido
to change the core, so much the better. -- Aaron Watters
ps: in case you haven't guessed I want "put" in order to be
able to do graphical type operations in some generality
using Numeric alone -- this is all that's missing from what
I can see. IE compute the indices of a 1d figure using
umath operations and then use put() to draw on those
indices into a Numeric array frame buffer. This, plus a screen
mapping operation and mouse interfaces
of some kind could serve as the basis
of an simple windowing/graphics system based in Python,
I believe. Of course it'd be nontrivial, but after a good
start I think the Python community (via starship, maybe)
could grow some neat stuff...
fonts == arrays of bytes
cropping == slicing
colormap == choose
map_to_screen == slice assignment...
just a daydream...
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From phil@geog.ubc.ca Wed Jul 2 02:54:04 1997
From: phil@geog.ubc.ca (Phil Austin)
Date: Tue, 1 Jul 1997 18:54:04 -0700
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited
In-Reply-To: <10497.199707012256@laplace>
References: <10497.199707012256@laplace>
Message-ID: <199707020154.SAA22149@curlew.geog.ubc.ca>
Aaron> I'm reeling about not following this discussion, but I
Aaron> suspect the best approach is for Jim H. to "just do it". I
Aaron> personally begin to think that excess overloading of
Aaron> indexing is not advisable, possibly confusing. I vote for
Aaron> "take()" as is with dual "put()" or whatever. Of course,
Aaron> my experience with numeric matrix tools such as these is
Aaron> infinitesimal, so my opinion may be of limited merit. If
Aaron> we can get by without asking Guido to change the core, so
Aaron> much the better. -- Aaron Watters
Here's the perspective of someone who intends to teach NumPy to 3rd
year geography majors without much exposure to computers: I would
<much> rather have the S syntax describe by PSC and APM than the
current numeric functions. Like Python, it takes a while to
penetrate, but once it does, even naive computer users typically find
it intuitive. Additional evidence is available in the S-news
archives, where hundreds of real neophytes have asked questions of the
old timers. A couple of array examples are typically enough to get the
beginners started (and there are some good public domain S tutorials
that could be pirated for this). Once the NumPy prototype code is
available, the indexing should sell itself to members of this list and
to the broader community.
Phil
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Wed Jul 2 04:01:21 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Tue, 01 Jul 1997 23:01:21 -0400
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take"
revisited
In-Reply-To: <199707020154.SAA22149@curlew.geog.ubc.ca>
References: <10497.199707012256@laplace>
<10497.199707012256@laplace>
Message-ID: <3.0.1.32.19970701230121.006a3bac@popd.netcruiser>
At 06:54 PM 7/1/97 -0700, Phil Austin wrote:
>
>Here's the perspective of someone who intends to teach NumPy to 3rd
>year geography majors without much exposure to computers: I would
><much> rather have the S syntax describe by PSC and APM than the
>current numeric functions.
Especially since a lot of GIS style stuff is easier to do that way, right?
> Once the NumPy prototype code is
>available, the indexing should sell itself to members of this list and
>to the broader community.
There are two other things beyond upgraded indexing which are needed before
NumPy will take over the numerical world.
1. A decent graphics package should be glued on pretty tightly.
2. Installation must be simplified.
Then, I would agree with you - NumPy will be very, very popular.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Tue Jul 1 19:15:35 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Tue, 01 Jul 1997 14:15:35 -0400
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <Pine.HPP.3.95.970701073416.29412A-100000@laura.llnl.gov>
References: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser>
Message-ID: <3.0.1.32.19970701141535.006d53f8@popd.netcruiser>
At 07:41 AM 7/1/97 -0700, Zane C. Motteler wrote:
>On Mon, 30 Jun 1997, Andrew P. Mullhaupt wrote:
>
>>
>>Can you index a multi-dimensional array with a one-dimensional array?
>>
>>Can you index a one-dimensional array with a multi-dimensional array?
>
>Both cases are ambiguous in NumPy. NumPy arrays are not necessarily
>stored contiguously (transpose merely exchanges some dimensions and
>strides, but doesn't move any elements).
They shouldn't be ambiguous at all. And if you're thinking of what I
think you're thinking of - well, I don't think anyone would want _that_ at
all. I think you interpret the indices as referring to the _locations at
which the elements are stored_ and this is not always true.
The way to sensibly index a multi dimensional array with a one-dimensional
array is _not_ to interpret the indices in terms of the possibly unpredictable
order the elements are stored in, but to treat the one-dimensional array as
indices into the 'full dimensional' indices of the array as determined by
the shape of the array. Thus, it is completely unambiguous, independent of
the order in which the actual elements are laid out, and it also
corresponds to what most users would expect.
The other case, indexing a one-dimensional array by a multi-dimensional
array is also completely unambiguous - you are using the elements of the
multi-dimensional array as indices into the one-d array and then giving the
result the shape of the original m-d array. Similar to the previous case,
this does not depend on the order in which the elements of the one-d array
are actually
stored.
> Secondly, to do these two things, even if arrays are contiguous,
>means that users have to know whether arrays are stored in row-major or
>column-major order, which is antithetical to the idea of a high-level
>language.
Actually, it doesn't mean that at all. It is quite clear that you can
construct a mathematical theory of APL indexing, which supports these
constructs without any reference to row or column majority. (I've alluded
to this mathematical construction before.) In fact, as I have pointed out
above, the _indexing_ can be (and in some cases in python _needs_ to be)
independent of the storage order.
However, for reasons of speed, whether you tell them or not, people will
find out many things about how the elements of arrays are stored. The APL
implementation community tried very hard for a long time to hide this from
users to no avail.
How, for example, will a user be able to pass an array to his favorite
multi-dimensional graphics package, without he can put the elements in the
right places for that package? It is quite simple to see that users _need_
be able to find out in any particular case, as necessary, the order of the
elements.
The goal of the higher level language is to reduce this need to a minimum,
_but no further_.
>>
>>If array0 is d-dimensional and arrayk is 1-dimensional for 1<=k<=d, does
>>array0[array1, ..., arrayd] refer to the elements corresponding to the
>>Cartesian product of array1 ... arrayd? It should. What is the shape
>>of the result when some of the index arrays have length 1?expression.)
>
>Getting too esoteric here. I'm not even too sure how useful this would
>be. It really demands intimate knowledge of the internal structure of
>arrays.
I think you're confused about the meaning of this based on your previous
statements.
As to useful, it is perhaps the single most common case of indexing in
array languages, and not only that, perhaps the _most efficient_.
Here's an example:
a = reshape(range(12), [3,4])
which looks like
0 1 2 3
4 5 6 7
8 9 10 11
then you want
a[[0,2], [2,3]]
to be
2 3
10 11
right? I think a lot of people have been thinking that this would be the
case (like APL, S, Matlab, etc.).
The question here is actually not _can_ you do this, (I can easily give a
good implementation of what I outlined so far). But we are not at that
point yet.
The indexing questions I have asked are all about determining what behavior
people want. There are many consistent behaviors which people coming from
different languages are already happy with, and the trick here is to figure
out which one people want. There are _no_ issues of "can't get there from
here" in the context of Python's particular legacy choices - the questions
are all in the "do we _want_ to get there from here" sort.
There is no question that indexing will get improved in the way people use
Numerical Python - that is not the issue either. Too many people have done
too much cool stuff (for decades) for that to be a serious issue.
The question is not _how these things can be done_. Almost everything that
anyone has suggested so far falls well within the scope of things that I
know can be done quite practically, and other people may know how to do
many things I don't, so as far as we have seen to date, it's all quite doable.
The serious issue is _which flavor to choose_. It will be a really good
thing if the people in the discussion make themselves familiar with the
various choices before trying to implement something serious - or else we
will end
up with several parallel efforts which conflict in some aspects. I can
think of at least two completely reasonable ways to specify indexing which
are not compatible. For example - APL's rigid respect of dimensions as
opposed to S's "cyclic reuse" policy - There are advantages and
disadvantages to both. I know which one I like better, but either would be
fine for python.
The reason that indexing is _different_ than other cases where the new
functionality has "function" syntax is that _beacuse_ we call it "indexing"
we are all talking about syntax which is going to overlap (read collide)
heavily in different implementations. So it is likely that only _one_
approach can survive. It would be really nice if this choice isn't
determined solely by the first version to be implemented.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Tue Jul 1 19:33:55 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Tue, 01 Jul 1997 14:33:55 -0400
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199707011515.RAA00060@lmspc1.ibs.fr>
References: <9478.199707011235@laplace>
Message-ID: <3.0.1.32.19970701143355.006dacac@popd.netcruiser>
At 05:15 PM 7/1/97 +0200, Konrad Hinsen wrote:
>
>> This would have shape the concatentaion of a.shape[0],
>> b.shape[:-1] and a.shape[1+b.shape[-1]:] and a[:,b][i,j,k] would be
>> the same as a[i][b[j,k]]
>
>And be the first step to the perlization (or J-ization) of NumPy:
>combinations of funny symbols having a meaning not deducible from
>the meanings of their pieces. Besides, Guido will probably refuse
>any syntax change of that kind!
Me too. I think a better approach to adding flavors/bells/whistles to
indexing is by adding a default argument to the index, as in
a[i, j, k, hint=flavor_woo_woo_17]
for exceptional cases. This approach works well in S, which means that
is doesn't make bees live in the heads of many naive users.
>First we have to agree on what we want, and that could be difficult...
Yes. This, believe it or not, is the hard part.
>Then do an implementation in a separate module, with function syntax,
>just for testing. In parallel, check with Jim (are you following all
>this?) about the status of the NumPy source code - no point in
>modifying code that is still changing. Finally, put it all together
>and be happy...
Absolutely.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From edcjones@access.digex.net Wed Jul 2 03:27:42 1997
From: edcjones@access.digex.net (Edward C. Jones)
Date: Tue, 01 Jul 1997 22:27:42 -0400
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited
References: <10497.199707012256@laplace>
Message-ID: <33B9BC9E.75317C47@access.digex.net>
P.S.Craig@durham.ac.uk wrote:
> I have decided to stick my neck out and try to bring a
> little order to the discussion about extending NumPy's
> indexing that has been going on over the last week or so.
...
I would like to see block capability added to NumPy. Memory management,
numerical analysis, image processing, computer vision, and signal
processing all need blocks. Examples include transposing large matrices,
zooming or shrinking images, quadtrees, and wavelets.
The most useful current capability of this sort is "reduceat". The
manual says:
|ufunc.reduceat(a, indices, axis=0)
|
| This is a weird function, and most people should just ignore it. It
| will reduce a to each of the given indices so that as new size along
| the given axis will be the same as the length of indices.
This function is not weird. It is natural, powerful and useful. It
reduces each block to one value.
Something like the following are needed (all defined recursively):
* A slice that does not make a copy, so I can modify a part of an image.
* Efficiently view a big array as an array of blocks (each an array
itself) and vice versa.
* Convert an array of arrays to and from a big array.
* The ability to extract, insert, and manipulate sub-blocks of blocks.
For example, consider an interlaced image as an array of blocks, each
with two rows. Construct a pair of new images containing the even and
odd rows. Process these two images separately and recombine them.
Please don't make changes in Python or NumPy indexing schemes without
consider blocks and their users.
What is the best way to put blocks into NumPy?
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Tue Jul 1 18:19:14 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Tue, 01 Jul 1997 13:19:14 -0400
Subject: [MATRIX-SIG] Data filtering
In-Reply-To: <199707011307.PAA32490@lmspc1.ibs.fr>
Message-ID: <3.0.1.32.19970701131914.006d9be0@popd.netcruiser>
At 03:07 PM 7/1/97 +0200, Konrad Hinsen wrote:
>I'll soon need to do some data filtering, and before writing my own
>I'd like to ask whether someone has already done it (in Python, of
>course). What I need is a low-pass filter, preferably one that can
>treat long data streams in smaller segments.
I haven't done this stuff in python, but I have quite a bit of stuff
which would be very easy for me to put into python and give to you. The
question is; can you be more specific about the filter you need? Is it
single or multichannel? Do you have issues with phase as well as magnitude?
Is the segmentation requirement for efficiency or an actual requirement?
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Wed Jul 2 12:21:37 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Wed, 2 Jul 1997 07:21:37 -0400
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited
Message-ID: <199707021137.HAA29599@dante.mh.lucent.com>
> axisoffset = (slice(None,None,None),)*axis
> for i in range(len(seq)):
> a[axisoffset+(seq[i],)] = b[axisoffset+(i,)]
hmmm. After looking at this code I see that part of my
problem was my lack of understanding of the overloading
of indexing. I didn't know that
ar[ array([1,2]) ]
would work. Since it does (without copying anything)
:) a Python implementation of
"give" or "set" or whatever you want to call it actually might
be fast enough for practical purposes. I suppose each
iteration of the loop would require only one header malloc,
which might be inexpensive enough, and a C implementation
might not be too much better. Comment withdrawn?
I knew it was likely I was being dumb.
Where are the details of all the functionality of Numeric
set out? I have to say that it's a shame there isn't
better documentation that I can find. Sorry if I missed it
in DA's tutorial or somewhere else I'm unaware of.
In particular things that aren't immediately obvious to
me are the reference vs. copy semantics, how indexing/slicing
works exactly, probably more I haven't run into yet.
-- Aaron Watters
note: fancy indexing is fine if you experts think it
would be a Good Thing, but remember that Guido
does not change the syntax lightly, for good reason.
Just imagine what damage the regex community
could have done by now if he hadn't resisted. My
preference would be to be conservative here until
maybe a clear consensus arrives in many months.
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Wed Jul 2 12:38:04 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Wed, 2 Jul 1997 07:38:04 -0400
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited
Message-ID: <199707021137.HAA29603@dante.mh.lucent.com>
Sorry. Can't shut up!
> |ufunc.reduceat(a, indices, axis=0)
> |
> | This is a weird function, and most people should just ignore it. It
> | will reduce a to each of the given indices so that as new size along
> | the given axis will be the same as the length of indices.
>
> This function is not weird. It is natural, powerful and useful. It
> reduces each block to one value.
Okay, let's just say the description is weird. A simple two line
example of its use?
> Something like the following are needed (all defined recursively):
>
> * A slice that does not make a copy, so I can modify a part of an image.
> * Efficiently view a big array as an array of blocks (each an array
> itself) and vice versa.
> * Convert an array of arrays to and from a big array.
> * The ability to extract, insert, and manipulate sub-blocks of blocks.
> For example, consider an interlaced image as an array of blocks, each
> with two rows. Construct a pair of new images containing the even and
> odd rows. Process these two images separately and recombine them.
If you are willing to pay a bit in overhead it seems to me that
it might be possible to prototype the concept(s) as a Python class
(or set of Python classes) that wrap an array. After you're happy
with the Python implementation you could "push it down" to C
if it wasn't fast enough. In fact I'd recommend all such proposals
have Python prototypes. It might be the case that some of the
prototypes could be optimized well enough at the Python level
to be just fine. -- Aaron Watters
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Wed Jul 2 19:01:59 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Wed, 2 Jul 1997 20:01:59 +0200
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <3.0.1.32.19970701143355.006dacac@popd.netcruiser>
(amullhau@ix.netcom.com)
Message-ID: <199707021801.UAA04417@lmspc1.ibs.fr>
> Me too. I think a better approach to adding flavors/bells/whistles to
> indexing is by adding a default argument to the index, as in
>
> a[i, j, k, hint=flavor_woo_woo_17]
That's unfortunately impossible with the current Python syntax. An index
expression must consist of standard expressions or slices separated by
commas. No keyword arguments.
Aaron's comment on this:
> > a[i, j, k, hint=flavor_woo_woo_17]
>
> I'm reeling about not following this discussion, but I suspect
> the best approach is for Jim H. to "just do it". I personally
> begin to think that excess overloading of indexing is not
> advisable, possibly confusing. I vote for "take()" as is with dual
Until now we haven't discussed much more overloading - just the
addition of sequence types (including arrays) as indices! And I think
this kind of operation should be a form of indexing simply because
people coming from other array languages *expect* it to be.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Wed Jul 2 19:13:58 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Wed, 2 Jul 1997 20:13:58 +0200
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited
In-Reply-To: <10497.199707012256@laplace> (P.S.Craig@durham.ac.uk)
Message-ID: <199707021813.UAA04447@lmspc1.ibs.fr>
> indexing semantics based on ideas found in S and APL. I would like to
> suggest the following program of development:
...
Looks good!
> d) It is interesting to note that if the take (and presumably also
> multitake) function returned a reference object instead of copying
> data to a new array, then the give (and multigive) function would
> be trivial to implement.
Right.
> The difficulty with a reference implementation for indexing by
> arbitrary sequences is that the resulting array will be less
> efficient than the basic NumPy array. The implementation is fairly
> straightforward and would have the advantage that a statement such
> as
>
> a[[1,3,4]][2]=1
>
> would actually have the desired effect of changing a[4].
It would have another advantage, which I consider even more important:
consistency. In current NumPy, indexing always returns a reference sharing
the data space with the indexed array. If indexing with sequences creates
a copy instead, then things get messy. To know the result of an indexing
operation, you would have to analyze whether any part of the index
is a sequence.
> The other difficulty with a reference implementation for take is
> that the default behaviour would have to remain copying. Otherwise,
> existing code could well break.
take() has to keep on copying, but sequence indexing doesn't have to.
In fact, separating the two has the advantage of providing *both*
functionalities: take() returns a copy that is time-efficient to use,
and indexing returns a reference that is memory-efficient.
> Actually, once the basics of a reference implementation of take
> have been sorted, multitake is a trivial function to write
> efficiently by repeated use of take.
Right.
> e) More exotic indexing ideas, such as those found in S and advocated
> by Andrew Mullhaupt and me, would be the subject of individual or
> collective experimentation using python classes for the time
> being. I think that the ideas in a)-c) and possibly also d) would
That seems like a good compromise.
> To my mind, the big question is whether we think d) is a good idea. If
In my opinion yes!
> difficult to implement, I think. The problem is that the two
> implementations cannot really be separated if we want efficiency. I
Who cares? At the Python level the difference is invisible, and
C extensions tend to ask for a contiguous copy of all input arrays
anyway (since there is no C API for indexing).
> I'm sorry to have been so long-winded. If anyone is still reading at
> this point and is still interested, please comment. Otherwise, I might
> just go ahead by myself anyway sometime in the next couple of months.
Fine with me!
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Wed Jul 2 19:22:20 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Wed, 2 Jul 1997 20:22:20 +0200
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited
In-Reply-To: <33B9BC9E.75317C47@access.digex.net> (edcjones@access.digex.net)
Message-ID: <199707021822.UAA04471@lmspc1.ibs.fr>
> I would like to see block capability added to NumPy. Memory management,
Much of it is already there! You just have to play around with shapes.
Example: I frequently work with matrices that logically consist of 3x3
blocks. For some operations I must work on the blocks, for others on
the whole matrix. So here we go:
matrix_of_blocks = zeros((n, 3, n, 3), Float)
big_matrix = reshape(matrix_of_blocks, (3*n, 3*n))
Since reshape() returns a reference sharing the same data space,
I can use the two arrays interchangeably.
The only restriction is that the blocks must be of equal size.
> * A slice that does not make a copy, so I can modify a part of an image.
That's what indexing already does!
> * Efficiently view a big array as an array of blocks (each an array
> itself) and vice versa.
> * Convert an array of arrays to and from a big array.
Possible with restrictions (see above).
> * The ability to extract, insert, and manipulate sub-blocks of blocks.
Within the same restrictions, simply done by indexing.
Question: do you see a need for supporting non-equal-sized blocks?
That could become very messy.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Wed Jul 2 19:26:56 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Wed, 2 Jul 1997 20:26:56 +0200
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited
In-Reply-To: <199707021137.HAA29599@dante.mh.lucent.com>
(arw@dante.mh.lucent.com)
Message-ID: <199707021826.UAA04487@lmspc1.ibs.fr>
> of indexing. I didn't know that
>
> ar[ array([1,2]) ]
>
> would work. Since it does (without copying anything)
It does? Now I am a bit surprised...
> Where are the details of all the functionality of Numeric
> set out? I have to say that it's a shame there isn't
In the code! ;-)
> better documentation that I can find. Sorry if I missed it
I agree, but until someone volunteers to do a better documentation,
we'll have to live with it. I learned much by experimentation, but
of course I use NumPy in almost everything I do...
> In particular things that aren't immediately obvious to
> me are the reference vs. copy semantics, how indexing/slicing
> works exactly, probably more I haven't run into yet.
Yes, that should be addressed in a tutorial!
> note: fancy indexing is fine if you experts think it
> would be a Good Thing, but remember that Guido
> does not change the syntax lightly, for good reason.
There will be no need to change the syntax!
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From mclay@nist.gov Wed Jul 2 14:26:06 1997
From: mclay@nist.gov (Michael McLay)
Date: Wed, 2 Jul 1997 09:26:06 -0400
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <3.0.1.32.19970701141535.006d53f8@popd.netcruiser>
References: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser>
<Pine.HPP.3.95.970701073416.29412A-100000@laura.llnl.gov>
<3.0.1.32.19970701141535.006d53f8@popd.netcruiser>
Message-ID: <199707021326.JAA31468@fermi.eeel.nist.gov>
Andrew P. Mullhaupt writes:
> I think you're confused about the meaning of this based on your previous
> statements.
>
> As to useful, it is perhaps the single most common case of indexing in
> array languages, and not only that, perhaps the _most efficient_.
>
> Here's an example:
>
> a = reshape(range(12), [3,4])
>
> which looks like
>
> 0 1 2 3
> 4 5 6 7
> 8 9 10 11
>
> then you want
>
> a[[0,2], [2,3]]
>
> to be
>
> 2 3
> 10 11
>
> right? I think a lot of people have been thinking that this would be the
> case (like APL, S, Matlab, etc.).
>
> The question here is actually not _can_ you do this, (I can easily give a
> good implementation of what I outlined so far). But we are not at that
> point yet.
Just for fun I created a trivial implementation. It only works for
2-d arrays and it requires making the index either a list of lists, or
a 2-d array. A C implementation would obviously be needed and it
should be generalized to n-dimensional arrays.
This notation does break the orthogonality in the use of sequences as
indexes, but is that really a problem? This is already true for
dicitonary indexes. Dictionaries can be indexed with tuples, but
dictionaries cannot be indexed with lists. Perhaps for array type
objects it is also resaonable that indexing with different sequence
types can yield different results.
What is the arguement for making array[1,3,4] == array[[1,3,4]]
#!/usr/bin/env python
from Numeric import *
class C:
def __init__(self,arg):
self.data = array(arg)
def __getitem__(self, arg):
if type(arg) == type([]) or type(arg) == type(array([])):
res = []
for x in arg[0]:
res1 = []
res.append(res1)
for y in arg[1]:
res1.append( self.data[x,y])
return res
else:
return self.data[arg]
# this code just sets up the x.data inside the demonstration class.
a3x4 = reshape(arange(12),[3,4])
x = C(a3x4)
print x.data
print "\nUsing a gather syntax"
print x[[[0,2], [2,3]]]
print "\nNnow back to the existing indexing"
print x[2]
print "\nTuples still work as before"
print x[0,2]
a = array([[0,2], [2,3]])
print "\nThe gather syntax could also use arrays as an index"
print x[a]
print "\n using take"
dict = {}
dict [1,3,4 ] = "a tuple as an index"
print "dict indexing isn't orthogonal",
print dict [1,3,4 ]
print "dict indexing with lists doesn't work"
dict [[1,3,4]] = "a list as an index"
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Wed Jul 2 19:37:44 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Wed, 2 Jul 1997 14:37:44 -0400
Subject: [MATRIX-SIG] reverse of take?
Message-ID: <199707021834.OAA14440@dante.mh.lucent.com>
> > > a[i, j, k, hint=flavor_woo_woo_17]
> >
> > I'm reeling about not following this discussion, but I suspect
> > the best approach is for Jim H. to "just do it". I personally
> > begin to think that excess overloading of indexing is not
> > advisable, possibly confusing. I vote for "take()" as is with dual
>
> Until now we haven't discussed much more overloading - just the
> addition of sequence types (including arrays) as indices! And I think
> this kind of operation should be a form of indexing simply because
> people coming from other array languages *expect* it to be.
People keep mentioning other languages to me; I wish I
knew what they were talking about :-). Seriously, I think
people will whine a bit and then get back to work...
OK, my problem with sequence-generalizes-indices is that
it seems to me that
Ar[Ar1][Ar2] = x
either won't work properly or will require a change to what
I understand to be the relatively elegant internal representation
for arrays in NumPy as they exist. I wouldn't recommend
messing up the current implementation for this special case.
The resulting bugs and weird side effects
could be impressive, I think. If I'm wrong (as I think I may
have been regarding "dual of take" which might be fine in Python)
I take it all back.
-- Aaron Watters
checkout
http://www.chipanalyst.com/report/editorials/edit11_6.html
The number of articles like this one keep piling up...
(but he should have said "geek tragedy" rather than
"Greek tragedy")
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Thu Jul 3 01:07:41 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Wed, 02 Jul 1997 20:07:41 -0400
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take"
revisited
In-Reply-To: <199707021137.HAA29599@dante.mh.lucent.com>
Message-ID: <3.0.1.32.19970702200741.006d82a0@popd.netcruiser>
At 07:21 AM 7/2/97 -0400, Aaron Watters wrote:
>would work. Since it does (without copying anything)
>:) a Python implementation of
>"give" or "set" or whatever you want to call it actually might
>be fast enough for practical purposes. I suppose each
>iteration of the loop would require only one header malloc,
>which might be inexpensive enough, and a C implementation
>might not be too much better.
In the one dimensional case, I'd expect a factor of 5-10 or so.
In the multidimensional case, you can do some things which will
make this a factor which increases rapidly with the number of
dimensions.
>Where are the details of all the functionality of Numeric
>set out?
There are some notes from talks by Jim Hugunin on the web.
>note: fancy indexing is fine if you experts think it
> would be a Good Thing, but remember that Guido
> does not change the syntax lightly, for good reason.
That's OK. We don't need to change the syntax per se. We
want to change the semantics.
As to how good it is, you can get an indication of how
useful this sort of thing is by the fact that several
interpreters which had strict limits on indexing (APL
at various stages, Matlab until the most recent version,
Speakeasy) were _forced_ by their user communities to
extend the indexing operation over time. Pascal and C,
in every other sense far better languages than Fortran IV,
didn't put a _dent_ in the scientific computing community
for a _very long time_ because they absolutely sucked for
arrays. (C still does).
> My
> preference would be to be conservative here until
> maybe a clear consensus arrives in many months.
It may very well take months to figure out what everyone
wants, but probably not many months.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From phil@geog.ubc.ca Thu Jul 3 03:29:28 1997
From: phil@geog.ubc.ca (Phil Austin)
Date: Wed, 2 Jul 1997 19:29:28 -0700
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take"
In-Reply-To: <3.0.1.32.19970702200741.006d82a0@popd.netcruiser>
References: <199707021137.HAA29599@dante.mh.lucent.com>
<3.0.1.32.19970702200741.006d82a0@popd.netcruiser>
Message-ID: <199707030229.TAA27421@curlew.geog.ubc.ca>
>>>>> "AM" == Andrew P Mullhaupt <amullhau@ix.netcom.com> writes:
AM> As to how good it is, you can get an indication of how useful
AM> this sort of thing is by the fact that several interpreters
AM> which had strict limits on indexing (APL at various stages,
AM> Matlab until the most recent version, Speakeasy) were _forced_
AM> by their user communities to extend the indexing operation
AM> over time. Pascal and C, in every other sense far better
AM> languages than Fortran IV, didn't put a _dent_ in the
AM> scientific computing community for a _very long time_ because
AM> they absolutely sucked for arrays. (C still does).
And C++ scientific programmers have adopted the Fortran90 Index
notation. Here's part of a demo from Dan Quinlan's A++ array
package--i and j are vectors:
void PDE::Nine_Point_Stencil ( Index & i , Index & j )
{
Solution (i,j) = ( (Mesh_Size * Mesh_Size) * Right_Hand_Side (i,j) +
Solution (i+1,j) + Solution (iFrom owner-matrix-sig@python.org Thu Jul 3 09:42:10 1997
Received: (from majordom@localhost) by python.org (8.7.5/8.7.3)id JAA10638 for matrix-sig-people; Thu, 3 Jul 1997 09:41:56 -0400 (EDT)
Received: from ibs.ibs.fr (ibs.ibs.fr [192.134.36.253]) by python.org (8.7.5/8.7.3) with SMTPid JAA10633 for <matrix-sig@python.org>; Thu, 3 Jul 1997 09:41:53 -0400 (EDT)
Received: from lmspc1.ibs.fr (hinsen@lmspc1.ibs.fr [192.134.36.141]) by ibs.ibs.fr (8.6.12/8.6.12) with ESMTP id PAA26170; Thu, 3 Jul 1997 15:37:42 +0200
Received: (from hinsen@localhost)
by lmspc1.ibs.fr (8.8.5/8.8.5) id PAA07644;
Thu, 3 Jul 1997 15:36:25 +0200
Date: Thu, 3 Jul 1997 15:36:25 +0200
Message-Id: <199707031336.PAA07644@lmspc1.ibs.fr>
From: Konrad Hinsen <hinsen@ibs.ibs.fr>
To: arw@dante.mh.lucent.com
CC: arw@mh.lucent.com, matrix-sig@python.org
In-reply-to: <199707031209.IAA28794@dante.mh.lucent.com>
(arw@dante.mh.lucent.com)
Subject: Re: [MATRIX-SIG] reverse of take?
Sender: owner-matrix-sig@python.org
Precedence: bulk
> Ok, this gets fun. Are we talking about a "pointer array"?
> IE an array of *references* into another array? One could
Right.
> even imagine an array of references into several different
> arrays, but the refcounting issues might get complicated here...
Indeed, and how would you create such an array? By combined indexing
from two arrays? I don't think that makes much sense...
> Actually, maybe an array of references into one other array
> might not look too different from the current implementation
> -- just like the current array implementation, but storing pointers
Exactly. The current array implementation basically knows two kinds
of objects: an "array data" object, which users don' see directly,
and an "array reference" object, which stores a pointer to the data
object plus information on the arrangement of elements (first & last
index plus stride for each dimension). Indexing creates a new reference
object, which refers to the same data object.
So what we are talking about now is an alternative implementation of
the reference object which musts store more than just three numbers
per dimension. There are several possible implementations; I suppose
the easiest is a list of integer indices per dimension, referring to
the array data object.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Thu Jul 3 14:50:09 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Thu, 3 Jul 1997 09:50:09 -0400
Subject: [MATRIX-SIG] reverse of take?
Message-ID: <199707031346.JAA24301@dante.mh.lucent.com>
> So what we are talking about now is an alternative implementation of
> the reference object which musts store more than just three numbers
> per dimension. There are several possible implementations; I suppose
> the easiest is a list of integer indices per dimension, referring to
> the array data object.
I say the easiest and maybe the most efficient is an actual array of
pointers *into* the other array (not sequences of indices). There are
probably nuances I'm missing. -- Aaron Watters
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From P.S.Craig@durham.ac.uk Thu Jul 3 15:33:38 1997
From: P.S.Craig@durham.ac.uk (P.S.Craig@durham.ac.uk)
Date: Thu, 3 Jul 1997 15:33:38 +0100 (BST)
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199707031346.JAA24301@dante.mh.lucent.com> from "Aaron Watters" at Jul 3, 97 09:50:09 am
Message-ID: <12834.199707031433@laplace>
Aaron Watters writes:
>
> > So what we are talking about now is an alternative implementation of
> > the reference object which musts store more than just three numbers
> > per dimension. There are several possible implementations; I suppose
> > the easiest is a list of integer indices per dimension, referring to
> > the array data object.
>
> I say the easiest and maybe the most efficient is an actual array of
> pointers *into* the other array (not sequences of indices). There are
> probably nuances I'm missing. -- Aaron Watters
>
Actually, I suspect both have their merits in different contexts. The
merit of Aaron's proposal is complete flexibility. The merit of
Konrad's, which was also my initial intention, is that we can save
memory and some computing time for such an array which has shape
(100,100,100) since under Konrad's proposal this requires 300 integer
indices and under Aaron's requires 1000000 pointers.
The choice between the two, if we cannot accept having three
implementations of array referencing, depends on the future of
indexing beyond my earlier proposal. The really exotic ideas may well
need Aaron's version though I have to admit I havent thought about
this aspect at all carefully yet.
Peter Craig
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Thu Jul 3 16:22:34 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Thu, 3 Jul 1997 17:22:34 +0200
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199707031346.JAA24301@dante.mh.lucent.com>
(arw@dante.mh.lucent.com)
Message-ID: <199707031522.RAA07980@lmspc1.ibs.fr>
> I say the easiest and maybe the most efficient is an actual array of
> pointers *into* the other array (not sequences of indices). There are
> probably nuances I'm missing. -- Aaron Watters
With pointers you need one per array element, whereas with integers
you need only one per index. In other words, for an array of shape
(10, 20, 30) you need either 10x20x30=6000 pointers or 10+20+30=60
integers. One the other hand, integer indices become a bit more
complex when we allow higer-rank indices, whereas a pointer
implementation remains the same.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Thu Jul 3 16:46:15 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Thu, 3 Jul 1997 11:46:15 -0400
Subject: [MATRIX-SIG] reverse of take?
Message-ID: <199707031542.LAA16461@dante.mh.lucent.com>
----------
> From: Konrad Hinsen <hinsen@ibs.ibs.fr>
> With pointers you need one per array element, whereas with integers
> you need only one per index. In other words, for an array of shape
> (10, 20, 30) you need either 10x20x30=6000 pointers or 10+20+30=60
> integers. One the other hand, integer indices become a bit more
> complex when we allow higer-rank indices, whereas a pointer
> implementation remains the same.
Yes, and in the "circle" or "fractal border" on byte image
graphics example you would not get
much savings in space and also perhaps a speed penalty by saving
indices versus pointers, because the indices would be completely
irregular down to the pixel level. I suppose the
question becomes how common is regularity?
You matrix guys would know better than I. For graphics type operations
I can imagine essentially 2 common cases: rectangular subregions (slices as
they now exist), and more or less complete irregularity. I can't comment on other
possible applications.
-- Aaron Watters
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Thu Jul 3 18:48:18 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Thu, 3 Jul 1997 19:48:18 +0200
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199707031542.LAA16461@dante.mh.lucent.com>
(arw@dante.mh.lucent.com)
Message-ID: <199707031748.TAA08475@lmspc1.ibs.fr>
> Yes, and in the "circle" or "fractal border" on byte image
> graphics example you would not get
> much savings in space and also perhaps a speed penalty by saving
Ehhh... Why? The space requirements depend only on the shape of the
array, not its contents. So the difference is negligible only for
something large x small (e.g. 1000x2).
> indices versus pointers, because the indices would be completely
> irregular down to the pixel level. I suppose the
Ah, I see, you want index a 2d array with a list of index tuples, right?
I hadn't even considered this possibility until now. But I can still come
up with an equivalent integer-index solution that is a superset of the
previously discussed one: for each dimension, you store an n-dimensional
array of integer indices, which more or less correspond to the indices
used in the subscript, except that they are adjusted to point to the
array data object directly.
On the other hand, a pointer-based system has clear speed advantages
if the subarray is accessed more often than it is created.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Thu Jul 3 19:59:51 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Thu, 3 Jul 1997 14:59:51 -0400
Subject: [MATRIX-SIG] reverse of take?
Message-ID: <199707031856.OAA20850@dante.mh.lucent.com>
> On the other hand, a pointer-based system has clear speed advantages
> if the subarray is accessed more often than it is created.
Yes, I imagine a system where a button object, say, is given a matrix
into which to write its representation when it it told to draw itself.
In the ideal case writes into this matrix should propagate all the
way back to the frame buffer. For rectangular buttons, as I understand it,
Numeric slices will do this (or something close) now. For other shapes
it wouldn't. A heck maybe rects are good enough for that.
Also, (changing my mind here) for, say, fonts it would be nice
to share an array of indices rather than making a new pointer
array for each character. I think you've convinced me that
sequences of indices are the *right* way, Konrad. -- Aaron Watters
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Thu Jul 3 23:53:51 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Thu, 03 Jul 1997 18:53:51 -0400
Subject: [MATRIX-SIG] reverse of take?
In-Reply-To: <199707031542.LAA16461@dante.mh.lucent.com>
Message-ID: <3.0.1.32.19970703185351.006d1cec@popd.netcruiser>
At 11:46 AM 7/3/97 -0400, Aaron Watters wrote:
>
>Yes, and in the "circle" or "fractal border" on byte image
>graphics example you would not get
>much savings in space and also perhaps a speed penalty by saving
>indices versus pointers, because the indices would be completely
>irregular down to the pixel level.
You may or may not do better with pointers in this case.
If you are allowed to have pointers to arrays as array elements, then
it is usually not necessary to have separate implementation concerns
for indices and pointers - you can control that at the higher level
language where you have more leverage.
> I suppose the
>question becomes how common is regularity?
>You matrix guys would know better than I. For graphics type operations
>I can imagine essentially 2 common cases: rectangular subregions (slices as
>they now exist), and more or less complete irregularity. I can't comment
on other
>possible applications.
Regularity is by far the most common case. Most non-numerical applications
that use matrices as objects are "dense" matrices.
Your version of 'irregularity' corresponds to one style of "sparse" matrices,
and there are many applications for that too. Sometimes the pointers for
sparse matrices are organized into linked lists for each dimension (e.g. a
list of lists by row and/or by columnn).
There are many "structured" sparse cases, with the most important being
band matrices - which are implemented by embedding them in an index-like
storage.
Now in graphics, (where you seem to be coming from) you find both to be
useful. The dense versions are good for image processing and the sparse
versions can be useful for the more computational geometric things.
Long experience in scientific computing and graphics shows that you really
can't settle for "all indices" vs. "all pointers".
There is a reason for this.
As time goes on, computers get bigger, faster, and cheaper. The problems
that people do on the computers correspond to larger and larger sizes "N".
This means that where in the past, an O(N^2) or O(N^3) dense matrix algorithm
with a small constant becomes less attractive than the O(N) or O(N^2)
algorithm for structured matrices with previously prohibitive constants.
This favors the use of flexible pointer-based structures and algorithms.
By the same token, CPUs are becoming more and more superscalar/parallel,
but memory systems are only keeping up in _size_, they are not keeping up
in speed.
This means that instruction bandwidth is becoming free. Memory bandwidth is
becoming more precious. This means that being control bound is less and
less tolerable - if you can't predict the branches and profit by
speculative execution, then you will not get the most of your machine. This
_strongly_ favors the highly regular algorithms and structures of the dense
representation.
What these trends seem to be doing is making it more important to choose
the correct implementation for the problem, and making it harder to "fake"
one style with the other. APL bit the bullet about ten years ago, with the
advent of what I call "modern" APL - where an array can be an element of an
array, which gives you the pointer style operations APL had previously
lacked. Fortran had to bite this bullet at about the same time with F90,
but going in the other direction.
So a good language is really going to want to go both ways. Maple and
Matlab have slightly different approaches to getting this without requiring
any syntax modification.
Python's object orientation is plenty strong enough to handle this sort of
distinction - you can have a class of dense arrays and a class of sparse
arrays and nobody gets hurt.
However, it makes _zero_ sense to have the common type of 2-way linked list
sparse matrices with the current restriction of indexing to slices, and that's
sort of why we're here.
I also want to put some perspective on what is "esoteric" in indexing.
Everything that has been brought up so far is very tame and you can find it
going back at least ten years in several different successful languages.
It's not like there is a lot of technical risk on the table.
_Esoteric_ is what the post-APL languages are getting into such as K where
there is no distinction between array indexing and function calls, so an
expression such as (and I'm sparing everyone the sight of actual K code)
goo = foo(, , z)
means that
goo(x, y)
is the same as
foo(x, y, z)
The K-speakers call this "currying", which I believe is a term originating
in the lambda calculus and the Lisp community.
So far in this thread nobody is proposing anything which isn't extremely
vanilla as far as array languages go.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From phil@geog.ubc.ca Fri Jul 4 02:34:48 1997
From: phil@geog.ubc.ca (Phil Austin)
Date: Thu, 3 Jul 1997 18:34:48 -0700
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take"
In-Reply-To: <3.0.1.32.19970703024015.006d8698@popd.netcruiser>
References: <3.0.1.32.19970702200741.006d82a0@popd.netcruiser>
<199707021137.HAA29599@dante.mh.lucent.com>
<199707030229.TAA27421@curlew.geog.ubc.ca>
<3.0.1.32.19970703024015.006d8698@popd.netcruiser>
Message-ID: <199707040134.SAA32006@curlew.geog.ubc.ca>
>>>>> "Andrew" == Andrew P Mullhaupt <amullhau@ix.netcom.com> writes:
Andrew> At 07:29 PM 7/2/97 -0700, Phil Austin wrote:
>> And C++ scientific programmers have adopted the Fortran90 Index
>> notation.
Andrew> I find that while the F90 indexing is better than C,
Andrew> Pascal, etc., it's pretty thin soup compared to most array
Andrew> languages. Are they going to improve it for F95?
There's an "enhanced where" (slightly more powerful boolean
constructs, see the appended features list).
The cutting edge is now off wrestling with OO-Fortran.
The glacial calm among the language conservatives in comp.lang.fortran
make it a very restful place to visit when the tumult in some of the
other newsgroups is raising your blood pressure.
* Enhanced WHERE construct [with nested WHERE, masked ELSEWHERE, and construct
names]
* The FORALL statement and construct
* PURE user-defined procedures
* ELEMENTAL user-defined procedures
* Pointer initialization [i.e., the NULL( ) intrinsic]
* Derived-type structure default initialization
* Automatic deallocation of allocatable arrays [at exit from a procedure]
* CPU_TIME intrinsic
* Enhanced CEILING and FLOOR intrinsic functions [with KIND argument]
* Enhanced MAXLOC and MINLOC intrinsic functions [with DIM argument]
* Enhanced SIGN intrinsic function [can detect negative real zero]
* Printing of -0.0
* Generic identifier allowed in END INTERFACE statement
* Zero-length formats [I, B, O, Z, F expands field width to fit data]
* Comments allowed in namelist input
* New obsolescent features [as specified in F95]
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Fri Jul 4 13:40:47 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Fri, 4 Jul 1997 14:40:47 +0200
Subject: [MATRIX-SIG] Data filtering
In-Reply-To: <3.0.1.32.19970701131914.006d9be0@popd.netcruiser>
(amullhau@ix.netcom.com)
Message-ID: <199707041240.OAA11383@lmspc1.ibs.fr>
> I haven't done this stuff in python, but I have quite a bit of stuff
> which would be very easy for me to put into python and give to you. The
> question is; can you be more specific about the filter you need? Is it
> single or multichannel? Do you have issues with phase as well as magnitude?
> Is the segmentation requirement for efficiency or an actual requirement?
For my personal needs, any low-pass filter for real data is good enough.
Segmentation would be nice for efficiency, but is not essential. But
I would like to encourage work on a general-purpose signal processing
library to go with NumPy. This is probably as important in practice as
the linear algebra library, although for different users.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From phil@geog.ubc.ca Sat Jul 5 03:48:31 1997
From: phil@geog.ubc.ca (Phil Austin)
Date: Fri, 4 Jul 1997 19:48:31 -0700
Subject: [MATRIX-SIG] ZPL array language
In-Reply-To: <3.0.1.32.19970703185351.006d1cec@popd.netcruiser>
References: <199707031542.LAA16461@dante.mh.lucent.com>
<3.0.1.32.19970703185351.006d1cec@popd.netcruiser>
Message-ID: <199707050248.TAA04681@curlew.geog.ubc.ca>
>>>>> "Andrew" == Andrew P Mullhaupt <amullhau@ix.netcom.com> writes:
Andrew> _Esoteric_ is what the post-APL languages are getting into
Andrew> such as K where there is no distinction between array
Andrew> indexing and function calls, so an expression such as (and
Andrew> I'm sparing everyone the sight of actual K code)
I'm not sure how esoteric it is, but a group at the University of
Washington has just preempted the end of the alphabet with ZPL--an
array language for parallel processors. They have source and sample
code at:
http://www.cs.washington.edu/research/zpl/
Phil
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Sun Jul 6 21:26:04 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Sun, 6 Jul 1997 16:26:04 -0400 (EDT)
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited
In-Reply-To: <199707021826.UAA04487@lmspc1.ibs.fr>
Message-ID: <Pine.SGI.3.95q.970706161221.22626B-100000@maigret>
On Wed, 2 Jul 1997, Konrad Hinsen wrote:
> > Where are the details of all the functionality of Numeric
> > set out? I have to say that it's a shame there isn't
> > better documentation that I can find. Sorry if I missed it
>
> I agree, but until someone volunteers to do a better documentation,
> we'll have to live with it. I learned much by experimentation, but
> of course I use NumPy in almost everything I do...
>
> > In particular things that aren't immediately obvious to
> > me are the reference vs. copy semantics, how indexing/slicing
> > works exactly, probably more I haven't run into yet.
>
> Yes, that should be addressed in a tutorial!
I'll add these to the TODO list for the tutorial, but don't
count on me getting too much done in the near future (see
http://maigret.cog.brown.edu/hugo/hugo14.html
for the reason why =).
--da
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hyoon@nyptsrv1.etsd.ml.com Tue Jul 8 15:18:43 1997
From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant)
Date: Tue, 8 Jul 1997 10:18:43 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
Message-ID: <199707081418.KAA05563@ptq3.etsd>
Hello fellow snake handlers,
I am attempting to do some Trading (stocks) related number crunching
in Python. I am having quite a difficulty trying to implement my old
Gauss logic to Python. It should be dealt in a different way; however,
my mind is still in the old 2D complete matrix stage that I am finding
it difficult to find what I wanna to do in Python.
=20
# I have some x period of info into Python lists/arrays. Of course as
of any financial data there are missing values. So, I stored them as
None, thinking that None is closest to msg values, but, as some of the
readers would have realized, none of the nonzero, choose, equal, =
where, etc...
can be performed on array or list with None values. I get:
TypeError: function not supported for these types, and can't=20
coerceto supported types
Now some might say why not just use either zeros or negative values. =
Well,
if I am just dealing with price and vols that's okay, but I am also =
dealing
with other info like earnings, which can be easily negative.
There is at least distinction in mathmatics/databases about missing, =
undefined,
and not available. This is very thorny issue in numerical computing. =
Is msg =3D=3D
msg? (This is undefined) What's truly missing vs. undefined? This is =
quite a
issue. For people brave enogh, I encourage Joe Celko's articles in =
DBMS with
same title, which can be obtained through DBMS. He really does good =
job of=20
explaining the issues. There are other issues like +&- infinity NA and =
so on,
which Joe does not get into, but overall the article serves as good =
introduction
to the concepts.
May be all this issues have been considered and Python has way to =
handle all this,
but I could not find it. I would really like to see find_msg & =
index_msg find of
functions. The most often used way to deal with missing values are :
a. Just copy over from last period or do some estimate based on =
others, which is
bit dangerous due to introduction of serial correlation.
b. Get rid of the entire row of the observation. Which means throw =
the whole stock
or the whole day from the model. Less dangerous, but could be =
problematic, espcially
if you don't have too many observation to play with.
=20
find_msg should return matrix of same dimention filled with 0,1. =
index_msg should
only work on 1 dim and give me back index to an array. I can use =
find_msg for take
kind of function, or rever of take.
=20
=04 Hopefully I can do this without loops!
Thanks for reading,
Hoon,
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Tue Jul 8 16:52:41 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Tue, 8 Jul 1997 11:52:41 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
Message-ID: <199707081549.LAA06143@dante.mh.lucent.com>
Hey! Null values! I had a whole chapter in my
dissertation regarding nulls!
There are many kinds of nulls:
overdefined (computation error)
|
[normal non-null values here] ...
|
unknown/applicable inapplicable
\ /
unknown/maybe-inapplicable
How a query engine or computation should handle this
depends on which one you mean. However, in general
I think if you want to "layer" null values into a matrix you
might want to use an "indicator mask" which identifies the
null/non-null values as a separate array.
Let me know if this makes sense
----------
From: Hoon Yoon - IPT Quant <hyoon@nyptsrv1.etsd.ml.com>
To: matrix-sig@python.org
Subject: [MATRIX-SIG] Much ado about nothingness.
Date: Tuesday, July 08, 1997 10:18 AM
Hello fellow snake handlers,
I am attempting to do some Trading (stocks) related number crunching
in Python. I am having quite a difficulty trying to implement my old
Gauss logic to Python. It should be dealt in a different way; however,
my mind is still in the old 2D complete matrix stage that I am finding
it difficult to find what I wanna to do in Python.
# I have some x period of info into Python lists/arrays. Of course as
of any financial data there are missing values. So, I stored them as
None, thinking that None is closest to msg values, but, as some of the
readers would have realized, none of the nonzero, choose, equal, where, etc...
can be performed on array or list with None values. I get:
TypeError: function not supported for these types, and can't
coerceto supported types
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hyoon@nyptsrv1.etsd.ml.com Tue Jul 8 17:28:31 1997
From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant)
Date: Tue, 8 Jul 1997 12:28:31 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
Message-ID: <199707081628.MAA05660@ptq3.etsd>
Aaron,
This is what I love about working in Python. I had pretty hard time
making vendors understand why different nothingness mattered so much.
Now I am actually getting educated here.
I don't think I am stretching too much, when I say there isn't a good
stat package or DBMS out there that handles missings sufficiently.
For my computational needs, I think a mask of nulls will do nicely.
Going back to how I am used to handling it.
Gauss has something called packer, which packs away any rows that
contains missing value, which is a (.) dot in Gauss. In Python this
could be much more generalized. I can get index of non missing or
missing by row or column or N-th dim. And do either take or reverse
of take. This is bit costly on memory, however. I am actually creating
equal dimension (even at at bits) of 1's and zeros, which is not too
bad. I also noticed that whenver I used None in my array, I get a complex
array. Nothing should be uniform and should not be this expensive.
For backward filling, this is always has been tricky business. If I
have one time value missing, I can use the above 1/0 array move the
original array by 1 dimension backward and mult the result, but what
do I do if I have multiple period of missings? I guess a repeat is
in order untill I get "None in Target_array" is 0. Of course I am working
under premise that the first row of N-th dimention is not itself has
missing. That could be really unfun.
Not to make too much fun of Gauss, but msg==msg was 1. None of the
stat package I know has more than one kind of nothingness for input.
I guess this means that Python does not have naitive Null handler
upto now? I wouldn't be terribly surprised, but I think it needs one.
I am still getting confused about the N-th dimension thing. I am sure
it's good thing, but confused still.
How did everyone dealt with this before?
Thank you for the reply,
Hoon,
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Tue Jul 8 18:47:59 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Tue, 8 Jul 1997 13:47:59 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
Message-ID: <199707081744.NAA26463@dante.mh.lucent.com>
I'm not sure I'm following you. Maybe a simple example would
help. What do you want to do and what is a slow way to do it
(using lots of python loops if necessary). Once we see that
someone (likely not me) can suggest how to speed it up using
Numeric (and maybe altering the representation).
Guessing:
Suppose you have a 2d array of prices for securities by time with another 2d array
of same shape that indicate which price listings are invalid/unknown.
Suppose you wish to apply the heuristic of copying the youngest price
into the unknown positions, and further suppose the prices at time 0
are all known.
It seems to me you could pass a "price vector" across the time dimension,
picking out new valid prices and masking in most recent valid prices to
the source array for null positions. My skills with numeric are such
that I'd have to work on this for a while to get it, so I'll let you think about
it instead. It probably would amount to about 10 lines if that of NumPy.
It's likely that I suppose wrong however, and I could be off base in
any number of other ways. --Aaron Watters
----------
> From: Hoon Yoon - IPT Quant <hyoon@nyptsrv1.etsd.ml.com>
> To: matrix-sig@python.org; arw@mh.lucent.com
> Subject: Re: [MATRIX-SIG] Much ado about nothingness.
> Date: Tuesday, July 08, 1997 12:28 PM
>
> Aaron,
>
> This is what I love about working in Python. I had pretty hard time
> making vendors understand why different nothingness mattered so much.
> Now I am actually getting educated here.
>
> I don't think I am stretching too much, when I say there isn't a good
> stat package or DBMS out there that handles missings sufficiently.
> For my computational needs, I think a mask of nulls will do nicely.
>
> Going back to how I am used to handling it.
> Gauss has something called packer, which packs away any rows that
> contains missing value, which is a (.) dot in Gauss. In Python this
> could be much more generalized. I can get index of non missing or
> missing by row or column or N-th dim. And do either take or reverse
> of take. This is bit costly on memory, however. I am actually creating
> equal dimension (even at at bits) of 1's and zeros, which is not too
> bad. I also noticed that whenver I used None in my array, I get a complex
> array. Nothing should be uniform and should not be this expensive.
> For backward filling, this is always has been tricky business. If I
> have one time value missing, I can use the above 1/0 array move the
> original array by 1 dimension backward and mult the result, but what
> do I do if I have multiple period of missings? I guess a repeat is
> in order untill I get "None in Target_array" is 0. Of course I am working
> under premise that the first row of N-th dimention is not itself has
> missing. That could be really unfun.
> Not to make too much fun of Gauss, but msg==msg was 1. None of the
> stat package I know has more than one kind of nothingness for input.
>
> I guess this means that Python does not have naitive Null handler
> upto now? I wouldn't be terribly surprised, but I think it needs one.
> I am still getting confused about the N-th dimension thing. I am sure
> it's good thing, but confused still.
>
> How did everyone dealt with this before?
> Thank you for the reply,
>
> Hoon,
>
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hyoon@nyptsrv1.etsd.ml.com Tue Jul 8 20:04:21 1997
From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant)
Date: Tue, 8 Jul 1997 15:04:21 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
Message-ID: <199707081904.PAA05746@ptq3.etsd>
Aaron and MSIG:
I am not quite that concerned about the speed of the following code.
It runs in about 2sec on my Ultra Sparc. I got a lot of loops in
there to check the data. This is fine for 20 or 30 stocks, but
any more would prove problematic.
What I have much more concern for is that I am not happy that
Null, missing, or None values are foreign concepts to NumPy. An
array time should not be complex. And there should be much easier
way to pull matching and NOT criteria. Reverse of Take would be
very nice indeed. I guess I am griping, because I have to get whole
new thought process going here to get what I could do in 20 minutes
in Gauss rather than 2 hours. Still, I think there should be a more
elegent ways to handle Null data.
About 80% modeling is dealing with data. Hardest to deal with is
the issue with missing data. I should not continously use loops to
deal with this in a complex matrix. There should be a set missing
code that does not effect the matrix to default to complex. Pretty
much all other grips about handling could be classed away over time
in an elegent manner; however, I cannot change the fact that too many
operators like equal, where, etc... neglects to deal with Nulls.
I should be able to nonzero(equal(a, None)) to get index of Nones.
And most operation should be programmed with missing data poitn in
mind.
To credit the Python, I must say even with loops, the ideas are much
more readable than anything else I have ever written. Given how wonderful
Python has been so far, I am amazed that None/missing has not been addressed
in NumPy.
Thanks Aaron, I hope the following code generally give you sense of
80% of my work. There are few more places, I would have dropped loops
like > .80 could have been greater, the holidays could have been chopped
away at one shot, etc...
Hoon,
p.s.: Despite my nagging, I won't trade Python for anything at this point.
I just wanna see it improved to the point that I won't have to learn any
other language. I would like to better business graphics, like GNUplot tied
to this and better statistics (I am hoping to tie in something else if
necessary), and this will be perfect for my purposes.
*****************************************************************
import shelve
from dateproc import *
from Numeric import *
d = shelve.open('p_v_rum')
dates = d['i_dates'] # get dates that keys this time series
dates.sort()
trd_d = dates[:] # This will have all holidays removed later
holi = d['i_holi'] # get all available holidays
holi_i = []
for hday in holi.keys(): # getting rid of holidys
if len(holi[hday]) < (len(d.keys())*0.8): continue # At least 80%
stocks msg
try: hsx = dates.index(hday) # if > 80%: see if it's available.
except ValueError: continue
else:
holi_i.append(hsx) # holiday indexes to be used on
retrv.
del trd_d[hsx] # Get rid holidays leaving only
Tradins
tgt_date = chkdate('19970422') # This my Target date
try: dsx = trd_d.index(tgt_date) # See if the date is one of index Trd
dates
except ValueError: dsx = add.reduce(less(trd_d, tgt_date)) # if not goto next
date
dte_fwd = 6 # I need 5 dates forward prices
dte_bwd = 5 # I need backward as well
trd_date = trd_d[dsx-dte_bwd:dsx+dte_fwd] # get Trading Range
st_d = dates.index(trd_date[0]) # get date range, since that's how it's
stored
ed_d = dates.index(trd_date[-1])
h_sx = nonzero(less_equal(holi_i,ed_d)*greater_equal(holi_i,st_d))
# find Holidays Index inside the range, I will use to del later
stocks = ['ITX','IBM','AAPL'] # my ticker list
result = []
for stk in stocks:
iadj = dates.index(d[stk]['began']) # Time Series began, so need
to adj.
close = d[stk]['close'] # get closing prices
retrv = close[st_d-iadj:ed_d-iadj+1] # get only section interested
in
for hidx in h_sx: # getrid of all holidays in
retrv
del retrv[hidx]
if retrv[0] == None: # if first one is None
bk=0
while st_d-iadj > iadj: # go back in time to get
last know P
bk=bk+1
if close[st_d-iadj-bk] != None:
retrv[0] = close[st_d-iadj-bk]
break
while None in retrv: # if there are None values
try: Nsx = retrv.index(None)
except ValueError: break # no more None's done!
else: retrv[Nsx] = retrv[Nsx-1] # back fill using old
prices
if len(retrv) == len(trd_date):
result.append(retrv)
else:
print 'Error Wrong size!'
break
result = array(result) # turn it into a Numeric array!
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From motteler@laura.llnl.gov Tue Jul 8 20:50:23 1997
From: motteler@laura.llnl.gov (Zane C. Motteler)
Date: Tue, 8 Jul 1997 12:50:23 -0700 (PDT)
Subject: [MATRIX-SIG] Much ado about nothingness.
In-Reply-To: <199707081904.PAA05746@ptq3.etsd>
Message-ID: <Pine.HPP.3.95.970708124847.28397B-100000@laura.llnl.gov>
Hi Hoon,
I've been following your correspondence with Aaron. It seems to me that
you are missing an important concept in criticizing NumPy (and other
Python functions) for not dealing with None.
The primary purpose in creating Numerical arrays is to have a PyObject
containing contiguous numerical data that can be rapidly processed in
compiled code. The elements of a NumPy array must ipso facto all be
numbers of the same type (e. g., real, integer, etc.). None, however, is
not a number of any type but rather is a PyObject. This it seems as if
you are asking for NumPy arrays which may contain elements of different
types. But if you want arrays which contain objects of different types,
then a Python list is about all we have available, and then of course
you don't have handy matrix operations, etc.
It seems to me that if you want array-like objects that can contain
PyObjects as well as numbers, and yet can somehow meaningfully have
array functions and operations applied to them, then you need your own
Python extension defining a new type. You can write your own or hire
somebody to do it.... ;-} (I could, I think it would be an interesting
exercise, but I don't have the time....)
NumPy was designed just for one purpose, namely to do fast numerical
calculations with non-sparse arrays containing objects of just one
numerical type. I think what you are asking for is a horse of a
different color.
I hope that I have helped to clarify matters rather than muddy the
Watters. (Oops, pardon the pun, Aaron.)
Cheers
Zane
-----------------------------------------------------------------------------
Zane C. Motteler, Ph. D. ___________ALSO:_________________
Computer Scientist | Professor Emeritus of Computer|
Lawrence Livermore National Laboratory | Science and Engineering |
P O Box 808, L-472 | California Polytechnic State |
Livermore, CA 94551-9900 | University, San Luis Obispo |
510/423-2143 ---------------------------------
FAX 510/423-9969
zcm@llnl.gov or motteler@laura.llnl.gov or zmottel@calpoly.edu
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hyoon@nyptsrv1.etsd.ml.com Tue Jul 8 21:23:33 1997
From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant)
Date: Tue, 8 Jul 1997 16:23:33 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
Message-ID: <199707082023.QAA05787@ptq3.etsd>
Hello Prof,
Well, I think what you just illustrated was difference between
a Physical scientist and Social scientist(if one could call it that).
Whether it's econometrics, taking poles on election, or analyzing
stock trends, we take missing data as a matter of fact. There is no
control over what tests we can perform and data is always full of=20
holes. No experiment or model is really controlled. So, there goes
80% of our time.
I freely admit that both of you are much more capable with systems=04=04=
and especially at Python; however, when it comes to financial data,
I think I can claim some expertise. And every statistical package
designed for social science have heavy weight on how to deal with
missing data. And I know this is the biggest concerns for financial
modeling, more so than anything else I know. How else would I get to
know all this different missing types? I am an Econ major, who only
took 2 computer classes (but I still think I am right).
Oh, about our earlier correspondence about Narcisse. I meant to =
purchase
Narcisse package, if license could be purchased. Being a Berkeley =
graduate,
I have great respect for Triple L. I would not presume to just hire one =
on
demand. Besides getting consulting contract around here is HELL! Thanks=20
for the thought, however.
I still think missing values should be part of the matrix. It
should probably not None, since as you have mentioned that it introduces
additional Python object inside the array. Something little more
elegent would be nice. I guess I could process the info, prior to
putting into the array, but I would much rather deal with a matrix
manipulation.
Thanks for your thought, I really appreciate your input.
Hoon,
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hugunin@javanet.com Tue Jul 8 21:43:26 1997
From: hugunin@javanet.com (Jim Hugunin)
Date: Tue, 08 Jul 1997 16:43:26 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
References: <199707082023.QAA05787@ptq3.etsd>
Message-ID: <33C2A66E.7D2D192C@javanet.com>
I think that Zane did a great job of summarizing the intentions of NumPy
to provide efficient dense raw numeric arrays, not to provide each
particular brand of array that anyone might want.
In your particular case what you need to do is write a small Python
class that wraps a numeric array and provides the extra functionality
you need for missing data. Internally it would hold two arrays, one of
the actual data, and another one of ones and zeros indicating which data
was valid (just like Aaron suggested).
If you want to use NumPy on arrays of missing data you will need to
write this Python class yourself. The point of NumPy is to make sure
that people have the efficient underlying array objects that they need
to build these higher level abstractions. If you don't have the
time/inclination to build this small wrapper class yourself, then NumPy
is the wrong package for you. If you do, perhaps you could contribute
your code back to the community so that more people can benefit from it.
-Jim
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From phil@geog.ubc.ca Tue Jul 8 21:52:31 1997
From: phil@geog.ubc.ca (Phil Austin)
Date: Tue, 8 Jul 1997 13:52:31 -0700
Subject: [MATRIX-SIG] Much ado about nothingness.
In-Reply-To: <199707082023.QAA05787@ptq3.etsd>
References: <199707082023.QAA05787@ptq3.etsd>
Message-ID: <199707082052.NAA22223@curlew.geog.ubc.ca>
>>>>> "Hoon" == Hoon Yoon <hyoon@nyptsrv1.etsd.ml.com> writes:
Hoon> I still think missing
Hoon> values should be part of the matrix. It should probably not
Hoon> None, since as you have mentioned that it introduces
Hoon> additional Python object inside the array. Something little
Hoon> more elegent would be nice. I guess I could process the
Hoon> info, prior to putting into the array, but I would much
Hoon> rather deal with a matrix manipulation. Thanks for your
Hoon> thought, I really appreciate your input.
There was some discussion on this list about a year ago concerning
standard IEEE NaN's (this undelies Splus' null data handler I think).
Concerns were expressed about performance and portability, but if
those issues were manageable then there's an obvious need for Python NaN
capability in signal/image processing applications.
Phil
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hyoon@nyptsrv1.etsd.ml.com Wed Jul 9 15:01:08 1997
From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant)
Date: Wed, 9 Jul 1997 10:01:08 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
Message-ID: <199707091401.KAA06541@ptq3.etsd>
Gentlemen:
I know some people are bit frustrated with this. And my earlier posts
have been some what compremised by my lack of underlying mechanics.
First of all, I added None Python object to my array not because I
thought adding this object was appropriate, but this was about the most
closes
All the other stuff like filling in the last known value,
'hot-record-replacement', statistical inference via EM algorithm, robust
pseudo-values,
etc., are useful in their place and _mutually contradictory_.
t thing to NaN or missing in Gauss, Ox, and Splus (my matrix langs)
that I could find. I actually thought it was a bad idea, because it changed
my matrix to complex and it should have been basic Int type. Cost of this
overhead was clearly high.
As Andrew mentioned:
All the other stuff like filling in the last known value,
'hot-record-replacement', statistical inference via EM algorithm, robust
pseudo-values,
etc., are useful in their place and _mutually contradictory_.
(snip)
How I would deal with missing data once identified is completely depended
on kind of modeling I do. I was illustrating some basic methodology that I
would use this missing identifier. Once missing has been specified, I should
know what to do with it. As I have said no-package out there deals with msgs
well enough, but also there is no stat package out there without a missing
identifier inside an array that I know of. (I don't know mathlab, but above
mentioned plus SAS has one.) As everyone here realizes there are a lot of
missings and some can be easily controlled before it goes in. Some can't be
deal with untill it screws up your model. A very large number, often created
by dividing by near zero often creeps up. To generate this number you just
need to do a operation on two seeming innoncent numbers.
Missing in other package is an error code, which can be operated on but
simply returns missing error code instead of blowing up the program.
msg > 1 ? = msg
msg==msg ? = msg
msg+1 = msg
msg/1 = msg
1/msg = msg
add.reduce([1, 3, 4, msg, 4]) = msg
(If I want the answer, msg should either be taken out or turn to some other
number. As may said, this should be done prior, but too often how this is
done is completely conditional on other matrix and I also have to update
other matricies as well).
Any operation done on msg will turn up msg, because it's an error condition.
And this goes back to garbage collection and ease of developemnt issues.
I am not sure [0,1] shadow matrix could deal with this adaquetly. I guess
I still have not thought and know about these methods.
I got 256MB on my Sparc and Pentium, the reason I have it is because
financial data especially realtime is extremely vast with a lot of missing
data points.
I hope some people are not suggesting that NumPy is for computer scientist
types only and whatever I should find for my use I have to build one. As I
have said, every matrix language I ever used of looked had missing. If someone
told me that particular package does not have a some specific statistical
method like Logit, I would understand and build my own, but missing is way
too low level for someone like me.
Frankly, I would love to pay someone to do this; however, lack of organized
structure in Python is problem. I believe Python is a wonderful language and
due to its free and open nature, I would like to be able to pay someone to
get this going and contribute it back to the community. This would benefit
both my employer and community quite well. Merrill, however, will not deal
with individuals and have hard time dealing with smalll operators off the
Web. I had Ox developer contacted and we agreed on a very nice fat support
contract recently only to have it turned down when the purchasing called
the good professor and found out that he works by himself in his office.
Never mind the fact that Ox is extremely fast and the developer was
incredibly helpful and only reason we agreed on a nice support contract
was because I could not get a consulting contract. The purchasing basically
said we do not deal with individuals, look for something else. I would love
to have someone come up with a support company for Python with some
affilliation to some established entity. Pay the person to added needed
functionality and contribute general purpose codes back to commnunity. I
can reasonably do this every beginning of fiscal year on a contracted
support. Any takers? ;)
I guess without the missing issues settled, I will probably need to
link in some other package for analytics and graphics to Python if
possible and only use NumPy for Derivative Calcs, where the inputs are
less uncertain than Portfolio Trading. It's not a perfect solution, but
nothing is so uncertain.
Hoon,
P.S.: I still welcome any more input to putting NA as an additinal class.
I don't think I am really capable yet, but I would like to contribute
something back if I can. Oh, my earlier posts about using Gauss like
operators in Python is clearly whining. I should just get used to my new
environment, but, unless NumPy is not for social science and financial
modeling, I believe I have justifiable complaint given so many other
languages I've used in past.
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Wed Jul 9 15:42:55 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Wed, 9 Jul 1997 10:42:55 -0400 (EDT)
Subject: [MATRIX-SIG] Much ado about nothingness.
In-Reply-To: <199707091401.KAA06541@ptq3.etsd>
Message-ID: <Pine.SGI.3.95q.970709103827.26688F-100000@maigret>
On Wed, 9 Jul 1997, Hoon Yoon - IPT Quant wrote:
[ a lot of stuff ]
> I am not sure [0,1] shadow matrix could deal with this adaquetly.
I believe most of us believe that it can. The reason no one has done it
yet (that I know) is that 1) we don't need it for our 'day jobs', and 2)
our TODO lists are big. =)
> Frankly, I would love to pay someone to do this; however, lack of
> organized structure in Python is problem. I believe Python is a
> wonderful language and due to its free and open nature, I would like to
> be able to pay someone to get this going and contribute it back to the
> community. This would benefit both my employer and community quite well.
> Merrill, however, will not deal with individuals and have hard time
> dealing with smalll operators off the Web.
This situation is recognized, and will change.
--david
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From phil@geog.ubc.ca Wed Jul 9 17:34:23 1997
From: phil@geog.ubc.ca (Phil Austin)
Date: 09 Jul 1997 09:34:23 -0700
Subject: [MATRIX-SIG] namespaces, constructors, and global variables
Message-ID: <m3bu4cgp68.fsf@curlew.geog.ubc.ca>
The following message is a courtesy copy of an article
that has been posted as well.
We're working on a generic plotting interface to gnuplot, gist, ncar
graphics, etc., and are currently stumped by our lack of understanding
of namespaces. The interface puts a plot together by creating a
series of objects (plots, axes, ticks, legends, etc.) with methods to
add or modify fonts, colors, line widths, whatever. The state of the
plot is stored in a global dictionary, and an execute method goes
through the dictionary and builds the plot for the appropriate
software package. An example:
>>> from PlotAll import *
>>> plotIt=PlotDriver(device="screen")
>>> plot1=BuildPlot(time, xarray)
>>> leftAxis=BuildAxis(plot1,axis="left",label="xarray")
>>> print Master['plot1']['leftAxis']['label']
xarray
Our problem is that we would like a botAxis and a leftAxis object to
be created by default when a plot object is instantiated without the
argument "axes=False". I'm appending a version of PlotAll.py, which
does this without a hitch from within the interpreter:
>>> import PlotAll
>>> PlotAll.runit()
Master.keys(): ['plotCount', 'device', 'plot1']
inside BuildAxis: Master.keys()= ['plotCount', 'device', 'plot1']
inside BuildAxis: Master.keys()= ['plotCount', 'device', 'plot1']
Bottom: Master['plot1']['leftAxis']['label']= ytest
>>> leftAxis
<BuildAxis instance at 820e290>
Running from the shell, however, doesn't work, because the global
variable Master doesn't appear to have it's keys:
<phil@curlew: src.dir/python.dir> python PlotAll.py
Master.keys(): ['plotCount', 'device', 'plot1']
inside BuildAxis: Master.keys()= []
Can someone walk me through why this is failing, and how I can fix it?
Thanks, Phil
______________________________PlotAll.py_______________________________
Master={}
global Master
class PlotDriver:
global Master
def __init__(self,device="postscript"):
self.RCSId="$Id: PlotAll.py,v 1.5 1997/07/09 16:31:05 phil Exp $"
Master['plotCount']=0
Master['device']=device
class BuildPlot:
global Master
def __init__(self,xdata,ydata,plotType="linePlot",axes="default"):
Master['plotCount'] = Master['plotCount'] + 1
self.plotName="plot%d" % Master['plotCount']
Master[self.plotName]={}
Master[self.plotName]['xdata']=xdata
Master[self.plotName]['ydata']=ydata
Master[self.plotName]['plotType']=plotType
if(axes=="default"):
import __main__
plot=self
global Master
ns=vars()
print "Master.keys(): ", Master.keys()
code=["from PlotAll import *",
"global Master",
"leftAxis=BuildAxis(plot,axis=\"left\",label=\"ytest\")",
"botAxis=BuildAxis(plot,axis=\"bot\",label=\"x\")"]
for stmt in code:
exec stmt in ns,__main__.__dict__
class BuildAxis:
global Master
def __init__(self,Plot,axis="left",label="default",limit="default"):
global Master
print "inside BuildAxis: Master.keys()=", Master.keys()
self.plotName=Plot.plotName
self.axisName="%sAxis" % axis
Master[self.plotName][self.axisName]={}
Master[self.plotName][self.axisName]['label']=label
Master[self.plotName][self.axisName]['limit']=limit
def runit():
global Master
xarray=[1,2,3,4]
time=[5,6,7,8]
#controls items common to all plots
plotIt=PlotDriver(device="screen")
#build an individual plot
plot=BuildPlot(time, xarray)
print "Bottom: Master['plot1']['leftAxis']['label']=", Master['plot1']['leftAxis']['label']
if __name__ == "__main__":runit()
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From ted_horst@il.us.swissbank.com Wed Jul 9 19:15:41 1997
From: ted_horst@il.us.swissbank.com (Ted Horst)
Date: Wed, 9 Jul 97 13:15:41 -0500
Subject: [MATRIX-SIG] Much ado about nothingness.
References: <199707091401.KAA06541@ptq3.etsd>
Message-ID: <9707091815.AA18826@ch1d162nwk>
Has anybody looked at the Missing type from DigiCool ? It has the desired
behavior for arithmetic operations, but doesn't work correctly for math or
umath functions. I have no idea how hard this would be to get working, or even
if this is the proper approach, but somebody else is at least looking at the
problem. You can get it from http://www.digicool.com/releases/ExtensionClass/.
Comments ?
Ted Horst
(not speaking for SwissBank)
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From ted_horst@il.us.swissbank.com Wed Jul 9 19:15:41 1997
From: ted_horst@il.us.swissbank.com (Ted Horst)
Date: Wed, 9 Jul 97 13:15:41 -0500
Subject: [MATRIX-SIG] Much ado about nothingness.
References: <199707091401.KAA06541@ptq3.etsd>
Message-ID: <9707091815.AA18826@ch1d162nwk>
Has anybody looked at the Missing type from DigiCool ? It has the desired
behavior for arithmetic operations, but doesn't work correctly for math or
umath functions. I have no idea how hard this would be to get working, or even
if this is the proper approach, but somebody else is at least looking at the
problem. You can get it from http://www.digicool.com/releases/ExtensionClass/.
Comments ?
Ted Horst
(not speaking for SwissBank)
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From phil@geog.ubc.ca Wed Jul 9 19:37:28 1997
From: phil@geog.ubc.ca (Phil Austin)
Date: 09 Jul 1997 11:37:28 -0700
Subject: [MATRIX-SIG] making a text label from an object instance?
Message-ID: <m390zggjh3.fsf@curlew.geog.ubc.ca>
The following message is a courtesy copy of an article
that has been posted as well.
For my plotting package I would like to have the default axes labels
be the names of the objects being plotted. For example, if the
command is:
plot(time[:,4],vel[:,3,2])
I would like to generate strings containing "time[:,4]" and
"vel[:,3,2]". Is this possible with Python?
For those who know Splus, I'm looking for the equivalent of
deparse(substitute(time[:,4]))
Regards, Phil
--
------------------------------------------------------------
Phil Austin (phil@geog.ubc.ca)
Department of Geography, Tel: (604) 822-2663
University of British Columbia, B.C. Fax: (604) 822-6150
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From jim.fulton@digicool.com Wed Jul 9 20:57:39 1997
From: jim.fulton@digicool.com (Jim Fulton)
Date: Wed, 09 Jul 1997 15:57:39 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
References: <199707091401.KAA06541@ptq3.etsd> <9707091815.AA18826@ch1d162nwk>
Message-ID: <33C3E5D9.46F5@digicool.com>
Ted Horst wrote:
>
> Has anybody looked at the Missing type from DigiCool ? It has the desired
> behavior for arithmetic operations, but doesn't work correctly for math or
> umath functions. I have no idea how hard this would be to get working, or even
> if this is the proper approach, but somebody else is at least looking at the
> problem. You can get it from http://www.digicool.com/releases/ExtensionClass/.
Thankyou for mentioning this. I guess this is the nudge I need to add a
couple of
$0.01. In part, the Missing module in ExtensionClass is intended to
fulfill a promise I made on this list too long ago.
When I used to work at USGS, I had to deal with missing values alot.
For my needs, I considered any database or data analysis tool, including
ones I wrote, that didn't make provision for missing data to be almost
useless. The way I've seen this dealt with is to reserve some special
values (not necessarily IEEE standard ones), as missing values. One
package I dealt with (P-Stat) actually had three special vallues,
Missing1 (123456e20), Missing2 (123457e20), and Missing3 (123458e20),
that could be used to represent different kinds of missing data. For
example, this package used Missing3 for things like log(0). In any
case, all computations had to check for missing values. While this adds
significantly to the complexity of numerical software, and slightly
reduces it's efficiency, I think the gain is well worth the price.
I think that it would be a good thing for NumPy to reserve one or more
special floating point missing values internally and should expose these
in Python using my (truly trivial) Missing module or something like it.
I realize that this would be a significant undertaking, although perhaps
there is
available funding for it.
I feel odd making this suggestion, given that, in my current line of
business, I have little stake in the outcome.
Jim
--
Jim Fulton Digital Creations
jim@digicool.com 540.371.6909
## Python is my favorite language ##
## http://www.python.org/ ##
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From jody@sccsi.com Wed Jul 9 20:53:10 1997
From: jody@sccsi.com (Jody Winston)
Date: Wed, 9 Jul 1997 14:53:10 -0500
Subject: [MATRIX-SIG] Much ado about nothingness.
In-Reply-To: <9707091815.AA18826@ch1d162nwk> (message from Ted Horst on Wed, 9
Jul 97 13:15:41 -0500)
Message-ID: <199707091953.OAA27040@friday.sccsi.com>
Here's a simple Missing class:
import sys
import types
class Missing:
"""An object that defines math operations on missing values.
"""
def __init__(self, value=0, missing=-999.25):
"""This method constructs the object.
The following exceptions are thrown:
o If the value is not number.
"""
if not((type(value) != types.IntType) or
(type(value) != types.LongType) or
(type(value) != types.FloatType) or
(type(value) != types.ComplexType)):
raise TypeError, "Not number"
self.value = value
self.missing = missing
def absent(self):
return self.value == self.missing
def __eitherMissing(self, other):
if self.absent():
return 1
elif other.absent():
return 1
def __repr__(self):
"""Called by the print statement and conversions (reverse quotes) to
compute the string representation of an object."""
return str(self.value) + \
" (" + \
"missing = " + str(self.missing) + \
")"
def __str__(self):
return str(self.value) + \
" (" + \
"missing = " + str(self.missing) + \
")"
def __sum(self, other, sign1, sign2):
if isMissing(other):
if self.__eitherMissing(other):
result = self.missing
else:
result = sign1 * self.value + sign2 * other.value
missing = self.missing
else:
if self.absent():
result = self.missing
else:
result = sign1 * self.value + sign2 * other
missing = self.missing
return Missing(result, missing)
def __add__(self, other):
return self.__sum(other, 1, 1)
__radd__ = __add__
def __sub__(self, other):
return self.__sum(other, 1, -1)
def __rsub__(self, other):
return self.__sum(other, -1, 1)
def __cmp__(self, other):
if isMissing(other):
if self.absent() and other.absent():
return 0
diff = self.__sum(other, 1, -1)
return cmp(diff.value, 0)
def __mul__(self, other):
if isMissing(other):
if self.__eitherMissing(other):
result = self.missing
else:
result = self.value * other.value
missing = self.missing
else:
if self.absent():
result = self.missing
else:
result = self.value * other
missing = self.missing
return Missing(result, missing)
__rmul__ = __mul__
def __div__(self, other):
if isMissing(other):
if self.__eitherMissing(other):
result = self.missing
else:
result = self.value / other.value
missing = self.missing
else:
if self.absent():
result = self.missing
else:
result = self.value / other
missing = self.missing
return Missing(result, missing)
def __rdiv__(self, other):
if isMissing(other):
if self.__eitherMissing(other):
result = self.missing
else:
result = other.value / self.value
missing = self.missing
else:
if self.absent():
result = self.missing
else:
result = other / self.value
missing = self.missing
return Missing(result, missing)
def __pow__(self, other):
if isMissing(other):
if self.__eitherMissing(other):
result = self.missing
else:
result = pow(self.value, other.value)
missing = self.missing
else:
if self.absent():
result = self.missing
else:
result = pow(self.value, other)
missing = self.missing
return Missing(result, missing)
def __rpow__(self, other):
if isMissing(other):
if self.__eitherMissing(other):
result = self.missing
else:
result = pow(other.value, self.value)
missing = self.missing
else:
if self.absent():
result = self.missing
else:
result = pow(other, self.value)
missing = self.missing
return Missing(result, missing)
def __abs__(self):
if self.absent():
return Missing(self.value, self.missing)
return Missing(abs(self.value), self.missing)
def __pos__(self):
return self
def __neg__(self):
if self.absent():
return Missing(self.value, self.missing)
return Missing(-self.value, self.missing)
def __nonzero__(self):
return self.value != 0
def isMissing(x):
return hasattr(x, "value") and hasattr(x, "missing")
def main(argv):
a = Missing()
if (a + 10) != 10:
raise ValueError
b = Missing(20, -100)
if (a + b) != Missing(20):
raise ValueError
if __name__ == '__main__':
main(sys.argv)
sys.exit(0)
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Wed Jul 9 21:18:50 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Wed, 9 Jul 1997 16:18:50 -0400 (EDT)
Subject: [MATRIX-SIG] Much ado about nothingness.
In-Reply-To: <199707091953.OAA27040@friday.sccsi.com>
Message-ID: <Pine.SGI.3.95q.970709161633.27308A-100000@maigret>
On Wed, 9 Jul 1997, Jody Winston wrote:
> Here's a simple Missing class:
> [...]
Note that a version of the Numeric and umath modules which know about
Missing's is also needed, so that take(x,y) and log(x) do the right thing.
--da
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From jim.fulton@digicool.com Wed Jul 9 21:35:49 1997
From: jim.fulton@digicool.com (Jim Fulton)
Date: Wed, 09 Jul 1997 16:35:49 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
References: <Pine.SGI.3.95q.970709161633.27308A-100000@maigret>
Message-ID: <33C3F625.188A@digicool.com>
David Ascher wrote:
>
> On Wed, 9 Jul 1997, Jody Winston wrote:
>
> > Here's a simple Missing class:
> > [...]
>
> Note that a version of the Numeric and umath modules which know about
> Missing's is also needed, so that take(x,y) and log(x) do the right thing.
Right. Basically, *anything* that deals with data at the C
level would have to check for missing data.
This is especially problematic when dealling with external
libraries.
Jim
--
Jim Fulton Digital Creations
jim@digicool.com 540.371.6909
## Python is my favorite language ##
## http://www.python.org/ ##
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From vennerbc@musc.edu Thu Jul 10 00:36:12 1997
From: vennerbc@musc.edu (Bradley C. Venner)
Date: Wed, 09 Jul 1997 19:36:12 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
References: <199707091401.KAA06541@ptq3.etsd> <9707091815.AA18826@ch1d162nwk>
Message-ID: <33C4206C.167E@musc.edu>
I have found this discussion very interesting. As an academic, though,
I feel the need for more authority-citing. So I will pull out two big
guns, one from the database field and one from the statistical
community, on the representation of missing data.
>From the database side, C. J. Date has been the biggest critic of a
default value for "NULLS" in SQL. He has a recent 5-part column in
Database Programming and Design on this subject which is very
informative. My misrepresentation of his position is that no default
scheme is available to handle missing information. His view of
relational databases is that the fields in a database are considered to
be a 'type'. Each type may have one or more default types, but that
operators on these default values must be written.
>From the statistical side, Rubin's work on multiple imputation expresses
a sympathetic concern. To do multiple imputation one must specify a
mechanism for why the data is missing. The other known data values may
then be used to make up new values for the missing data. However, to
acknowledge that one made these values up, the proceedure is repeated
several times, so that several equally likely data sets are created.
Statistics can be computed by applying your analysis not only to just
one dataset but to several. Things like confidence intervals may then
represent the uncertainty introduced by the missing data. For example,
the new NHANES data tapes will contain several datasets, forcing the
investigator to confront that his answer may depend on which set he
chooses.
In practice, I think an intermediate class with strong NumPy <->
database bindings is probably necessary for analysis purposes. (ahh,
indirection, your friend and mine) Such a class, a la data.frame in
S-Plus, could represent both categorical and numeric data. Methods for
this class for converting to NumPy arrays for use in analysis would
force the user to decide how to treat the missing values. Defaults such
as dropping rows with missing data could be available. I think such a
class would have a number of uses outside of missing data applications,
particularly for data mining. The suggestions on defining a new class
so far are a good place to start.
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Thu Jul 10 02:24:42 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Wed, 09 Jul 1997 21:24:42 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
In-Reply-To: <33C4206C.167E@musc.edu>
References: <199707091401.KAA06541@ptq3.etsd>
<9707091815.AA18826@ch1d162nwk>
Message-ID: <3.0.1.32.19970709212442.0080d380@popd.netcruiser>
At 07:36 PM 7/9/97 -0400, Bradley C. Venner wrote:
>a sympathetic concern. To do multiple imputation one must specify a
>mechanism for why the data is missing. The other known data values may
>then be used to make up new values for the missing data.
Nothing like this should be _part_ of the language, since it is not
going to be possibly to specify a mechanism which is good in all cases.
What you _can_ put into the language is an efficient way to determine that
there _are_ missing values, and (in many cases), a good way to determine
_which_ values are missing.
Everything else should be done by the user on top of this, and will vary
greatly from application to application.
>In practice, I think an intermediate class with strong NumPy <->
>database bindings is probably necessary for analysis purposes.
Interestingly enough, I do not expect ever to use a "database" in the
classical sense. The reason is that most database models offer very
poor performance in the applications I use - large scale financial
computing.
What I have done in serveral different platforms is to keep the data
in a form on which my applications directly operate. As a result, I do all
sorts of statistical computations, yet no database is even remotely
important, or even useful. We've always _wanted_ to find a database which
could measure
up, (I once spent two years evaluating all the remotely viable choices),
but I don't think I can be competitive using a database in the usual sense.
It may be somewhat surprising but the majority of companies in my area
do the same thing.
> Such a class, a la data.frame in
>S-Plus, could represent both categorical and numeric data. Methods for
>this class for converting to NumPy arrays for use in analysis would
>force the user to decide how to treat the missing values.
The data.frame class is a powerful tool, and something like it will be
implemented in Python, probably sooner, rather than later. Right now
the hitch is mostly what to do about indexing.
Although I enthusiastically support a data.frame class for python, I
see no reason why this bears on whether or not a user cares about missing
values. People with missing values will expect the data.frame to treat
missing values consistently with the methods from the parent classes,
and so _those_ methods are where the issue crops up.
> Defaults such
>as dropping rows with missing data could be available. I think such a
>class would have a number of uses outside of missing data applications,
>particularly for data mining. The suggestions on defining a new class
>so far are a good place to start.
Just make sure that particular methods of handling missing data are part
of a subclass, _not_ part of the data.frame class itself. You won't have
a good single method of handling missing data for most applications -
even if you are talking about handling the exact same missing value
instance being acted on by the exact same methods but in different situations.
What tends to be _easy_ to standardize is a mechanism for representing
missing values. It's definitely not a good idea to try to standardize
the methods of handling missing values, which is as application dependent
as anything I can imagine.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Thu Jul 10 10:21:04 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Thu, 10 Jul 1997 11:21:04 +0200
Subject: [MATRIX-SIG] Much ado about nothingness.
In-Reply-To: <9707091815.AA18826@ch1d162nwk> (message from Ted Horst on Wed, 9
Jul 97 13:15:41 -0500)
Message-ID: <199707100921.LAA18102@lmspc1.ibs.fr>
> Has anybody looked at the Missing type from DigiCool ? It has the desired
Noticed, but not looked at...
> behavior for arithmetic operations, but doesn't work correctly for math or
> umath functions. I have no idea how hard this would be to get working, or even
Umath support is easy: just implement methods for each function. For any
unknown type, umath tries to translate the function into a method call, i.e.
sin(x) --> x.sin()
exp(x) --> x.exp()
Anyone implementing numerical types, in C extensions or as Python classes,
should provide this set of methods.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Thu Jul 10 10:39:04 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Thu, 10 Jul 1997 11:39:04 +0200
Subject: [MATRIX-SIG] namespaces, constructors, and global variables
In-Reply-To: <m3bu4cgp68.fsf@curlew.geog.ubc.ca> (message from Phil Austin on
09 Jul 1997 09:34:23 -0700)
Message-ID: <199707100939.LAA18142@lmspc1.ibs.fr>
> We're working on a generic plotting interface to gnuplot, gist, ncar
> graphics, etc., and are currently stumped by our lack of understanding
Great! However, I wonder whether a unification of plotting packages
is the right way to go. I think it would be not much more effort, but
much more useful, to write Python code that generates low-level graphics
elements (lines, circles, etc.) directly to make up a plot. Then there
could be several low-level drivers that render this representation
via Tk canvases, OpenGL, PostScript, or whatever.
This may sound like reinventing wheels, but given the problems and/or
limitations that all known plotting packages cause, it may be the
better way to happiness.
> of namespaces. The interface puts a plot together by creating a
> series of objects (plots, axes, ticks, legends, etc.) with methods to
> add or modify fonts, colors, line widths, whatever. The state of the
> plot is stored in a global dictionary, and an execute method goes
I would not recommend to use *any* global objects unless you have a
very good reason. If I understand your intention well, it would be
impossible to construct two plots in parallel. Why that restriction?
Just define a "plot" class that contains all the data.
> Running from the shell, however, doesn't work, because the global
> variable Master doesn't appear to have it's keys:
>
> <phil@curlew: src.dir/python.dir> python PlotAll.py
> Master.keys(): ['plotCount', 'device', 'plot1']
> inside BuildAxis: Master.keys()= []
>
> Can someone walk me through why this is failing, and how I can fix it?
Without analyzing anything in detail, I'll point to a critical
spot:
> class BuildPlot:
...
> code=["from PlotAll import *",
> "global Master",
> "leftAxis=BuildAxis(plot,axis=\"left\",label=\"ytest\")",
> "botAxis=BuildAxis(plot,axis=\"bot\",label=\"x\")"]
> for stmt in code:
> exec stmt in ns,__main__.__dict__
__main__ will clearly be different in the two situations. And like
global variables, accessing __main__ is something that a well-behaved
Python program should not do.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Thu Jul 10 10:43:22 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Thu, 10 Jul 1997 11:43:22 +0200
Subject: [MATRIX-SIG] making a text label from an object instance?
In-Reply-To: <m390zggjh3.fsf@curlew.geog.ubc.ca> (message from Phil Austin on
09 Jul 1997 11:37:28 -0700)
Message-ID: <199707100943.LAA18154@lmspc1.ibs.fr>
> For my plotting package I would like to have the default axes labels
> be the names of the objects being plotted. For example, if the
> command is:
>
> plot(time[:,4],vel[:,3,2])
>
> I would like to generate strings containing "time[:,4]" and
> "vel[:,3,2]". Is this possible with Python?
No. And it's not even clear what the outcome should be in more
complicated situations, such as:
def foo(x):
apply(plot, 2*(x,))
I wouldn't worry too much about default labels. I find them rarely
useful. It's simple enough to pass an explicit string.
Konrad.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Thu Jul 10 12:42:37 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Thu, 10 Jul 1997 07:42:37 -0400
Subject: [MATRIX-SIG] Much ado about nothingness.
Message-ID: <199707101148.HAA04852@dante.mh.lucent.com>
----------
> From: Konrad Hinsen <hinsen@ibs.ibs.fr>
...
> Umath [null] support is easy: just implement methods for each function. For any
> unknown type, umath tries to translate the function into a method call, i.e....
Nothing about nulls is easy, as CJ Date will confirm (although
beyond that he's usually wrong regarding Nulls imho :) ).
Basically, the standard database theory breaks down and
becomes totally useless when nulls are introduced and treated
seriously. CJ Date and the like tend to conclude that "nulls
are bad" and just throw them out (a convenience only a
theoretician has).
EG Consider the table
name age
john 5
fred 6
lisa 5
wally null
lola null
select those names with age 5.
IMHO the correct answer it *two* sets
inner = {john,lisa}, outer={john, fred, lisa, lola}
The "correct" answer is somewhere in between
(if we had better information).
what is the average age? null, I think.
rephrased, what is the average among
the known ages? (5+6+5)/3 (wally, lola thrown out).
Thus for accumulation calculations (including
len) null values sometimes should be thrown
out, ie
known_sum([1,2,3,Null,3]) = 9
known_min_len([1,3,3,Null,3]) = 4 (not 5!)
here I say "known min len" because more generally
the Null might be a place holder that stands for
*several* missing values.
On the other hand there are "not applicable" nulls
too
item pages
ipwp 300
pp 900
win95 NA
what are the average number of pages? (900+300)/2
what has 900 pages? {pp}
(win95 doesn't enter the calculation, it has no pages)
"unknown" Nulls are also a special case of more general
"partial information" values, such as
numeric intervals, where
sum([interval(5,6), interval(7,9)]) = interval(12,15)
Gosh, you could write a whole dissertation
on the topic (but don't, I already did :) ).
In sum, I suspect there is no quicky answer for
treating nulls -- there might be uniform approaches,
but I'd like to understand the required semantics
very clearly before modifying numeric to support
them. I suspect differing apps require radically
differing semantices...
-- Aaron Watters
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Thu Jul 10 12:44:47 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Thu, 10 Jul 1997 07:44:47 -0400
Subject: [MATRIX-SIG] namespaces, constructors, and global variables
Message-ID: <199707101148.HAA04855@dante.mh.lucent.com>
> From: Konrad Hinsen <hinsen@ibs.ibs.fr>
> > We're working on a generic plotting interface to gnuplot, gist, ncar
> > graphics, etc., and are currently stumped by our lack of understanding
>
> Great! However, I wonder whether a unification of plotting packages
> is the right way to go. I think it would be not much more effort, but
> much more useful, to write Python code that generates low-level graphics
> elements (lines, circles, etc.) directly to make up a plot. Then there
> could be several low-level drivers that render this representation
> via Tk canvases, OpenGL, PostScript, or whatever.
Seconded. This is the proper way to design it IMHO. If you do it
carefully I don't even think you'll see any particular performance problems.
--Aaron Watters
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From phil@geog.ubc.ca Thu Jul 10 22:50:28 1997
From: phil@geog.ubc.ca (Phil Austin)
Date: Thu, 10 Jul 1997 14:50:28 -0700
Subject: [MATRIX-SIG] namespaces, constructors, and global variables
In-Reply-To: <199707101148.HAA04855@dante.mh.lucent.com>
References: <199707101148.HAA04855@dante.mh.lucent.com>
Message-ID: <199707102150.OAA03037@curlew.geog.ubc.ca>
>>>>> "Konrad" == Konrad Hinsen <hinsen@ibs.ibs.fr> writes:
Konrad> Great! However, I wonder whether a unification of plotting
Konrad> packages is the right way to go. I think it would be not
Konrad> much more effort, but much more useful, to write Python
Konrad> code that generates low-level graphics elements (lines,
Konrad> circles, etc.) directly to make up a plot. Then there
Konrad> could be several low-level drivers that render this
Konrad> representation via Tk canvases, OpenGL, PostScript, or
Konrad> whatever.
>>>>> "Aaron" == Aaron Watters <arw@dante.mh.lucent.com> writes:
Aaron> Seconded. This is the proper way to design it IMHO. If
Aaron> you do it carefully I don't even think you'll see any
Aaron> particular performance problems. --Aaron Watters
Thirded. I'm operating under some constraints however:
1) I think I've provided some evidence that, on a coding project like
this, I'll be staying in the shallow end of the pool.
2) I need a single interface that will produce simple plots under
linux and Windows 95 (student labs start Sept. 3), and camera ready
publication plots under linux and solaris. For the latter I require
a superset of the capabilities demonstrated at:
http://ngwww.ucar.edu/ngdoc/ng4.0.1/ug/ncl/tutor/session3.html (For
people interested in pricing for this package--an unlimited
use/unlimited-time University site licence cost us about $Can 900
(there's no windows port). A typical commercial alternative with a
fairly primitive array language (IDL/PV-Wave) is $2500 per seat).
3) I need the camera-ready plots within two weeks.
My guess is that someone might eventually write a portable set of
Java/tk/opengl/etc. classes to do much of what I want, and with luck it
will happen before 1999 and could even be GPL'ed. Near-term is going
to require some improvisation, however.
Konrad adds:
Konrad> I would not recommend to use *any* global objects unless
Konrad> you have a very good reason. If I understand your
Konrad> intention well, it would be impossible to construct two
Konrad> plots in parallel. Why that restriction?
and:
Konrad> __main__ will clearly be different in the two
Konrad> situations. And like global variables, accessing __main__
Konrad> is something that a well-behaved Python program should not
Konrad> do.
The plot layout is now encapsulated, so no more globals, thanks. On
the second issue, I was trying to get default behavior that
produces a plot with very few keystrokes, while maintaining the ability to
customize the axis, labels, ticks, etc. after the plot command is typed.
Specifically, I'm looking for three different user levels:
Novice:
_______
BuildPlot(x,y) #x-y line plot with default axes
#also creates leftAxis and botAxis--their set methods
#can be used to modify defaults if desired as in:
Advanced Novice: change the default axis parameters following the plot command
_______________
BuildPlot(x,y)
leftAxis.set(label="Temperature",font="urdu",fontSize="5pt")
Intermediate to Advanced:
________________________
Here, we want a plot with two ordinates of different colors on the
rhs, offset horizontally from each other by 0.2 units, one a
logrithmic pressure axis with values increasing downward, one a linear
height axis with values increasing upward:
plot1=BuildPlot(temp,pressure,corner=(0,0),size=(2,2),axes="F",log="y")
#no default axes, but plot size and location
#specified, ordinate is logarithmic
leftAxis=BuildAxis(plot1,axis="bot")
rightAxis=BuildAxis(plot1,axis="right",color="red",type="inverted")
#now build another plot of temperature/height, positioned on top of
#plot1, but draw only a new right axis
plot2=BuildPlot(x,z,corner=(0,0),size=(2,2),type="None",axes="F")
rightAxis=BuildAxis(plot2,axis="right",bottom=(2.2,0),color="green")
______________
If default behavior that creates an unasked for leftAxis and a botAxis
in __main__ is risky, then I can just drop the idea that the default
axes can be modified, and move my novices quickly into the
intermediate stage if necessary.
I'd love to see some other approaches that balance this
terseness/flexibility ratio in different ways.
Regards, Phil
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Fri Jul 11 00:23:03 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Thu, 10 Jul 1997 19:23:03 -0400
Subject: [MATRIX-SIG] making a text label from an object instance?
In-Reply-To: <199707100943.LAA18154@lmspc1.ibs.fr>
References: <m390zggjh3.fsf@curlew.geog.ubc.ca>
Message-ID: <3.0.1.32.19970710192303.0081a8d0@popd.netcruiser>
At 11:43 AM 7/10/97 +0200, Konrad Hinsen wrote:
>I wouldn't worry too much about default labels. I find them rarely
>useful. It's simple enough to pass an explicit string.
Don't do a lot of automatically generated plots, eh?
Default labels are _very_ useful. I don't see a really easy way
to do it in python, but it's one of the things people really
like when it's available.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From phil@geog.ubc.ca Fri Jul 11 00:49:16 1997
From: phil@geog.ubc.ca (Phil Austin)
Date: Thu, 10 Jul 1997 16:49:16 -0700
Subject: [MATRIX-SIG] making a text label from an object instance?
In-Reply-To: <3.0.1.32.19970710192303.0081a8d0@popd.netcruiser>
References: <m390zggjh3.fsf@curlew.geog.ubc.ca>
<199707100943.LAA18154@lmspc1.ibs.fr>
<3.0.1.32.19970710192303.0081a8d0@popd.netcruiser>
Message-ID: <199707102349.QAA03552@curlew.geog.ubc.ca>
>>>>> "Andrew" == Andrew P Mullhaupt <amullhau@ix.netcom.com> writes:
Andrew> At 11:43 AM 7/10/97 +0200, Konrad Hinsen wrote:
>> I wouldn't worry too much about default labels. I find them
>> rarely useful. It's simple enough to pass an explicit string.
Andrew> Don't do a lot of automatically generated plots, eh?
Andrew> Default labels are _very_ useful. I don't see a really
Andrew> easy way to do it in python, but it's one of the things
Andrew> people really like when it's available.
I'd agree with this--looking at a suprising result in your notebook
several weeks after the plot was produced, it's very reassuring to
know that there is no possible way the axis labels could have been
swapped. The variables in the label can be made more informative
using named dimensions and range indexes, so that contours of a 4-d
array of vertical velocity don't really need a title:
contour(vertical_velocity["500milibars","300Kisotherm",lats,lons])
combined with a time-stamp that can be printed on the plot and used in
an RCS comment when you check in the plotting script, it's possible to
create 500 fully documented data plots without any programmer
intervention, and recover any one of them six months later from the
raw netcdf data after your grad student has spilled root beer
on the original.
There's no looking back though. Splus has the treasured
"deparse(substitute())" but they've doubled their site licence fees
and seem to be preparing to abandon Unix as a supported platform.
Brian Ripley (http://lib.stat.cmu.edu/S/MASS2/), for example, recently
posted a message in S-news that sounded positively bitter. Picking up
some Splus refugees who happen to be world-renown in
bootstrap/LOESS/ARIMA techniques would be major coup for Python
(albeit a fairly distant prospect).
Regards, Phil
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Fri Jul 11 11:38:22 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Fri, 11 Jul 1997 12:38:22 +0200
Subject: [MATRIX-SIG] making a text label from an object instance?
In-Reply-To: <3.0.1.32.19970710192303.0081a8d0@popd.netcruiser>
(amullhau@ix.netcom.com)
Message-ID: <199707111038.MAA21796@lmspc1.ibs.fr>
> At 11:43 AM 7/10/97 +0200, Konrad Hinsen wrote:
> >I wouldn't worry too much about default labels. I find them rarely
> >useful. It's simple enough to pass an explicit string.
>
> Don't do a lot of automatically generated plots, eh?
I do, but on the screen. I rarely print plots other than for
seminars or publications.
> Default labels are _very_ useful. I don't see a really easy way
> to do it in python, but it's one of the things people really
> like when it's available.
OK, so here's how to do them in Python:
def plot(x, y):
label_x = ''
if type(x) == type(''):
label_x = x
x = eval(x)
label_y = ''
if type(y) == type(''):
label_y = y
y = eval(y)
do_the_plot(x, label_x, y, label_y)
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Fri Jul 11 15:18:35 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Fri, 11 Jul 1997 10:18:35 -0400
Subject: [MATRIX-SIG] making a text label from an object instance?
In-Reply-To: <199707111038.MAA21796@lmspc1.ibs.fr>
References: <3.0.1.32.19970710192303.0081a8d0@popd.netcruiser>
Message-ID: <3.0.1.32.19970711101835.00814dd0@popd.netcruiser>
At 12:38 PM 7/11/97 +0200, Konrad Hinsen wrote:
[ I wrote:]
>> Default labels are _very_ useful.
>OK, so here's how to do them in Python:
>def plot(x, y):
Looks good for most uses. Solves that.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From ptaney@usgs.gov Fri Jul 11 15:31:21 1997
From: ptaney@usgs.gov (Paul Taney, Computer Specialist, Reston, VA )
Date: Fri, 11 Jul 1997 10:31:21 -0400
Subject: [MATRIX-SIG] camera-ready
Message-ID: <199707111431.KAA26561@mailrvares.er.usgs.gov>
Phil:
>3) I need the camera-ready plots within two weeks.
I cannot suggest programatic solutions for you; you are
getting the best advice there! But I am a viz specialist
on SGIs here at USGS and I'd like to suggest a very usable
alternative to paper plots. Unless you need something
quite large (which I do not) prepare final output
in this manner; as do my contacts at our primary viz
software vendor, Dynamic Graphics:
1) In a windowless room darkened to hide the bezel of monitor
2) With a good, large monitor (this is a cross-platform solution ;)
3) With a 35mm SLR with a 85mm or 105mm lens and heavy tripod --
placing lens exactly on normal line from centroid of monitor
3.1) A Hasselblad would also do...
4) Using 100 ASA Ectachrome slide film -- or some other 100 ASA
5) Shoot screen at f11 for 1+ sec and again at 2 sec ("bracketing")
Both shots should be usable in my experience.
Look at the cost/time savings over film recorders at $6000+;
and quality is as good or better; those monitors are itsy-bitsy.
This may not be what you need for some reason, but it is a
professional solution.
paul
ptaney@usgs.gov @ Just when you think you're
U.S.Geological Survey @ going to be decorated...
reston, va 22092 @ Nothing happens.
ph: 703-648-5598 @ -- Chogyam Trungpa
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From chase@att.com Fri Jul 11 17:08:04 1997
From: chase@att.com (Chase, Chris)
Date: Fri, 11 Jul 1997 12:08:04 -0400
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited
Message-ID: <9707111609.AA28965@hoccson.ho.att.com>
Allowing arrays to be indexes (or any sequence object for that matter)
in the array module is very doable. But it is a huge amount of work.
At different times during the past year I have looked at concepts and
implementations for this. But after looking at the code (last year and
again this week) it would require rewriting a majority of the routines
in the array module and associated files. This is because the use of
strides is heavily embeded in the code and there is no separation
between array representation and the interface and helper functions.
The job is more difficult because the comments about what the code is
trying to do are very scarce (but I have a very old version of the
sources).
Some side comments:
Slices that go outside array bounds are silently projected onto the
bounds while take() generates an error when the index vector goes
outside the bounds. If arrays were allowed as indexes should they
behave like slices or like take()?
The behavior of the array module is that indexing an array produces a
reference to the original data unless a scalar is produced. In allowing
arrays as indexes, the array reference representation would have to keep
copies of the index array. This does not take any more space than
actually producing a copy of the data than producing a reference.
Reshaping has to generate a copy unless the input array was already
contiguous. If arrays as well as slices are allowed as indexes and kept
in the array representation, then an array reshape could also produce a
reference to the original data.
Arrays as indexes kept in the representation also allow compact
representations of array replication (e.g. repeating certain
dimensions).
I am interested in attempting to make the modifications. But I think
that the job is too big (i.e. too time-consuming of my spare time) to
complete alone. Also there may be opposition to making extensive
changes within the code (although the array object interface would
remain the same) because some individuals may depend on the actual
representation/implementation.
Chris Chase
chase@att.com
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Fri Jul 11 22:54:39 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Fri, 11 Jul 1997 17:54:39 -0400
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take"
revisited
In-Reply-To: <9707111609.AA28965@hoccson.ho.att.com>
Message-ID: <3.0.1.32.19970711175439.0081d9c0@popd.netcruiser>
At 12:08 PM 7/11/97 -0400, Chase, Chris wrote:
>
>Slices that go outside array bounds are silently projected onto the
>bounds while take() generates an error when the index vector goes
>outside the bounds. If arrays were allowed as indexes should they
>behave like slices or like take()?
>
Actually, _functions_ such as take should not be tied to the behavior
of indices. You can write all sorts of functions to do anything you
want, so the user shouldn't have an expectation of a connection between
index semantics and function semantics except for the _very few_
necessary functions which explicitly involve the shape of the array.
So for consistency they can behave like slices, but I'm a bit surprised
at how you claim slice indices work.
So I tried some examples and I think slice indices are broken. For example:
Python 1.4 (Feb 24 1997) [C]
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> a = reshape(range(12), [3, 4])
>>> a[0:3, 0:4]
0 1 2 3
4 5 6 7
8 9 10 11
>>> a[0:4, 0:4]
0 1 2 3
4 5 6 7
8 9 10 11
>>> a[-1:4, 0:4]
8 9 10 11
>>> a[4:0, 0:4]
EmptyArray
>>> a[3:0, 0:4]
EmptyArray
>>> a[3:0:-1, 0:4]
0 0 209 0
8 9 10 11
4 5 6 7
>>> a[3:0:-1, 4:0:-1]
1 0 209 0
0 11 10 9
8 7 6 5
>>>
Whoa! Let's try that again:
Python 1.4 (Feb 24 1997) [C]
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> a = reshape(range(12), [3, 4])
>>> a = reshape(range(12), [3, 4])
>>> a[3:0:-1, 4:0:-1]
1936654336 7103790 1328443513 1768843081
0 11 10 9
8 7 6 5
>>> a = reshape(range(12), [3, 4])
>>> a[3:0:-1, 4:0:-1]
0 1545120 41 0
0 11 10 9
8 7 6 5
Broken, Yes?
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From chase@att.com Sat Jul 12 01:34:56 1997
From: chase@att.com (Chase, Chris)
Date: Fri, 11 Jul 1997 20:34:56 -0400
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited
Message-ID: <9707120036.AA16323@hoccson.ho.att.com>
> ----------
> From: Andrew P. Mullhaupt[SMTP:amullhau@ix.netcom.com]
> Sent: Friday, July 11, 1997 5:54 PM
> To: Chase, Chris; 'matrix-sig@python.org'
> Subject: RE: [MATRIX-SIG] A proposal (LONG) - "reverse of take"
> revisited
>
> At 12:08 PM 7/11/97 -0400, Chase, Chris wrote:
> >
> >Slices that go outside array bounds are silently projected onto the
> >bounds while take() generates an error when the index vector goes
> >outside the bounds. If arrays were allowed as indexes should they
> >behave like slices or like take()?
> >
>
> Actually, _functions_ such as take should not be tied to the behavior
> of indices.
>
Sorry for the confusion, but that is not the question I was trying to
ask. I was asking what should the behavior of arrays as indexes be with
respect to array bounds and I was illustrating two possible behaviors
now seen in NumPy. Reasonable possible behaviors for arrays used as
indexes when the indexes are outside the array bounds:
1) project the indexes onto the bounds. This takes the intersection of
the bounds and the index dropping individual indexes outside the bounds.
This is the type of behavior currently exhibited by slices. The problem
with clipping is that you get back fewer elements than expected.
2) project the indexes by clipping. This clips indexes outside the
bounds to the bounds. For example, an index of [20] on a size 3 array
uses [2] as the index. This is the way IDL array indexes behave (which
is incongruous with the errors that out-of-bounds scalar indices
generate in IDL). For some applications this can be convenient while in
other situations you would prefer an error.
3) generate an error. This is the behavior of take() when the index is
out of bounds.
> You can write all sorts of functions to do anything you
> want, so the user shouldn't have an expectation of a connection
> between
> index semantics and function semantics except for the _very few_
> necessary functions which explicitly involve the shape of the array.
>
> So for consistency they can behave like slices, but I'm a bit
> surprised
> at how you claim slice indices work.
>
> So I tried some examples and I think slice indices are broken. For
> example:
>
> Python 1.4 (Feb 24 1997) [C]
> Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
> >>> a = reshape(range(12), [3, 4])
> >>> a[0:3, 0:4]
> 0 1 2 3
> 4 5 6 7
> 8 9 10 11
> >>> a[0:4, 0:4]
> 0 1 2 3
> 4 5 6 7
> 8 9 10 11
>
The above is the slice clipping behavior.
> Python 1.4 (Feb 24 1997) [C]
> Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
> >>> a = reshape(range(12), [3, 4])
> >>> a = reshape(range(12), [3, 4])
> >>> a[3:0:-1, 4:0:-1]
> 1936654336 7103790 1328443513 1768843081
> 0 11 10 9
> 8 7 6 5
>
This is indeed a bug in the slice clipping. The bug is in arrayobject.c
in routine slice_GetIndices():
if (*start < 0) *start = 0;
else if (*start > length) *start = length;
should be something like:
if (*start > (length-1)) *start = length-1;
if (*start < 0) *start = 0;
Chris
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Sat Jul 12 04:26:06 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Fri, 11 Jul 1997 23:26:06 -0400
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take"
revisited
In-Reply-To: <9707120036.AB16323@hoccson.ho.att.com>
Message-ID: <3.0.1.32.19970711232606.00804ce0@popd.netcruiser>
At 08:34 PM 7/11/97 -0400, Chase, Chris wrote:
>
> I was asking what should the behavior of arrays as indexes be with
>respect to array bounds and I was illustrating two possible behaviors
>now seen in NumPy. Reasonable possible behaviors for arrays used as
>indexes when the indexes are outside the array bounds:
>
>1) project the indexes onto the bounds. This takes the intersection of
>the bounds and the index dropping individual indexes outside the bounds.
>This is the type of behavior currently exhibited by slices. The problem
>with clipping is that you get back fewer elements than expected.
This is not normally considered a reasonable behavior, however it isn't
a disaster.
You get all sorts of weird ideas for this kind of stuff in different
array languages. APL gives an error for any access outside the shape
of an array.
Now S-plus is _pretty_ weird for this:
> A <- array(1:6, dim=c(3,2))
> A
[,1] [,2]
[1,] 1 4
[2,] 2 5
[3,] 3 6
> A[3, 2]
6
> A[6]
6
> A[7]
NA
> A[1,4]
Error in A[1, 4]: Array subscript (4) out of bounds. should be at most 2
Dumped
> A[7] <- 7
> A
[,1] [,2]
[1,] 1 4
[2,] 2 5
[3,] 3 6
> A[7]
7
> A[1,4] <- 7
Error in A[1, 4] <- 7: Array subscript (4) out of bounds. should be at most 2
Dumped
Cool, right? I don't want to get into a discussion of why this happens, just
want to point out that it comes from the way S-plus uses object orientation
- it has to do with arrays acting like a subclass of vectors but having
different methods for out-of-bounds accesses.
But wait! Matlab is the _exact converse_!. You can't interrogate an element
outside the shape of the array, but if you _assign_ one then the array
expands. But not if it's a vector.
Moral of the story: This is an issue where you can adopt really screwy
ideas and people might still buy and use your product. So I suppose
silently truncating indices might make sense to somebody, although
at the moment it seems somewhat risky and eccentric.
This is not to say that it doesn't matter, though. People who make mistakes
and over-access an array probably will appreciate errors being raised.
People who want to work with sparse matrices will absolutely hate those
errors.
>2) project the indexes by clipping. This clips indexes outside the
>bounds to the bounds. For example, an index of [20] on a size 3 array
>uses [2] as the index. This is the way IDL array indexes behave (which
>is incongruous with the errors that out-of-bounds scalar indices
>generate in IDL). For some applications this can be convenient while in
>other situations you would prefer an error.
Just so. Another array language, another different way of handling out of
bounds indices. Can anyone think of two different languages which handle
this the same way?
>3) generate an error. This is the behavior of take() when the index is
>out of bounds.
Good old APL style. This has the virtue that even if it isn't making you
happy, it's easy to understand and expect. You could do worse.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From isenmann@physik.tu-cottbus.de Sun Jul 13 13:41:35 1997
From: isenmann@physik.tu-cottbus.de (G.Isenmann)
Date: Sun, 13 Jul 1997 14:41:35 +0200 (MDT)
Subject: [MATRIX-SIG] sorry, but I am the idiot
Message-ID: <Pine.SGI.3.96.970713144117.25145A-100000@lucy.physik.tu-cottbus.de>
I am very sorry for injecting old messages (about end of May) into the
list.
I wanted to resent a collection of saved/unread messages to myself, but
because of a stupid mistake, they were sent to the addess of the mailing-
list (sometimes also to the author of the message).
Goetz Isenmann isenmann@physik.tu-cottbus.de
Tel: +49 355 69 3004
FAX: +49 355 69 3011
Lehrstuhl fuer Theoretische Physik, Karl-Marx-Str. 17, 03044 Cottbus, Germany
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Sun Jul 13 20:30:01 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Sun, 13 Jul 1997 15:30:01 -0400
Subject: [MATRIX-SIG] [PYTHON MATRIX-SIG] SWIG, NumPy and C++
templated arrays
In-Reply-To: <199705282335.QAA29653@curlew.geog.ubc.ca>
Message-ID: <3.0.1.32.19970713153001.0080cc80@popd.netcruiser>
At 04:35 PM 5/28/97 -0700, Phil Austin wrote:
>The code:
>
> Cutmod.cpp defines a C++ function "cut", which cuts data into a
> series of histogram bins.
>Here's the way I would implement cut in Python:
[deleted]
Yes. That's pretty much the straightforward way. I'd like to take advantage
of this example to show how more flexible indexing can be quite powerful.
Since I don't have the kind of indexing needed in NumPy to do the example,
I'll do it in S. This also helps demonstrate the efficiency of the resulting
code since S has a "built-in" solution which calls C functions which work
just about exactly the way your python module does, give or take some extra
memory overhead involved in the way S calls these C functions.
Also, the S I will use here is essentially the same as NumPy, so I hope most
readers will get this example. One thing which isn't obvious is that c(p,q)
concatenates p and q.
Here is an S function which does the job. Note that the vector of bins
(cut-points)
must be sorted in ascending order, which is a typical restriction for this
kind of
function.
demux <- function(data, bins)
{
lb <- length(bins)
if (lb > 1) {
k <- lb %/% 2 # integer division
return(
c(demux(data[data < bins[k+1]], bins[bins <= bins[k]])
, demux(data[data >= bins[k+1]], bins[bins < bins[k]])
)
)
}
if (any(data < bins[1])) {
return(
c(list(data[data < bins[1]]), list(data[data >= bins[1]))
)
} else {
return(list(data))
}
}
WARNING: I tossed this off in about twenty minutes and so I think I've handled
the "edge" cases I'm not entirely sure. If there is such a bug in this code it
isn't likely to affect the timings - which is the thrust of this post, so I
won't worry about this issue. Just don't use this code as given and expect it
to be bullet- (or even spitball-) proof.
Remark: The algorithm as given _can_ be implemented in Python without using
indexing, and appealing to user-provided functions. If you write that down
I think you'll see that this version is a lot more readable. At least I
(and most other people who have used array languages) will think so.
Now there isn't much to this other than divide and conquer combined with
logical indexing, (one of the convenient flavors). The question is, given
the incredibly painful interpreter overhead of S-plus will this be even
close to usable? We will compare with the "built-in" solution:
split(data, cut(data, bins))
length of data: 100,000
bins are uniformly distributed in the data.
#bins demux built-in
5 0.7 1.0
20 1.1 1.2
100 1.7 1.8
1000 3.7 8.2
10000 17.1 81.0
Hmmm. Are we doing too well?
Scrutiny of the S-plus implementation of split.default shows that they build
the final list up with a loop in S over the bins, and S loops are superlinear.
Note that this loop is not over the data but over the bins, which is why
having
the loop over the bins in the interpreter is not very costly unless the number
of bins is on the same order as the number of data points. Since this loop
is the
only place the built-in solution spends a significant amount of time out of
C, we
can conclude that the S "built-in" solution is going to be a pretty good
indicator
of a completely C implemented version when the number of bins is small
compared to
the number of data points.
This being the case, it seems that the demux version is quite competitive
in S. With
a fast (read built-in) version of this kind indexing, a NumPy version would
likely
be about as fast as a C version unless the ratio of bins to data gets large.
Now for a fixed number of bins, distribution of bins, and distribution of
data,
as the amount of data N increases, our algorithm is O(N), and will outperform
any algorithm which sorts the data (which is Theta(N log N) unless there is
some
special circumstance of the data). This is because our algorithm compares the
data to the bin values - sorting involves comparing the data to the data.
This will
be a very small effect and might require enormous values of N to see it.
Just thought people might like to see what they might get with slicker
indexing.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hauser@lisboa.ifm.uni-kiel.de Tue Jul 15 14:20:05 1997
From: hauser@lisboa.ifm.uni-kiel.de (Janko Hauser)
Date: Tue, 15 Jul 1997 15:20:05 +0200 (CEST)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
Message-ID: <m0wo7WP-000syBC@lisboa.ifm.uni-kiel.de>
Hello here are some of my thoughts regarding the plot-discussion,
which comes up once in a while.
First some of the things I think a ubiquitous plotpackage should cover:
There a some fields wich can be identified:
The ``normal'' function plotting (x-y-plots) with lines or markers or
error-bars. Often one wants to have more than one y- or x-axis here.
All the different types of 2 or 2 1/2 dimensional plots, like
contour-plots, filled contours, vector- and surface-plots. This are
normaly plots on grid. More and more the grids are not equidistant and
the hole field of triangular grids is coming. The view of this grid
changes more from plane views to perspectiv views of more than one of
these grids. (hor. and vert. slices through a data-cube) I think here
stops the field of plotting and the field of visualiztion begins. I
mean that the more dimensions are involved into the plot, the more
arise the need for interactiv movement of the view, or changing of the
color-palettes. So here you must work with the plot or better the view
onto the data. These plots are seldom simple to be printed on
paper. The next step into this direction is the display of isosurfaces
and volume-rendering.
Of course all of this plottypes should be able to be animated, because
of the volume of plots we want to see :-)
OK, than there are field of business graphics, like bar and pie-charts
also in more than one dimensions. Histograms are the same class. Here
it is very important to have rich set of text-functions in boxes, with
shapes or raised against the plot itself.
Than there is the hole field of statistical plots, which I don't know,
but I see often some strange things in such plots.
There is currently no free package wich does all these things. So we
can't wrap up something. (If I'm wrong here, please say it to me )
And there will be more things in the future like building of simple
point and click interfaces to look at some preprocessed data. So there
should be the possibility to connect to a GUI-interface, like Tk.
At the moment there are different ways to plot from python. (write a
Matlab-readable ASCII-file and plot this with Matlab :0)
Honestly, I know from ways to plot over gnuplot, mathematica, pgplot,
tk/blt, plplot, gist.
Especially plplot/pgplot and gist support a wide range of
plotprimitives. So these packages can be and are used to build a new
plot-system in python. But, (there must always be a but, eh?) they are
used in a mixed way. The high-level functions, which are present are
used and the missing parts are build from scratch. I think this way
has some limits, because there is always a mix from changes which can
be done in python and things which must be done on the
c-language-level. One example is the axis-system in 3d in the gist
package. Gist has nice autoscaling and rescaling with mouse-support in
2d. But the axis in 3d are plotted with different routines, the
labeling happens in a different way.
The worst case is, that I use different and overlapping packages to
fullfill my plot-requirements. I think this is bad, because the hole
system becomes more and more a hack, rather then developing to a
kitchen-sink-plot-system with wide application-range. (I don't want to
say, that the current implementations are hacks, but if I try to
change something, than it will become a hack, but I must say here,
that I'm very happy to have something with which I can generate plots
with Numpy here and now)
Here are some more specific requirements:
1) Use standard or at least very portable graphics-libraries: If the
effort is done to build a new plot-system it should be availabel with
some drivers on all platforms, where Numpy is availabel. I think the
two most wiedly available standards are postscript and opengl, which
can both be accsessed trough python (I have a nearly running
postscript-driver, which supports all of the further down mentioned
primitives)
If one use librarys like the above, very different fields can be
attacked, like molecular-modelling with opengl without learning a new
graphics-system or flow-diagramms with postscript.
2) Plots should be interactiv in a way that one can interact with the
plot directly, point to some locations and get the corresponding
values. This is possible with the opengl-driver.
3) At least the 2d plots should have or be able to have publication
quality. That's the major drawback of opengl. So here jumps the
postscript-driver in. For the real 3d-plots (like volume-rendering)
good tiff-files should be enough. But the access to the driver should
be seperated, so one can generate a hole bunch of plots on a Cray
without the need of opengl. (Although SGI owns them :-))
4) Following plot-primitives should be supported by any driver:
markers at a point
lines
closed lines
filled areas (filled with one color or pattern)
bitmap areas (used for things like ligthing, shades)
text which should be rotateable (ascii and some special
fonts). The size of the text-surrounding box should be
known.
Here is some room for higher primitives, like box and vector,
errorbars and so on. But this can also directly be handled by the
plot-routines.
5) These primitives should be there in every backend.
6) The process queue can be:
data -> plot-primitive -> transformation -> scaling, normalization ->
put into viewport with aspect-ratio or big as possible
7) More than one window, more than one viewport in one window. The
viewport should be divided into a plot-area and the space for axis and
annotations. In this way the plots with multiple axis are not a
problem. Perhaps one needs axis in the plot-area, also this can be
done in that way. Each axis should be handled independant.
8) There should be a way for explicit axis-paths with
annotations. Standard creator and custom one. I think the
axis-handling is a crucial point in all existing packages. Normally the
hole axis-system is handled. One has now control over the strings, which
are plotted. So dates are really hard to plot. And what happens if one
wants to plot in polar-coordinates or with a cartographic projection?
The axises of the plot should have similar handles like every other
plotobject and each axis should be seperated.
9) The hole package should mainly be written in python, only then
everybody can change it to her/his will. And additions are easier,
because there is then one concept of plotting. So perhaps other
packages can be incorporated under the same concept or in clear
derivation of this concept. If there are performance problems these
can be addressed specifically through an extra c-module.
10) In the end of the queue the plotobjects should be in clear
data-structure, which can be saved for example or given to different
drivers. This data-structure is a class with some special dictionaries
for different plot-types and actions. To have here one standard form
of structuring is maybe a hart thing to achieve. The advantage is,
that new plotroutines must only produce such a structure, so they
don't need to interact with the driver directly. This reduces the
hacking with driver-tricks. (But everybody can use the driver-routines
for such tricks, if she/he is in a real need for this :-)
11) One additional drawback is the hole experience wich is incorporated in
other plotting-libraries. It's is not so easy to use one special
feature in one particular plotting-library, here is very much work in
redeveloping the circle. I don't know much about VTK for exapmle, but
I think all these great routines can't easyly be imported in this
framework, because VTK has it's one framework. But I think at the
moment an arriving VTK-module will not address the basic plotting
needs. And if we have general plot-objects, it should be possible
to write convertors to VTK-data-objects.
So, this is the end of the first part, which is mainly to bring some
things up for discussion. I will try to develop some ideas for the
system itself, but this will take some time, but the discussion can
start now.
__Janko
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From Dale_Bertrand@brown.edu Tue Jul 15 22:50:46 1997
From: Dale_Bertrand@brown.edu (Dale Bertrand)
Date: Tue, 15 Jul 1997 17:50:46 -0400 (EDT)
Subject: [MATRIX-SIG] Numeric Error
Message-ID: <XFMail.970715175433.Dale_Bertrand@brown.edu>
I've written a c function which returns a string representing 35
doubles. I wish to convert this string into a Numeric array for further
processing. Unfortunately, Numeric dumps a core when I call
Numeric.fromstring. I am sure the string is the right lenghth and have
no idea what is going on.
Shouldn't Numeric accept the data as long as it's the right length? Is
there something I'm missing?
Thanks in advance.
-- code --
f =
"/vision/medprojects/crisco/shocks/quad1-surfaces/quad1-0.00--0.50-00%d.raw"
% i
print "filename: ", f
shocks_string = sd.detect(f,135,135)
print "length of string: ", len(shocks_string)
print "string: ", shocks_string
aa = Numeric.fromstring(shocks_string, Numeric.Float)
print aa
-- output --
filename:
/vision/medprojects/crisco/shocks/quad1-surfaces/quad1-0.00--0.50-001.raw
length of string: 280
string: @G ^m@65*ðh?ðHfãQæ@JpnäN@cÕ
P¶ôÀH>sOz?¬@2aEaJ@[s?ónè(ÀCÍ¢ÆoÀ@U=lQAzÀdyÔ
Ø@[@@[¹ d@[q\ãOñø?ödW§à<ÀbÊJ%I8À/Ú¾¿î¢@RÒ/AÇxô?ø(1ÈL_@Udµ$
o@6À?Ó?ð;±ªú@aÏ_éÉÀ]þÔµ@DJ°¨¶@GöfáAü@5ñ|¬±Ú?ðÑe@/M(@evÚàÌ^À3u\ÖO
Segmentation fault (core dumped)
dcb@lems61 396 >
-- sig --
Dale Bertrand
(dcb@lems.brown.edu)
----------------------------------
E-Mail: Dale Bertrand <Dale_Bertrand@brown.edu>
Date: 15-Jul-97
----------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hyoon@nyptsrv1.etsd.ml.com Wed Jul 16 02:51:13 1997
From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant)
Date: Tue, 15 Jul 1997 21:51:13 -0400
Subject: [MATRIX-SIG] Calculating correlation Matrix
Message-ID: <199707160151.VAA00960@ptq3.etsd>
Hello,
The following should be fairly simple, but I am having problems
with it for some reason. I am attempting to calulate correlation
matrix of 2 D returns, but I cannot get a identitity diagnonal
using following code. I been messing around it for quite a while,
but can't figure out why meanc fails to begin with. It shoudl have
something to do with 2 D matricies I am used to vs more advanced
n-D in NumPy. I thought I made approapriate adj, but still doesn't
work.
Also, if anyone know of good stat func for NumPy, I would really
appreciate the source.
Thanks much in advance,
Hoon,
(the Baffled)
* none of these works for complex number, because I don't use them.
def vcm(x):
# var/cov matrix from a data matrix
return moment(x-transpose(meanc(x))) / (len(x)-1)
def moment(y):
return dot(transpose(y),y)
def meanc(x):
return up_1d(add.reduce(x) / len(x))
def varianc(x):
return up_1d((add.reduce((x-meanc(x))**2)/(len(x)-1)) ** .5)
def stdc(x):
var = varianc(x)
return up_1d(var ** .5)
def corrx(x):
vc = vcm(x)
std = up_1d(diagonal(vc))
return dot(vc,(1/dot(std, transpose(std))))
def up_1d(x):
if len(x.shape) == 1:
return reshape(x, (len(x),1))
else:
return x
ln_gold = log(gold_a[1:,:]/gold_a[:-1,:])
corr = corrx(ln_gold)
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From janne@avocado.pc.helsinki.fi Wed Jul 16 09:43:25 1997
From: janne@avocado.pc.helsinki.fi (Janne Sinkkonen)
Date: 16 Jul 1997 11:43:25 +0300
Subject: [MATRIX-SIG] Calculating correlation Matrix
In-Reply-To: hyoon@nyptsrv1.etsd.ml.com's message of "Tue, 15 Jul 1997 21:51:13 -0400"
References: <199707160151.VAA00960@ptq3.etsd>
Message-ID: <oavi2bv13l.fsf@avocado.pc.helsinki.fi>
hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) writes:
> Also, if anyone know of good stat func for NumPy, I would really
> appreciate the source.
> Thanks much in advance,
Here are some elementary functions including moments (and correlation
matrices) and something to help you with bootstrapping.
# This is from Konrad Hinsen.
import Numeric
def histogram(data, nbins, range = None):
data = Numeric.array(data)
if range is None:
min = Numeric.minimum.reduce(data)
max = Numeric.maximum.reduce(data)
else:
min, max = range
data = Numeric.repeat(data,
Numeric.logical_and(Numeric.less_equal(data, max),
Numeric.greater_equal(data,
min)))
bin_width = (max-min)/nbins
data = Numeric.array(Numeric.floor((data - min)/bin_width), Numeric.Int)
histo = Numeric.add.reduce(Numeric.equal(
Numeric.arange(nbins)[:,Numeric.NewAxis], data), -1)
bins = min + bin_width*(Numeric.arange(nbins)+0.5)
return Numeric.transpose(Numeric.array([bins, histo]))
# I use these as well
from Numeric import *
def mean(X):
N=X.shape[0]
return sum(X)/N
def variance(X):
N=X.shape[0]
return sum(X**2)/N-(sum(X)/N)**2
def sample_variance(X):
N=X.shape[0]
return (sum(X**2)-(sum(X)**2)/N)/(N-1)
def std(X):
return sqrt(variance(X))
def sample_std(X):
return sqrt(sample_variance(X))
def covariance(X):
N=X.shape[0]
Xt=transpose(X)
mX=sum(X)/N
# Note that dot() equals to sum(multiply.outer()).
# The latter is more memory-hungry.
return dot(Xt,X)/N-multiply.outer(mX,mX)
def covariance2(X,Y):
N,M=X.shape[0],Y.shape[0]
Xt=transpose(X)
mX,mY=sum(X)/N,sum(Y)/N
return dot(Xt,Y)/N-multiply.outer(mX,mY)
def correlation(X):
C=covariance(X)
V=diagonal(C)
return C/sqrt(multiply.outer(V,V))
def correlation2(X,Y):
C,VX,VY=covariance2(X,Y),variance(X),variance(Y)
return C/sqrt(multiply.outer(VX,VY))
def normalize(a):
m=mean(a)
s=std(a)
return (a-m)/s, m, s
def normalize_from(a,b):
m=mean(b)
s=std(b)
return (a-m)/s
# Euclidean distances
def squared_distances(X,Y):
return add.outer(sum(X*X,-1),sum(Y*Y,-1))- 2*dot(X,transpose(Y))
# Some functions to make bootstrapping easier.
from Numeric import *
from RandomArray import *
import string
import simpleStat
def bsIndices(n):
"Returns bootstrap indices for a sample of size N."
r=randint(0,n,n)
# The modulus is needless for future versions of Numpy
# and even now for (all?) non-Inter platforms.
return r%n
def bsVector(a):
"Returns the bootstrap vector generated from the bootstrap INDICES."
# This looks ugly but it is (relatively) fast.
b=concatenate((searchsorted(sort(a),arange(len(a))),(len(a),)))
return (b[1:]-b[:-1]).astype(Float)/len(a)
def bsStd(sample,estimator,n):
"Based on SAMPLE, estimates standard deviation of ESTIMATOR
using N bootstrap repetitions."
estimates=[]
for k in xrange(n):
ii=bsIndices(len(sample))
estimates.append(estimator(take(sample,ii)))
r=array(estimates)
return (simpleStat.mean(r),
simpleStat.sample_std(r))
def bsDistribution_w(sample,w_estimator,n):
"Based on SAMPLE, returns samples from the distribution of W_ESTIMATOR
using N bootstrap repetitions. W_ESTIMATOR takes the sample
and weights as its arguments."
estimates=[]
l=len(sample)
even=ones(l,Float)/l
for k in xrange(n):
ii=bsIndices(l)
estimates.append(w_estimator(take(sample,ii),even))
r=array(estimates)
return r
def bsStdBias(sample,w_estimator,n):
estimates=[]
l=len(sample)
B=zeros(l,Float)
even=ones(l,Float)/l
for k in xrange(n):
ii=bsIndices(l)
B=B+bsVector(ii)
estimates.append(w_estimator(take(sample,ii),even))
r=sort(array(estimates))
return (simpleStat.mean(r),
simpleStat.sample_std(r),
simpleStat.mean(r)-w_estimator(sample,B/n),
r[int(len(r)/2)],
r[int(3*len(r)/4)]-r[int(1*len(r)/4)])
--
Janne Sinkkonen <janne@iki.fi> <URL: http://www.iki.fi/~janne/ >
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From ghughes@cs.ucsb.edu Mon Jul 21 06:14:55 1997
From: ghughes@cs.ucsb.edu (Graham Hughes)
Date: Sun, 20 Jul 1997 22:14:55 -0700 (PDT)
Subject: [MATRIX-SIG] UserArray bug
Message-ID: <19970721051513.31201.qmail@a-abe.resnet.ucsb.edu>
--8323328-1804289383-869462113=:31199
Content-Type: TEXT/plain; CHARSET=US-ASCII
I noticed this bug when I was trying to get a hashable array with
UserArray:
>>> class ArrayWithHash(UserArray.UserArray):
... def __hash__(self):
... return hash(tuple(self))
...
>>> foo = ArrayWithHash([[ 0, 3, 1, 5, 3],
... [ 0, -2, 0, -1, 0],
... [ 0, 2, 0, 5, 0],
... [ 6, 3, 0, 3, 0],
... [ 0, 0, 0, -1, 0]])
>>> bar = ArrayWithHash([0, 0, 1, 0, 0])
>>> bar[2] = foo[0,2]
Traceback (innermost last):
File "<stdin>", line 1, in ?
File "/usr/local/lib/site-python/UserArray.py", line 36, in __setitem__
def __setitem__(self, index, value): self.array[index] = \
asarray(value,self.typecode)
TypeError: illegal argument type for built-in operation
>>>
(traceback broken manually)
This sort of behaviour isn't terribly helpful for me, because I really
need to be able to stick arrays in dictionaries, and I also need for
that code to work.
Suggestions?
--
Graham Hughes <ghughes@cs.ucsb.edu> MIME OK, PGP preferred
from stddisclaim import footer
pgp_fingerprint = "E9 B7 5F A0 F8 88 9E 1E 7C 62 D9 88 E1 03 29 5B"
--8323328-1804289383-869462113=:31199
Content-Type: APPLICATION/pgp-signature
-----BEGIN PGP MESSAGE-----
Version: 2.6.3
iQCVAwUBM9LwYCqNPSINiVE5AQHchgQAhLRNyZ0gnzMGk5NCNo7OSngSK8Q8gy4d
h+do4J6wFgvyULFjzx8z6wUNQSS+heF5nunGSUMUl2LmxEymiTn7BQ75H10v6Lms
y6mnBNiAwMPTop8Zno1JbF8qD9HAD8eD1gAgda6iOaaTCI/iZIOsYoptt4Jm34rd
/PC3ilj5oZk=
=H8e0
-----END PGP MESSAGE-----
--8323328-1804289383-869462113=:31199--
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Mon Jul 21 11:50:10 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Mon, 21 Jul 1997 12:50:10 +0200
Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take"
revisited
In-Reply-To: <3.0.1.32.19970711232606.00804ce0@popd.netcruiser>
(amullhau@ix.netcom.com)
Message-ID: <199707211050.MAA23915@lmspc1.ibs.fr>
> >3) generate an error. This is the behavior of take() when the index is
> >out of bounds.
>
> Good old APL style. This has the virtue that even if it isn't making you
> happy, it's easy to understand and expect. You could do worse.
Especially considering that
1) It is easy enough to do clipping etc. as an explicit separate step.
2) Python's exception handling means that an error is not an
intractable problem.
So I vote for this solution. In fact, I can't say I am too happy about the
clipping behaviour for slices, but there was little choice given that
clipping is part of the general sequence protocol.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Mon Jul 21 13:32:27 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Mon, 21 Jul 1997 14:32:27 +0200
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <m0wo7WP-000syBC@lisboa.ifm.uni-kiel.de> (message from Janko
Hauser on Tue, 15 Jul 1997 15:20:05 +0200 (CEST))
Message-ID: <199707211232.OAA24182@lmspc1.ibs.fr>
> First some of the things I think a ubiquitous plotpackage should cover:
>...
That's a long list, although of course all that is desirable. But
we need something shorter for getting started!
So let's look at this from an implementation point of view and see
what is essential and how much development can be parallelized.
An essential part seems to be the intermediate representation of
"graphics" (which needn't be limited to plots) in terms of elementary
objects such as lines, markers, circles etc. This must be defined
first. Then work can progress in two directions:
1) Low-level drivers, which turn the intermediate representation
into images, Tk objects, PostScript files, etc. They would also
take care of animation and editing (where it makes sense).
2) High-level plot-to-graphics code. This would consist of one
module per type of plot, and the modules would to a large
degree be independent of each other.
For plain plotting and also animation, the interfaces between the
modules seem straightforward. Editing, on the other hand, is
not so trivial, since the high-level plot objects must be able
to turn graphical information back into numerical data, which is
not always possible, at least not uniquely (think about editing
a contour plot!). Inquiry (getting the value for the point under
the mouse cursor, for example) is somewhere in between.
> 1) Use standard or at least very portable graphics-libraries: If the
> effort is done to build a new plot-system it should be availabel with
> some drivers on all platforms, where Numpy is availabel. I think the
My first attempt at a general low-level driver would be Tk canvases,
since Tk is widely supported. Together with PostScript for printing,
this would already make many people happy. Next is OpenGL, and then
the system-specific libraries.
But now for the really important question: do we have enough manpower
to complete such a project to a useful stage in a reasonable amount of
time? Yes, that's a call for volunteers. Starting with myself, I would
be able to do some useful work, but not necessarily on a predictable
time scale.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From furnish@acl.lanl.gov Mon Jul 21 15:56:55 1997
From: furnish@acl.lanl.gov (Geoffrey Furnish)
Date: Mon, 21 Jul 1997 08:56:55 -0600 (MDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <199707211232.OAA24182@lmspc1.ibs.fr>
References: <m0wo7WP-000syBC@lisboa.ifm.uni-kiel.de>
<199707211232.OAA24182@lmspc1.ibs.fr>
Message-ID: <199707211456.IAA03860@steam.acl.lanl.gov>
Konrad Hinsen writes:
> > First some of the things I think a ubiquitous plotpackage should cover:
> >...
>
> That's a long list, although of course all that is desirable. But
> we need something shorter for getting started!
>
> So let's look at this from an implementation point of view and see
> what is essential and how much development can be parallelized.
>
> An essential part seems to be the intermediate representation of
> "graphics" (which needn't be limited to plots) in terms of elementary
> objects such as lines, markers, circles etc. This must be defined
> first. Then work can progress in two directions:
>
> 1) Low-level drivers, which turn the intermediate representation
> into images, Tk objects, PostScript files, etc. They would also
> take care of animation and editing (where it makes sense).
>
> 2) High-level plot-to-graphics code. This would consist of one
> module per type of plot, and the modules would to a large
> degree be independent of each other.
>
> For plain plotting and also animation, the interfaces between the
> modules seem straightforward. Editing, on the other hand, is
> not so trivial, since the high-level plot objects must be able
> to turn graphical information back into numerical data, which is
> not always possible, at least not uniquely (think about editing
> a contour plot!). Inquiry (getting the value for the point under
> the mouse cursor, for example) is somewhere in between.
>
> > 1) Use standard or at least very portable graphics-libraries: If the
> > effort is done to build a new plot-system it should be availabel with
> > some drivers on all platforms, where Numpy is availabel. I think the
>
> My first attempt at a general low-level driver would be Tk canvases,
> since Tk is widely supported. Together with PostScript for printing,
> this would already make many people happy. Next is OpenGL, and then
> the system-specific libraries.
There are widespread complaints about the run-time performance of Tk
canvases. This figured prominently in the decision to make a seperate
Tk-aware plotting widget in PLplot. Besides our own experiences, I
have heard many other people complain about the performance of Tk
canvases (in contexts other than scientific plotting). Also, besides
performance, you also face color management issues if you adopt Tk
canvas. My opinion, and I think this is reflected in the
architectures of other plotting packages as well, is that good
plotting packages need to be able to manage their own color maps.
Cooperative behavior is good to avoid "color flashing", but at some
level, good rendering depends on good color control.
> But now for the really important question: do we have enough manpower
> to complete such a project to a useful stage in a reasonable amount of
> time? Yes, that's a call for volunteers. Starting with myself, I would
> be able to do some useful work, but not necessarily on a predictable
> time scale.
It seems to me that all this discussion about plotting is somewhat
orthogonal to the issues involved with making a good matrix facility.
Plotting is a natural client of a matrix facility, but is otherwise
independent. I don't see major issues in plotting which can
reasonably be expected to strongly influence matrix design or
implementation issues.
I wonder if it might be more appropriate to initiate a plotting-sig,
or a "scientific environment sig", or some such, and move the plotting
discussion which is not specifically matrix-sig-design-influencing,
off to said new sig.
I am /not/ volunteering to be the sig owner, but I do think it would
be better to segregate these discussions in that way. Opinions?
--
Geoffrey Furnish email: furnish@lanl.gov
LANL CIC-19 POOMA/RadTran phone: 505-665-4529 fax: 505-665-7880
"Here are your ball-peen hammers. Now go malleate some heads!" -Jim Morel
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From furnish@acl.lanl.gov Mon Jul 21 17:46:06 1997
From: furnish@acl.lanl.gov (Geoffrey Furnish)
Date: Mon, 21 Jul 1997 10:46:06 -0600 (MDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <Pine.SGI.3.95q.970721121848.20632B-100000@maigret>
References: <199707211456.IAA03860@steam.acl.lanl.gov>
<Pine.SGI.3.95q.970721121848.20632B-100000@maigret>
Message-ID: <199707211646.KAA06794@steam.acl.lanl.gov>
David Ascher writes:
> Do we need to discuss this on the meta-sig, as for new sigs, or does this
> count as a splinter group (faction? =))?
You have to go through meta-sig eventually, in order to get the PSA
folks to actually instantiate the list. But there is no harm in
hashing out the basic plan here, so that there is a unified and
basically community-accepted plan by the time you take it to meta-sig,
especially since it seems likely that the new list will draw most of
its (at least initial) participation from matrix-sig.
At this point I think we need to resolve two things:
1) What is the name of the proposed new sig. Is it plot-sig, or
sci-sig. This obviously impacts the sig charter and expected usage
significantly.
2) Who is the owner going to be? Its not a particularly big job, but
it should be someone who cares enough to keep things in order if
need be.
My view on 1) is that a plot sig would be thoroughly adjunct, whereas
a sci-sig might hijack matrix-sig. If the new sig ended up absorbing
all trafic from matrix sig, then there is no point in making the new
sig, and good reason not to do it. A plotting sig would draw off only
the plotting discussion from matrix-sig, and would not be a threat to
the existence or continued operation of matrix-sig. A sci-sig might,
I'm not sure. Others should voice opinion on that concern. Maybe its
a non issue, I'm just not sure.
Not that it would necessarily be inherently bad if sci-sig superceded
matrix-sig. That might be a reasonable progression. But whatever
happens should happen by design and because people want things to move
that way, rather than by unfortuitous happenstance.
My view on 2) is that it should be anybody but me. I'm already
running a different sig, and I'm too vested in PLplot. A sig should
be moderator-neutral.
--
Geoffrey Furnish email: furnish@lanl.gov
LANL CIC-19 POOMA/RadTran phone: 505-665-4529 fax: 505-665-7880
"Here are your ball-peen hammers. Now go malleate some heads!" -Jim Morel
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Mon Jul 21 17:33:13 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Mon, 21 Jul 1997 12:33:13 -0400 (EDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <199707211456.IAA03860@steam.acl.lanl.gov>
Message-ID: <Pine.SGI.3.95q.970721121848.20632B-100000@maigret>
Geoffrey Furnish writes:
> It seems to me that all this discussion about plotting is somewhat
> orthogonal to the issues involved with making a good matrix facility.
> Plotting is a natural client of a matrix facility, but is otherwise
> independent. I don't see major issues in plotting which can
> reasonably be expected to strongly influence matrix design or
> implementation issues.
>
> I wonder if it might be more appropriate to initiate a plotting-sig,
> or a "scientific environment sig", or some such, and move the plotting
> discussion which is not specifically matrix-sig-design-influencing,
> off to said new sig.
>
> I am /not/ volunteering to be the sig owner, but I do think it would
> be better to segregate these discussions in that way. Opinions?
Sure, seems like a fine idea. I do wonder where some things like the
S-like data.frame project which has been bandied about fit in -- it ties
in to the Numeric array as a backend mostly, and it certainly fits within
the 'scientific environment' concept, but it shares some interest in
potential python changes with the numpy project, e.g. on the indexing
issue. I guess I'd say we should dicuss that on the sci-sig, since it's
really not an issue for many NumPy users. As long as the plotting folk
don't mind a bit of discussion that they might not find interesting.
On Konrad's call for implementation volunteers, I'd love to help but
committing my time in significant chunks for the next few months would be
foolish. I could probably deal with owning a sig, but that's about it
given the other things on my plate.
Do we need to discuss this on the meta-sig, as for new sigs, or does this
count as a splinter group (faction? =))?
--da
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Mon Jul 21 23:56:32 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Mon, 21 Jul 1997 18:56:32 -0400 (EDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <199707211646.KAA06794@steam.acl.lanl.gov>
Message-ID: <Pine.SGI.3.95q.970721183712.21284B-100000@maigret>
On Mon, 21 Jul 1997, Geoffrey Furnish wrote:
> 1) What is the name of the proposed new sig. Is it plot-sig, or
> sci-sig. This obviously impacts the sig charter and expected usage
> significantly.
Ok, let's play it safe. As per the SIG creation guidelines, we need a SIG
name, short description, and longer blurb. Here's a first draft, please
critique it.
PLOT-SIG: Investigating and Developing Scientific and Business Plotting
Solutions for Python
Short blurb:
The purpose of this list is to develop, coddle together, adopt or
otherwise make available a package for plotting of scientific and
business plots of data.
Principles guiding the development of the software will include:
* Portability to the relevant platforms
* Ease of use
* Integration with other Python packages (e.g. NumPy)
* Quality of the software
* Quality of the output
----
This is meant to be pretty much independent of existing solutions,
although clearly some existing packages will fit more or less well. I'm
concerned that plot-sig will suffer the fate of gui-sig, which was to die
of starvation because the job at hand is just too big to start from
scratch, so people just keep using what they've been using, and no 'ideal'
solution comes to fruition.
I also wanted to add the principle of "object oriented design" to the list
above, but didn't know if that would get a consensus.
Regardless, I think it might be a good thing to come up with a list of
features each of us need, and rank them in terms of 'must have', 'want',
'would be nice'. Maybe this should wait until the sig is created, I don't
know.
Here's mine (off the top of my head):
MUST HAVE:
Line Plots
Point Plots (scatter plots)
Bar Plots
Error Bars
Fills (grey levels)
Symbols
Log scales
Bitmap preview (tk, gif, whatever)
Text
Production-quality PostScript output
Very Customizable
Works on Unix & NT
WOULD BE NICE:
Complete Python control over graph generation, graph objects, etc.
Ability to design new graph formats (e.g. Tufte's work)
Interactivity
Ability to do Batch work
Here's what I've used in the past (or use now):
Gnuplot
GLE
MetaPost
[mac GUI tools]
> 2) Who is the owner going to be? Its not a particularly big job, but
> it should be someone who cares enough to keep things in order if
> need be.
I guess I just volunteered.
We can keep the data.frame discussion (which is pretty small and probably
doesn't require a sig) on matrix-sig for now.
--da
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From singer@itl.atr.co.jp Mon Jul 21 00:41:56 1997
From: singer@itl.atr.co.jp (Harald Singer)
Date: Mon, 21 Jul 1997 08:41:56 +0900
Subject: [MATRIX-SIG] [Q] numerical python on SunOS and dlopen
Message-ID: <33D2A244.79EE6774@itl.atr.co.jp>
We just discovered numerical and are thrilled. I installed NumPy1.0b3
on HP-UX, OSF1 and Linux. We only have a problem with
SunOS(SS5/110(sun4m), SunOS4.1.4, 32MB RAM. gcc version 2.7.2.).
When we try to compile NumPy on SunOS as a shared library nearly
everything is OK. But when we try to use more than one of these
modules we get
ImportError: ld.so: conflicting usage of dlopen'ed dependents
SS5/110(sun4m), SunOS4.1.4, 32MB RAM
gcc version 2.7.2
riwa@als01<111> cd ~/NumPybuild
riwa@als01<112> mkdir SunOS
riwa@als01<113> cd SunOS
riwa@als01<114> lndir ../NumPy .
riwa@als01<115> make installdir=/usr/local/lang/python-1.4 -f Makefile.pre.in Makefile
riwa@als01<116> make CCSHARED=" -fPIC" CFLAGS="-ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H"
gcc -fPIC -ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H -I./Include -c ./arrayobject.c
gcc -fPIC -ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H -I./Include -c ./ufuncobject.c
ld arrayobject.o ufuncobject.o -o libnumpymodule.so
gcc -fPIC -ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H -I./Include -c ./multiarraymodule.c
ld multiarraymodule.o -L. -lnumpymodule -o multiarraymodule.so
ld: -lnumpymodule: No such file or directory
make: *** [multiarraymodule.so] Error 4
riwa@als01<117> pwd
/home/atrh31/riwa/NumPybuild/SunOS
riwa@als01<118> cp libnumpymodule.so libnumpymodule.so.1.0
riwa@als01<119> make CCSHARED=" -fPIC" CFLAGS="-ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H"
ld multiarraymodule.o -L. -lnumpymodule -o multiarraymodule.so
ld multiarraymodule.o -L. -lnumpymodule -o multiarraymodule.so
gcc -fPIC -ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H -I./Include -c ./fast_umathmodule.c
ld fast_umathmodule.o -L. -lnumpymodule -o fast_umathmodule.so
gcc -fPIC -ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H -I./Include -c ./umathmodule.c
ld umathmodule.o -L. -lnumpymodule -o umathmodule.so
....
riwa@als01<131> setenv PYTHONPATH "$HOME/NumPybuild/$OS:$HOME/NumPybuild/$OS/NumPy"
riwa@als01<132> setenv LD_LIBRARY_PATH "$LD_LIBRARY_PATH":$PYTHONPATH
riwa@als01<139> python
Python 1.4 (Jan 9 1997) [GCC 2.7.2]
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import multiarray
>>> import umath
Traceback (innermost last):
File "<stdin>", line 1, in ?
ImportError: ld.so: conflicting usage of dlopen'ed dependents
>>> ^D
We also try to link using "gcc -shared".
riwa@als01<255> gcc -shared arrayobject.o ufuncobject.o -o libnumpymodule.so
riwa@als01<256> cp libnumpymodule.so libnumpymodule.so.1.0
overwrite libnumpymodule.so.1.0? y
riwa@als01<257> gcc -shared multiarraymodule.o -L. -lnumpymodule -o multiarraymodule.so
riwa@als01<258> gcc -shared umathmodule.o -L. -lnumpymodule -o umathmodule.so
riwa@als01<259> python
Python 1.4 (Jan 9 1997) [GCC 2.7.2]
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import multiarray
>>> import umath
Traceback (innermost last):
File "<stdin>", line 1, in ?
ImportError: ld.so: conflicting usage of dlopen'ed dependents
>>> ^D
Any ideas? I would be grateful for any pointers.
--
Harald Singer, ATR ITL, tel +81-7749-5-1389 singer@itl.atr.co.jp
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From raj@grserv.med.jhmi.edu Tue Jul 22 01:21:02 1997
From: raj@grserv.med.jhmi.edu (RajGopal Srinivasan)
Date: Mon, 21 Jul 1997 20:21:02 -0400 (EDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <Pine.SGI.3.95q.970721183712.21284B-100000@maigret>
Message-ID: <Pine.SGI.3.94.970721201504.7405A-100000@cherubino>
On Mon, 21 Jul 1997, David Ascher wrote:
> On Mon, 21 Jul 1997, Geoffrey Furnish wrote:
>
> > 1) What is the name of the proposed new sig. Is it plot-sig, or
> > sci-sig. This obviously impacts the sig charter and expected usage
> > significantly.
>
> Ok, let's play it safe. As per the SIG creation guidelines, we need a SIG
> name, short description, and longer blurb. Here's a first draft, please
> critique it.
>
> PLOT-SIG: Investigating and Developing Scientific and Business Plotting
> Solutions for Python
>
>
This might be of intereset to the IMAGE-SIG, given that there
now is a package 'pilplot' that may be in development -
URL http://hem1.passagen.se/eff/pil/index.htm
raj
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Tue Jul 22 01:36:01 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Mon, 21 Jul 1997 20:36:01 -0400
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <Pine.SGI.3.95q.970721183712.21284B-100000@maigret>
References: <199707211646.KAA06794@steam.acl.lanl.gov>
Message-ID: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser>
I've been watching the discussion here, and it seems fine to split
plotting away as a discussion from arrays, except for one thing.
Maybe some kinds of plots are pretty hard to do without the array
objects.
I have in mind things like contour plots (traditional and color) and
three (or more) dimensional scatterplots.
Now Python is reasonably quick as interpreters go, but I think building
the plotting package on top of graphics primitives and the array operations
will be easier and lead to a better result that just doing the plotting.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Tue Jul 22 04:42:20 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Mon, 21 Jul 1997 23:42:20 -0400 (EDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser>
Message-ID: <Pine.SGI.3.95q.970721232455.21602D-100000@maigret>
On Mon, 21 Jul 1997, Andrew P. Mullhaupt wrote:
> I've been watching the discussion here, and it seems fine to split
> plotting away as a discussion from arrays, except for one thing.
>
> Maybe some kinds of plots are pretty hard to do without the array
> objects.
Definitely -- it's just that we can define those operations using NumPy
arrays without having to modify the architecture of NumPy, hence it need
not be discussed on the same SIG, or so I take Geoffrey's point. But yes,
NumPy should definitely be involved for contour plots, surface plots, 3d
projections onto 2d, line-fitting & interpolation, etc.
On the other point, my forgetting to mention PIL was simply that, a
temporary lapse of comprehensiveness -- PIL is definitely something which
should be taken into consideration, along with PLPlot, Narcisse, Gist,
Gnuplot, OpenGL, Tk, GLUT, etc. etc. All of these have problems and
features. How much we can use existing work and how much we have to do
from scratch is up for discussion, and will depend on too many factors to
list. Arrays I'm pretty sure we'll need, for the reasons mentioned by
Andrew among others, and at least we know what code to use for that part.
I've looked a bit at the architecture behind a few packages (gnuplot, gle,
gist, etc.), and some design decisions are clear winners, while others
clearly aren't. I feel it would behoove us to learn from history, should
a redesign be deemed necessary. The distinction between graphics engine
and plot driver allows maximum portability with potentially very
distributed workloads (you do Xterms, I do postscript). That driver
definition can be extremely limiting if it's not done carefully, and with
enough extensability in mind (more a problem in static languages). And so
on (look at how few packages handle interacting with graphics so as to
allow exploratory data analysis (see S+ for an example, DataDesk for
another, maybe StatLisp as well)). But all this should probably wait until
the SIG creation decision has been reached.
can-you-tell-i'm-unhappy-with-every-single-plotting-package-i've-tried-
and-i've-tried-a-lot-ly yours, --da
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Tue Jul 22 05:07:40 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Tue, 22 Jul 1997 00:07:40 -0400
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <Pine.SGI.3.95q.970721232455.21602D-100000@maigret>
References: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser>
Message-ID: <3.0.1.32.19970722000740.008142c0@popd.netcruiser>
At 11:42 PM 7/21/97 -0400, David Ascher wrote:
>On Mon, 21 Jul 1997, Andrew P. Mullhaupt wrote:
>
>> Maybe some kinds of plots are pretty hard to do without the array
>> objects.
>
>Definitely -- it's just that we can define those operations using NumPy
>arrays without having to modify the architecture of NumPy, hence it need
>not be discussed on the same SIG, or so I take Geoffrey's point.
If you read it that way, then fine.
> But yes,
>NumPy should definitely be involved for contour plots, surface plots, 3d
>projections onto 2d, line-fitting & interpolation, etc.
Just about any plot with a large data object is either handing an image in
a special graphics format off to a special graphics function or else you
would prefer that that large object had some array operations.
>can-you-tell-i'm-unhappy-with-every-single-plotting-package-i've-tried-
>and-i've-tried-a-lot-ly yours, --da
Can I have a witness? Amen, brother.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From Daniel.Michelson@smhi.se Tue Jul 22 07:52:51 1997
From: Daniel.Michelson@smhi.se (Daniel Michelson)
Date: Tue, 22 Jul 1997 08:52:51 +0200
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
References: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser> <3.0.1.32.19970722000740.008142c0@popd.netcruiser>
Message-ID: <33D458C3.28D9@smhi.se>
Hej,
I've also been monitoring the plotting discussions and can't quite grasp
why the xmgr package has not been mentioned in this context. As free
scientific plotting packages go, I find it quite complete and versatile,
although it, as all others, does have a number of limitations. I've
built a v e r y primitive interface to our Python-based weather radar
software using xmgr.
If it's worth looking into, do so at
http://plasma-gate.weizmann.ac.il/Xmgr/
This having been said, I use unix and xmgr is designed for X. I can't
say if/how it could be made more universal in a way which would suit the
entire Python community. Have others already written off xmgr because of
this?
I would certainly support a NumPy- or PIL-based plotting module, for
reasons previously presented by others on this sig, if it is finally
determined that this is a wheel which would have to be reinvented in
order for it to be acceptable...
best, -d
+------------------+--------------------------------------------------+
| | Daniel B. Michelson |
| /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute|
| \ | \/ | |__| | | S-601 76 Norrk=F6ping, Sweden |=
| \ | | | | | | Telephone: +46 11 - 15 84 94 |
| \/ | | | | | | Telefax: +46 11 - 17 02 07 |
| | Internet: Daniel.Michelson@smhi.se |
+------------------+--------------------------------------------------+
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From Anthony Baxter <arb@connect.com.au> Tue Jul 22 08:30:38 1997
From: Anthony Baxter <arb@connect.com.au> (Anthony Baxter)
Date: Tue, 22 Jul 1997 17:30:38 +1000
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: Your message of "Tue, 22 Jul 1997 08:52:51 +0200."
<33D458C3.28D9@smhi.se>
References: <33D458C3.28D9@smhi.se> <3.0.1.32.19970721203601.0081bac0@popd.netcruiser> <3.0.1.32.19970722000740.008142c0@popd.netcruiser>
Message-ID: <199707220730.RAA05491@jambu.off.connect.com.au>
I've rolled my own 2D plotting code in PIL (using /F's pilplot as a start).
It's not a fun game.
Has anyone thought about interfacing to the GNUplot-in-a-library? (gplotlib)
Anthony
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From R.Hooft@EuroMail.com Tue Jul 22 09:02:36 1997
From: R.Hooft@EuroMail.com (Rob Hooft)
Date: Tue, 22 Jul 1997 10:02:36 +0200 (MET DST)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <33D458C3.28D9@smhi.se>
References: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser>
<3.0.1.32.19970722000740.008142c0@popd.netcruiser>
<33D458C3.28D9@smhi.se>
Message-ID: <199707220802.KAA22657@nu>
>>>>> "DM" == Daniel Michelson <Daniel.Michelson@smhi.se> writes:
DM> Hej, I've also been monitoring the plotting discussions and can't
DM> quite grasp why the xmgr package has not been mentioned in this
DM> context.
One extra problem: It doesn't only need X, it needs Motif.....
As everyone else, I have written my own plotting package. The only
thing is: this was before the era of scripting languages. And WAY
before the era of NumPy..... So it is written in portable Fortran-77.
And no user-interface: just one command line. And guess what? It does
exactly what I want it to.... And all these features that all those
other people need are not in there.....
One thing I learned: do not write your own output. I can do a lot of
things with my plotting package because the output is generated in
Xfig code, and so there is an easy editor to change the detailed
lay-out of the plot. So although the lay-out choices of my plotting
program are very very basic, any user can make it into a super glossy
plot using xfig. And I didn't need to write all of the graphics
primitives for that!
I know nobody is interested in a plotting package like that.....
http://www.sander.embl-heidelberg.de/extern/scatter/
Rob
--
===== R.Hooft@EuroMail.com http://www.Sander.EMBL-Heidelberg.DE/rob/ ==
==== In need of protein modeling? http://www.Sander.EMBL-Heidelberg.DE/whatif/
Validation of protein structures? http://biotech.EMBL-Heidelberg.DE:8400/ ====
== PGPid 0xFA19277D == Use Linux! Free Software Rules The World! =============
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From steve@estel.uindy.edu Tue Jul 22 13:02:11 1997
From: steve@estel.uindy.edu (Steve Spicklemire)
Date: Tue, 22 Jul 97 07:02:11 -0500
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <Pine.SGI.3.95q.970721232455.21602D-100000@maigret> (message from David Ascher on Mon, 21 Jul 1997 23:42:20 -0400 (EDT))
Message-ID: <9707221202.AA11090@estel.uindy.edu>
I would certainly vote for SIG creation...
-steve
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From bvelasco@etsd.ml.com Tue Jul 22 14:03:46 1997
From: bvelasco@etsd.ml.com (Bill Velasco)
Date: Tue, 22 Jul 1997 09:03:46 -0400
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
Message-ID: <199707221303.JAA02817@nyptsrv1.etsd.ml.com>
Hello,
Would like to vote for a SIG creation. I would attempt
to help, but, since I am not much of a programmer, I would
not be much of help in core developement. I would like to
keep in loop and help out in other ways if I can.
Plotting is one of the few holes python still has. I would
like to see mouse interacted command, x-y pinpointer with
axis guide lines, and eventloop interacted updates. I am
also attempting to get a commercial software to Python
interactions as a project, but this is becoming a slow
process. I will try to get the wrapper published, once
this project is done.
Hope this works.
Hoon,
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From bvelasco@etsd.ml.com Tue Jul 22 14:03:46 1997
From: bvelasco@etsd.ml.com (Bill Velasco)
Date: Tue, 22 Jul 1997 09:03:46 -0400
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
Message-ID: <199707221303.JAA02817@nyptsrv1.etsd.ml.com>
Hello,
Would like to vote for a SIG creation. I would attempt
to help, but, since I am not much of a programmer, I would
not be much of help in core developement. I would like to
keep in loop and help out in other ways if I can.
Plotting is one of the few holes python still has. I would
like to see mouse interacted command, x-y pinpointer with
axis guide lines, and eventloop interacted updates. I am
also attempting to get a commercial software to Python
interactions as a project, but this is becoming a slow
process. I will try to get the wrapper published, once
this project is done.
Hope this works.
Hoon,
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Tue Jul 22 14:49:34 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Tue, 22 Jul 1997 09:49:34 -0400 (EDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <199707220730.RAA05491@jambu.off.connect.com.au>
Message-ID: <Pine.SGI.3.95q.970722094312.22334B-100000@maigret>
> Has anyone thought about interfacing to the GNUplot-in-a-library? (gplotlib)
gplotlib is way out of date w/ respect to gnuplot 3.6 dev't. I've talked
to the folks who are working on gnuplot, and while they are interested in
libraryfying the recent work, it's a significant job, IMHO because of poor
design decisions very early on, regarding the breakdown of gnuplot into
subparts and an inflexible driver interface definition. In other words,
don't hold your breath. A lot of their code can be borrowed however, if
we don't mind the GPL.
-da
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Tue Jul 22 15:29:41 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Tue, 22 Jul 1997 10:29:41 -0400
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
Message-ID: <199707221426.KAA01970@dante.mh.lucent.com>
>This having been said, I use unix and xmgr is designed for X. I can't
>say if/how it could be made more universal in a way which would suit the
>entire Python community. Have others already written off xmgr because of
>this?
For the purposes of business software, I think that might be a pretty
good reason to write it off, unless there is some more general portability
available which doesn't present itself from the url ref.
Rob Hooft writes:
>One thing I learned: do not write your own output. I can do a lot of
>things with my plotting package because the output is generated in
>Xfig code,
>From another perspective you could say "have clear OO interfaces
to an abstract back end" much like the SGML parser in the standard
libs interfaces to an abstract formatter which can be plug replaced
with something that talks tk, flat text, wpy, whatever... The down side
is that a lot of hard thinking should be done up front to keep the
interface clean (if it's not done it'll get hacked up and horrible, I promise).
A couple notes on this thread: 2 back ends come to mind strongly
1) opengl
2) awt/jpi as outlined in http://www.ndim.edrc.cmu.edu/dougc/jpi/Home.html
Of course, when Jim's pyjava gets working that too.
-- Aaron Watters
ps: Is there a standard book on the topic of plotting? It might provide
a framework for discussion.
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From motteler@laura.llnl.gov Tue Jul 22 15:36:57 1997
From: motteler@laura.llnl.gov (Zane C. Motteler)
Date: Tue, 22 Jul 1997 07:36:57 -0700 (PDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <199707220730.RAA05491@jambu.off.connect.com.au>
Message-ID: <Pine.HPP.3.95.970722072428.2186A-100000@laura.llnl.gov>
Hi folks,
Just a reminder that we have a working, pretty decent 2d-3d scientific
graphics package working with Gist (a public domain graphics engine from
LLNL). The Python interface is object-oriented, consisting of Graph
objects, geometric objects which can be shipped to Graph objects,
and Plotter objects which can be used to plot them. This is pretty much
as I described in my talks at the last two Python meetings. The 3-D
graphics is much enhanced since I first talked about it at the
Washington meeting last year.
Our distribution is available at our public ftp site ftp-icf.llnl.gov
in /ftp/pub/python/Current_snapshot.tgz and is updated with our latest
changes each Friday night. Available documentation leaves something to
be desired, as our framemaker documentation has not yet been approved
for public release (but we hope it soon will be, and will be available
on the web).
I am making this reminder because I would hate to see somebody duplicate
all of my hard work. The high level graphics is intended to be
independent of the underlying graphics engine (inasmuch as possible).
One need only design an appropriate Plotter object and a Python
extension in order to expand its coverage to a new graphics engine.
Cheers
Zane
-----------------------------------------------------------------------------
Zane C. Motteler, Ph. D. ___________ALSO:_________________
Computer Scientist | Professor Emeritus of Computer|
Lawrence Livermore National Laboratory | Science and Engineering |
P O Box 808, L-472 | California Polytechnic State |
Livermore, CA 94551-9900 | University, San Luis Obispo |
510/423-2143 ---------------------------------
FAX 510/423-9969
zcm@llnl.gov or motteler@laura.llnl.gov or zmottel@calpoly.edu
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From furnish@acl.lanl.gov Tue Jul 22 15:44:26 1997
From: furnish@acl.lanl.gov (Geoffrey Furnish)
Date: Tue, 22 Jul 1997 08:44:26 -0600 (MDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <3.0.1.32.19970722000740.008142c0@popd.netcruiser>
References: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser>
<Pine.SGI.3.95q.970721232455.21602D-100000@maigret>
<3.0.1.32.19970722000740.008142c0@popd.netcruiser>
Message-ID: <199707221444.IAA11137@steam.acl.lanl.gov>
Andrew P. Mullhaupt writes:
> At 11:42 PM 7/21/97 -0400, David Ascher wrote:
> >On Mon, 21 Jul 1997, Andrew P. Mullhaupt wrote:
> >
> >> Maybe some kinds of plots are pretty hard to do without the array
> >> objects.
> >
> >Definitely -- it's just that we can define those operations using NumPy
> >arrays without having to modify the architecture of NumPy, hence it need
> >not be discussed on the same SIG, or so I take Geoffrey's point.
>
> If you read it that way, then fine.
David read my message the way I intended it to be read. My view is
that plotting will /use/ NumPy, but probably does not need to
influence the development of NumPy too much. I think it would be
desirable if we could move all this plotting talk off the matrix-sig.
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Tue Jul 22 16:03:15 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 22 Jul 1997 17:03:15 +0200
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <199707211646.KAA06794@steam.acl.lanl.gov> (message from Geoffrey
Furnish on Mon, 21 Jul 1997 10:46:06 -0600 (MDT))
Message-ID: <199707221503.RAA29121@lmspc1.ibs.fr>
> At this point I think we need to resolve two things:
>
> 1) What is the name of the proposed new sig. Is it plot-sig, or
> sci-sig. This obviously impacts the sig charter and expected usage
> significantly.
A while ago I discussed this a bit with Jim Hugunin. We agreed that
the Matrix-SIG had more or less done its job, so it could be
transformed into a more general "scientific" SIG that would build on
NumPy. Of course array-related questions would also go there. I still
think that this transformation is a good idea.
But the question is whether very specific discussions about specific
implementation problems should go to that SIG. It could quickly
become too large. So here's another proposal:
- Let's have a Sci-SIG for general discussions about scientific
applications, including discussions of features of proposed
packages etc., and also announcements of test versions.
- Create separate mailing lists (not formal SIGs) for communication
between the people working in specific code, e.g. plotting or
array indexing.
Does this sound reasonable?
> 2) Who is the owner going to be? Its not a particularly big job, but
> it should be someone who cares enough to keep things in order if
> need be.
It seems to me that a SIG owner has very little work to do, so that
shouldn't be a problem. The real question is: Who will write the code? ;-)
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From furnish@acl.lanl.gov Tue Jul 22 16:04:32 1997
From: furnish@acl.lanl.gov (Geoffrey Furnish)
Date: Tue, 22 Jul 1997 09:04:32 -0600 (MDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <Pine.SGI.3.95q.970721183712.21284B-100000@maigret>
References: <199707211646.KAA06794@steam.acl.lanl.gov>
<Pine.SGI.3.95q.970721183712.21284B-100000@maigret>
Message-ID: <199707221504.JAA11180@steam.acl.lanl.gov>
David Ascher writes:
> On Mon, 21 Jul 1997, Geoffrey Furnish wrote:
>
> > 1) What is the name of the proposed new sig. Is it plot-sig, or
> > sci-sig. This obviously impacts the sig charter and expected usage
> > significantly.
>
> Ok, let's play it safe. As per the SIG creation guidelines, we need a SIG
> name, short description, and longer blurb. Here's a first draft, please
> critique it.
I basically like this proposal.
> PLOT-SIG: Investigating and Developing Scientific and Business Plotting
> Solutions for Python
>
> Short blurb:
>
> The purpose of this list is to develop, coddle together, adopt or
> otherwise make available a package for plotting of scientific and
> business plots of data.
^
I would just voice some concern about a charter which specifies a
singular solution. I say this because I think it is clear that there
are going to be multiple plotting solutions in Python whether they get
the blessing of the plot-sig or not. I would suggest preserving the
basic idea, but eliminating a proscription for a single solution.
Perhaps wording like:
The purpose of this list is to develop, coddle together, adopt or
otherwise make available Python support for plotting of scientific
and business data.
> Principles guiding the development of the software will include:
> * Portability to the relevant platforms
> * Ease of use
> * Integration with other Python packages (e.g. NumPy)
> * Quality of the software
> * Quality of the output
>
> ----
> This is meant to be pretty much independent of existing solutions,
> although clearly some existing packages will fit more or less well. I'm
> concerned that plot-sig will suffer the fate of gui-sig, which was to die
> of starvation because the job at hand is just too big to start from
> scratch, so people just keep using what they've been using, and no 'ideal'
> solution comes to fruition.
I haven't followed gui-sig, so I can't comment on the phenomenology
there. But I do think it would be useful to avoid a proscriptive
attempt to produce the "One True Python Plotting Package". Plotting
is clearly going to be an extension, not a language intrinsic.
Multiple solutions are going to be a fact of life. There are many
reasons for this, not the least of which is that many python users
also use other languages (heresy, I know), and consequently many will
seek multi-language solutions. A proscriptive attempt to generate a
single, Python-centric plotting package will obviously win many users,
and alienate other potential users. Conversely, a pluralist solution
will woo many, and disincline others. There is no "one size fits all,
right answer" to plotting. Rather, the more realistic expectation is
to develop high quality Python interfaces to the plotting facilities
which are available and of interest. And it is entirely conceivable
that such efforts can have an influential effect on the targets.
> I also wanted to add the principle of "object oriented design" to
> the list above, but didn't know if that would get a consensus.
I dont' think a consensus is necessary. In my view, a good sig for
something other than a core language extension, should have a
pluralistic charter. If you want to break programming interfaces down
into two categories, API's and OO interfaces, then Python needs access
to plotting support in both categories. The need for OO plotting
support is clear--but it will not be used by everyone--some will want
API oriented services too. The charter for the plot-sig should
accomodate both needs.
> Regardless, I think it might be a good thing to come up with a list of
> features each of us need, and rank them in terms of 'must have', 'want',
> 'would be nice'. Maybe this should wait until the sig is created, I don't
> know.
>
> Here's mine (off the top of my head):
>
> MUST HAVE:
> Line Plots
> Point Plots (scatter plots)
> Bar Plots
> Error Bars
> Fills (grey levels)
> Symbols
> Log scales
> Bitmap preview (tk, gif, whatever)
> Text
> Production-quality PostScript output
> Very Customizable
> Works on Unix & NT
>
> WOULD BE NICE:
> Complete Python control over graph generation, graph objects, etc.
> Ability to design new graph formats (e.g. Tufte's work)
> Interactivity
> Ability to do Batch work
It's a good list, but I don't think this kind of stuff needs to be
explicitly enumerated in the sig charter.
> Here's what I've used in the past (or use now):
> Gnuplot
> GLE
> MetaPost
> [mac GUI tools]
>
> > 2) Who is the owner going to be? Its not a particularly big job, but
> > it should be someone who cares enough to keep things in order if
> > need be.
>
> I guess I just volunteered.
Thanks much. You should join meta-sig, then post a note proposing the
creation of the new sig, explaining that you have the broad based
support of people in matrix-sig, and providing your charter
statement. Some discussion will ensue, and eventually they'll make
the new sig, setting you up as the owner.
--
Geoffrey Furnish email: furnish@lanl.gov
LANL CIC-19 POOMA/RadTran phone: 505-665-4529 fax: 505-665-7880
"Software complexity is an artifact of implementation." -Dan Quinlan
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Tue Jul 22 16:05:50 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 22 Jul 1997 17:05:50 +0200
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <199707211456.IAA03860@steam.acl.lanl.gov> (message from Geoffrey
Furnish on Mon, 21 Jul 1997 08:56:55 -0600 (MDT))
Message-ID: <199707221505.RAA29129@lmspc1.ibs.fr>
> There are widespread complaints about the run-time performance of Tk
> canvases. This figured prominently in the decision to make a seperate
> Tk-aware plotting widget in PLplot. Besides our own experiences, I
I have heard of this problem, and I certainly don't want to prevent
better Tk-integrations for plotting. But canvases are portable and
part of every Tk implementation, i.e. the most generally available
solution. And I expect them to be fast enough for many routine tasks,
such as simple 2d line plots. So I still think there should be a driver
for them, maybe in addition to a special plotting widget.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Tue Jul 22 16:07:55 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 22 Jul 1997 17:07:55 +0200
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <33D458C3.28D9@smhi.se> (message from Daniel Michelson on Tue, 22
Jul 1997 08:52:51 +0200)
Message-ID: <199707221507.RAA29137@lmspc1.ibs.fr>
> I've also been monitoring the plotting discussions and can't quite grasp
> why the xmgr package has not been mentioned in this context. As free
> scientific plotting packages go, I find it quite complete and versatile,
I use it frequently, but I don't see a specific relation to Python.
Like other "external program" solutions (e.g. Gnuplot), it can't
offer the amount of integration that is desirable.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Tue Jul 22 16:14:13 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Tue, 22 Jul 1997 11:14:13 -0400 (EDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <199707221504.JAA11180@steam.acl.lanl.gov>
Message-ID: <Pine.SGI.3.95q.970722111144.22453B-100000@maigret>
On Tue, 22 Jul 1997, Geoffrey Furnish wrote:
> I would just voice some concern about a charter which specifies a
> singular solution.
I accept that change and will adjust the wording. I agree, there are too
many needs to be met by a single solution.
> > MUST HAVE:
> > Line Plots
> > ...
> > WOULD BE NICE:
> > Complete Python control over graph generation, graph objects, etc.
>
> It's a good list, but I don't think this kind of stuff needs to be
> explicitly enumerated in the sig charter.
Oh, no, I just meant that list for discussion. Not for the charter.
I'll send a revised sig description to meta-sig. I will wait a day or two
to let others comment first, and/or challenge my sig-ownership =).
--da
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From furnish@acl.lanl.gov Tue Jul 22 16:17:59 1997
From: furnish@acl.lanl.gov (Geoffrey Furnish)
Date: Tue, 22 Jul 1997 09:17:59 -0600 (MDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <199707221503.RAA29121@lmspc1.ibs.fr>
References: <199707211646.KAA06794@steam.acl.lanl.gov>
<199707221503.RAA29121@lmspc1.ibs.fr>
Message-ID: <199707221517.JAA11223@steam.acl.lanl.gov>
Konrad Hinsen writes:
> - Let's have a Sci-SIG for general discussions about scientific
> applications, including discussions of features of proposed
> packages etc., and also announcements of test versions.
You want to just relabel matrix-sig as sci-sig? And generalize the
charter?
> - Create separate mailing lists (not formal SIGs) for communication
> between the people working in specific code, e.g. plotting or
> array indexing.
>
> Does this sound reasonable?
I do not favor non-sig mailing lists for discussion of major
subsystems. They are not really very open for people to
subscribe/unsubscribe, since one has to bother a person instead of a
daemon. One of the best things about a sig is that people can get on
and off without having to ask for help. Also, such ad-hoc lists are
not archived.
> > 2) Who is the owner going to be? Its not a particularly big job, but
> > it should be someone who cares enough to keep things in order if
> > need be.
>
> It seems to me that a SIG owner has very little work to do, so that
> shouldn't be a problem. The real question is: Who will write the code? ;-)
This sounds a little to me like the presumption of the one true
python-centric solution mentioned previously. This is arguably
reasonable for a direct language facility like the matrix stuff, but I
think it is basically misplaced loyalty when applied to non-language
intrinsic facilities. For instance, there is no "one true python
database" as far as I know, but rather (as I understand it) bindings
to databases that people use. That would seem to me to be the more
appropriate expectation with respect to plotting. So, who will write
the code? Plenty of people are already doing this. What is of
interest to the python community is to have good python support for
plotting facilities that already exist. And possibly to influence
them. But I don't see a need to produce a whole new python-oriented
plotting facility.
--
Geoffrey Furnish email: furnish@lanl.gov
LANL CIC-19 POOMA/RadTran phone: 505-665-4529 fax: 505-665-7880
"Software complexity is an artifact of implementation." -Dan Quinlan
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Tue Jul 22 16:48:32 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Tue, 22 Jul 1997 17:48:32 +0200
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <Pine.SGI.3.95q.970722111455.22453C-100000@maigret> (message from
David Ascher on Tue, 22 Jul 1997 11:17:47 -0400 (EDT))
Message-ID: <199707221548.RAA29265@lmspc1.ibs.fr>
David Ascher:
> This would take over matrix-sig's role as the place to ask questions about
> numpy?
Yes.
> > - Create separate mailing lists (not formal SIGs) for communication
> > between the people working in specific code, e.g. plotting or
> > array indexing.
>
> Why not make them sigs? The administrative burden is basically the same
> whether it's a sig or some other mailing list, and this way they get
> visibility on python.org, archival, etc.
I have the impression that creating a SIG is a rather complicated
process, whereas I can start a mailing list on Starship by just
typing a few commands. But that's the only reason, I have nothing
against SIGs in principle!
Geoffrey Furnish:
> You want to just relabel matrix-sig as sci-sig? And generalize the
> charter?
Either that or create a new SIG and let the old one die a few months
later. I don't know which is easier to do.
> I do not favor non-sig mailing lists for discussion of major
> subsystems. They are not really very open for people to
> subscribe/unsubscribe, since one has to bother a person instead of a
> daemon. One of the best things about a sig is that people can get on
> and off without having to ask for help. Also, such ad-hoc lists are
> not archived.
I was thinking of MailMan-based mailing lists on Starship, which are
technically the same as the SIGs on python.org, but without any
formalities attached. I do agree that openness and archiving are
important.
> This sounds a little to me like the presumption of the one true
> python-centric solution mentioned previously. This is arguably
Not "the one true" but "a good and easy-to-use one".
> reasonable for a direct language facility like the matrix stuff, but I
> think it is basically misplaced loyalty when applied to non-language
> intrinsic facilities. For instance, there is no "one true python
I think plotting is at the borderline. For standard tasks an interface
to existing plotting packages is fine, but a Python implementation
gives me the opportunity to modify a plot variety by subclassing in
Python. Since it seems that the major complaint about plotting package
X is "it doesn't have feature Y which I need urgently", I am inclined
to put a high value on this customizability.
In addition, there is the installation complexity problem posed by
external packages. To my surprise I received quite a few complaints
about my Molecular Modelling Toolkit related to its use of netCDF.
There were no real problems about netCDF, and installing it is a
simple task, but the simple fact of having to read another README and
running another set of tests seems to put off some people. NumPy
of course is bad enough (but will hopefully change), but at least
Python/C module combinations all work in much the same way.
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Tue Jul 22 17:10:28 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Tue, 22 Jul 1997 12:10:28 -0400
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
Message-ID: <199707221607.MAA02447@dante.mh.lucent.com>
> ...I would just voice some concern about a charter which specifies a
> singular solution. I say this because I think it is clear that there
> are going to be multiple plotting solutions in Python whether they get
> the blessing of the plot-sig or not. I would suggest preserving the
> basic idea, but eliminating a proscription for a single solution...
I think it would be nice to *promote* a singular solution, if possible.
If people want to do something different, the can and will. If the
singular solution was amenible to customization and extension,
so much the better.
> ... haven't followed gui-sig, so I can't comment on the phenomenology
> there. But I do think it would be useful to avoid a proscriptive
> attempt to produce the "One True Python Plotting Package"...
Actually, theres a good name: OTPPP!
I haven't followed the GUI sig either, but I've been disappointed
with the results. Frankly maybe if Guido had picked one gui
either that gui would be really really good now, or in self defense
another alternative would have gotten much better and replaced
it. As it is, I'm hoping that Fredrik and Mark H. will fight it out
for market share (perhaps with Jim A. silently sharpening wpy
in the background and pyawt in the wings) and both
pythonwin and tkinter will end up
really really good, or shamed by some clearly better alternative.
Endless discussions generate a lot of email...
If someone has the fortitude perhaps they could "just write it"
and use the Numeric/Graphics/Plot SIG as a source of suggestions,
much like Jim H did with Numeric. I prefer proofs of concepts
to discussions. We might still be wishing for a good Numerics
library if he hadn't. -- Aaron Watters
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From johann@physics.berkeley.edu Tue Jul 22 19:02:19 1997
From: johann@physics.berkeley.edu (Johann Hibschman)
Date: Tue, 22 Jul 1997 11:02:19 -0700 (PDT)
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <Pine.HPP.3.95.970722072428.2186A-100000@laura.llnl.gov>
Message-ID: <Pine.SUN.3.96.970722105852.15493A-100000@physics12.Berkeley.EDU>
On Tue, 22 Jul 1997, Zane C. Motteler wrote:
> Just a reminder that we have a working, pretty decent 2d-3d scientific
> graphics package working with Gist (a public domain graphics engine from
> LLNL). The Python interface is object-oriented, consisting of Graph
> objects, geometric objects which can be shipped to Graph objects,
> and Plotter objects which can be used to plot them. This is pretty much
> as I described in my talks at the last two Python meetings. The 3-D
> graphics is much enhanced since I first talked about it at the
> Washington meeting last year.
Does the Gist module still require patching the Python source? The last
time I checked, the latest version required a set of patches which managed
to confuse the latest version of SWIG, so I turned to other packages.
(SuperMongo, anyone?)
- Johann
---
Johann A. Hibschman | Grad student in Physics, working in Astronomy.
johann@physics.berkeley.edu | Probing pulsar pair production processes.
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Tue Jul 22 19:05:20 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Tue, 22 Jul 1997 14:05:20 -0400
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
Message-ID: <199707221824.OAA29302@dante.mh.lucent.com>
> From: Konrad Hinsen <hinsen@ibs.ibs.fr>
> I have the impression that creating a SIG is a rather complicated
> process, whereas I can start a mailing list on Starship by just
> typing a few commands. But that's the only reason, I have nothing
> against SIGs in principle!
The problem is not the mechanics, the problem is that
with too many sigs people don't know where to go and with
too few they get congested with peripheral stuff or
"orphan messages". I'd opt
for too few, myself, and people can learn how to use that
little delete button... I discovered it not to long ago...
really handy!
Actually I was distressed to see the pythonwin-sig go
away and it looks like maybe the db-sig may go that
way too (even though it has little activity, if you want
an answer on databases, that's the place to go -- your message
might be accidentally ignored on other lists, including the main list).
-- Aaron Watters
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From busby@icf.llnl.gov Tue Jul 22 20:15:03 1997
From: busby@icf.llnl.gov (L. Busby)
Date: Tue, 22 Jul 97 12:15:03 PDT
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
Message-ID: <9707221915.AA12700@icf.llnl.gov.llnl.gov>
Johann Hibschman wrote:
>Does the Gist module still require patching the Python source? The last
>time I checked, the latest version required a set of patches which managed
>to confuse the latest version of SWIG, so I turned to other packages.
>(SuperMongo, anyone?)
Yes, a patch is required for Python 1.4. Expect Python 1.5 to be
``Gist-ready''. We make available Python-1.4, NumPy, Gist, all patches
(including a patch which allows GNU readline to link against
Python/Gist), and several other LLNL modules in a gzipped tar file at
ftp://ftp-icf.llnl.gov/pub/python/Current_snapshot.tgz.
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From ptaney@usgs.gov Tue Jul 22 20:27:21 1997
From: ptaney@usgs.gov (Paul Taney, Computer Specialist, Reston, VA )
Date: Tue, 22 Jul 1997 15:27:21 -0400
Subject: [MATRIX-SIG] A NO-Proposal for a plot-package
Message-ID: <199707221927.PAA15419@mailrvares.er.usgs.gov>
Hi:
aaron wrote:
>I think it would be nice to *promote* a singular solution, if possible.
>If people want to do something different, the can and will. If the
>singular solution was amenible to customization...
There is a parallel discussion about certain extensions for
that coffee language going at java3d-interest@Sun.COM
And among the posters are Rikk Carey (rikk@best.com) of VRML fame
who suggested dividing the problem this way:
Basics - contains the base types Color, Vector, Matrix, etc.
Core - contains core groups, geometry, lights, attributes,
Viewpoints, and supports ROUTES and PROTOs
Interactive - adds sensors, interpolators, media, and scripts
Geometry - adds other fancy geometry (e.g. Extrusion, CSG)
SciViz - adds specialized nodes for SciViz
VR - adds specialized VR functionality (e.g. HMD support)
Animation - adds specialized nodes for Animation
etc.
I, too, would appreciate a singular solution, if possible. And as
a Viz specialist I appreciate this list's new direction.
paul
ptaney@usgs.gov @
smail: ms 431, reston, va @ I ain't goin' fishin' no damn rain.
voice: 703-648-5598 @ -- Thelma
home: 540-822-4223 @
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From ptaney@usgs.gov Wed Jul 23 00:09:14 1997
From: ptaney@usgs.gov (Paul Taney, Computer Specialist, Reston, VA )
Date: Tue, 22 Jul 1997 19:09:14 -0400
Subject: [MATRIX-SIG] books on plotting
Message-ID: <199707222309.TAA18554@mailrvares.er.usgs.gov>
Aaron Watters wrote:
> Is there a standard book on the topic of plotting? It might provide
> a framework for discussion.
What is plotting? I think of paper products when you say that.
There are 2 books on computational geometry that might help with this;
especially mapping sparce matricies (coords) to spatial data
structures (range-trees, convex hulls, etc.)
"Computational Geometry", Preperata and Shamos, Springer, 85
"Computational Geometry in C", J O'Rourke, Cambridge, 93
code may still be at 131.299.64.23, /pub/compgeom
AW (No not you Aaron, the _publisher) has a series (5) on OpenGL and
Inventor; see esp. _The Inventor Mentor_ if you want to get with
VRML. SGI Inventor is the mother of VRML97. Also see the Java3D
spec...
And of course the Graphics Gems series of 5 from AP. Also see the GIS
shelf at Computer Literacy (or wherever) esp. Unwin; he's got a site
somewhere in .uk.
Hope this helps,
paul
ptaney@usgs.gov @ Use the Source, Luke.
U.S.Geological Survey @ Use the Source!
reston, va 22092 @
ph: 703-648-5598 @ -- Linus T.
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Wed Jul 23 00:49:33 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Tue, 22 Jul 1997 19:49:33 -0400
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <199707221444.IAA11137@steam.acl.lanl.gov>
References: <3.0.1.32.19970722000740.008142c0@popd.netcruiser>
<3.0.1.32.19970721203601.0081bac0@popd.netcruiser>
<Pine.SGI.3.95q.970721232455.21602D-100000@maigret>
<3.0.1.32.19970722000740.008142c0@popd.netcruiser>
Message-ID: <3.0.1.32.19970722194933.00807220@popd.netcruiser>
At 08:44 AM 7/22/97 -0600, Geoffrey Furnish wrote:
>Andrew P. Mullhaupt writes:
> > If you read it that way, then fine.
>
>David read my message the way I intended it to be read. My view is
>that plotting will /use/ NumPy, but probably does not need to
>influence the development of NumPy too much. I think it would be
>desirable if we could move all this plotting talk off the matrix-sig.
OK I can agree completely with this.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hinsen@ibs.ibs.fr Wed Jul 23 10:20:26 1997
From: hinsen@ibs.ibs.fr (Konrad Hinsen)
Date: Wed, 23 Jul 1997 11:20:26 +0200
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
In-Reply-To: <Pine.HPP.3.95.970722072428.2186A-100000@laura.llnl.gov>
(motteler@laura.llnl.gov)
Message-ID: <199707230920.LAA31626@lmspc1.ibs.fr>
> Just a reminder that we have a working, pretty decent 2d-3d scientific
> graphics package working with Gist (a public domain graphics engine from
> LLNL). The Python interface is object-oriented, consisting of Graph
> objects, geometric objects which can be shipped to Graph objects,
> and Plotter objects which can be used to plot them. This is pretty much
Sounds good, and could be used as a basis for other low-level plotting
output. Not that Gist isn't nice, but at the moment I won't recommend
it to anyone who might then ask me for help with installation problems ;-)
> Our distribution is available at our public ftp site ftp-icf.llnl.gov
> in /ftp/pub/python/Current_snapshot.tgz and is updated with our latest
> changes each Friday night. Available documentation leaves something to
I realize that a "one-in-all" distribution is the easiest solution for you,
but if you'd like others to use your stuff, a separate package would
be more appropriate. Not only is it tough for a non-expert to extract
the right pieces from the distribution, but there is also the technical
problem of downloading a huge file via bad network links. I tried a few
times but never succeeded...
--
-------------------------------------------------------------------------------
Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr
Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28
Institut de Biologie Structurale | Fax: +33-4.76.88.54.94
41, av. des Martyrs | Deutsch/Esperanto/English/
38027 Grenoble Cedex 1, France | Nederlands/Francais
-------------------------------------------------------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From c9603085@zeus.hud.ac.uk Wed Jul 23 11:22:35 1997
From: c9603085@zeus.hud.ac.uk (MISS. K.L.COLLIER)
Date: Wed, 23 Jul 1997 11:22:35 +0100
Subject: [MATRIX-SIG] convolution
Message-ID: <199707231022.LAA00959@penelope.csm>
I'm trying to calculate the convolution of two arrays (an image and a point
spread function) using
Numerical python as follows:
#!/usr/local/bin/python
from Numeric import *
from FFT import *
raw_a = array ( [ 1,2,0,3,4,0,0,0,0 ] )
raw_a.shape=(3,3)
raw_b = array ( [ -1,1,0,-2,2,0,0,0,0 ] )
raw_b.shape=(3,3)
freq_2a = fft2d(raw_a)
freq_2b = fft2d(raw_b)
fab=freq_2a*freq_2b
res = inverse_fft2d(fab)
print "res"
print res
but there doesn't seem to be an inverse_fft2d function. Is there one?
Would this work anyway?
Do I have to have the two images the same size, i.e. pad out
the smallest one with zeros. If so do I do this symmetrically or
fill the zeros on the rhs and bottom of the matrix?
Does python automatically handle the complex numbers ok in the multiplication
in the frequency domain?
Other options:
1. I also notice that there is a convolve function in Numeric but that it is only 1D. If my psf is a gaussian i.e. is symmetric, is there a
way of doing 2 1D convolutions to produce a 2D one?
2. I believe it can also be done in matrix form if the image is converted
to a vector and the psf converted to a block circulant form. If the psf is only 5*5 then the matrix would be 25*25, BUT don't I need to first
pad out the psf to the same size as the image (512*512)
in which case the matrix would be way too big!
Many questions I know, but does someone know the best way to do it?
Thanks,
Karen Collier c9603085@zeus.ac.hud.uk
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hyoon@nyptsrv1.etsd.ml.com Wed Jul 23 22:35:57 1997
From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant)
Date: Wed, 23 Jul 1997 17:35:57 -0400
Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package
Message-ID: <199707232135.RAA02104@ptq3.etsd>
Zane C. Motteler Wrote:
> Just a reminder that we have a working, pretty decent 2d-3d scientific
>graphics package working with Gist (a public domain graphics engine from
>LLNL). The Python interface is object-oriented, consisting of Graph
>objects, geometric objects which can be shipped to Graph objects,
>and Plotter objects which can be used to plot them. This is pretty much
>as I described in my talks at the last two Python meetings. The 3-D
>graphics is much enhanced since I first talked about it at the
>Washington meeting last year.
Hello,
Gist is a nice package for what it does. I think it makes a good foundation
for a lot of 3d type of work, but falls short of :
>
> Here's mine (off the top of my head):
>
> MUST HAVE:
> Line Plots
> Point Plots (scatter plots)
> Bar Plots
> Error Bars
> Fills (grey levels)
> Symbols
> Log scales
> Bitmap preview (tk, gif, whatever)
> Text
> Production-quality PostScript output
> Very Customizable
> Works on Unix & NT
>
> WOULD BE NICE:
> Complete Python control over graph generation, graph objects, etc.
> Ability to design new graph formats (e.g. Tufte's work)
> Interactivity
> Ability to do Batch work
At least one person's idea of what it should do. I am had some frustration
with trying to get bar graph and non-numeric ticks. I also would like to
get 2 different XY axis for the graph, which I don't think is supported
in Gist. Basically I will be extremely satisfied with Gist + Excel type
of graphics. Has anyone seen Splus 4 graphs in Axim? It's completely obj
oriented. Would it be lovely to have Trellis type of analysis?
Just my few thoughts,
Hoon,
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From acura@vcn.bc.ca Thu Jul 24 02:49:02 1997
From: acura@vcn.bc.ca (Kelvin Lee)
Date: Wed, 23 Jul 1997 18:49:02 -0700 (PDT)
Subject: [MATRIX-SIG] two newbie questions
Message-ID: <Pine.GSO.3.95.970723184746.1188B-100000@opus.vcn.bc.ca>
I'm still struggling to get a handle on NumPy... Two newbie questions:
1) How much memory does an NxN Float16 matrix use?
2) What's the most efficient way of constructing a "correlation-like"
matrix of a given function for a data set? eg. using, say, RMSE instead
of r?
To show you where I'm at, I'm stuck with:
def correlation_like(X, function):
R = zeros((len(X), len(X)), Float16)
for j in range(len(X)):
for i in range(len(X)):
R[i,j] = function(X[i], X[j])
return R
Obviously this doesn't take into account symmetry etc. Any ideas to get me
thinking in a matrix kind of way would be appreciated!
Kelvin
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Thu Jul 24 05:07:13 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Thu, 24 Jul 1997 00:07:13 -0400 (EDT)
Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG
Message-ID: <Pine.SGI.3.95q.970723235541.25656A-100000@maigret>
Here's a revised version of the PLOT-SIG proposal. Please let me know (in
private if you wish) whether you find it appropriate and whether I should
submit it to the META-SIG for discussion.
I never did get a clear feeling for whether more folks felt like Konrad
that this should instead be an unofficial working group based on the
Starship instead of the Mothership (python.org). Feel free to voice
opinions...
Just to get my bias out in the open -- I am more interested in the second
subproject described below, but will of course encourage work on
interfacing with existing packages, in part because I expect to have plots
to produce before the new framework is in place...
Geoffrey (or others) -- if you wish to further specify the charter of what
I call the "API subproject", let me know.
-----
PLOT-SIG: Development of Data Plotting Solutions for Python
The purpose of this list is to develop, coddle together, adopt or
otherwise make available Python tools for plotting of scientific and
business plots of data.
Plotting needs vary greatly depending on each user's requirements.
Some users wish to see a better API between Python and existing
plotting libraries or programs. Others would rather see a new
framework which maximally leverages the OOP and dynamic strengths of
Python, at the cost of reinventing a few wheels. Both strategies will
be entertained by the SIG, with individual members contributing to
projects they wish to see furthered.
Principles guiding the development of all software will include:
* Ease of use.
* Clean Integration with relevant Python packages (NumPy, PIL, etc.).
* Quality of the software.
* Quality of the output.
One possible goal for the API subproject is to develop a package-
independent API for plots, which would produce reasonably similar results
regardless of the plotting package used as a backend (PLPlot, Gist/Yorick,
Gnuplot, etc.), in the same spirit as the interface defined by the DB-SIG.
Package-specific extensions could naturally be provided as well.
The goals for the new framework subproject need to be further specified by
the SIG, but include:
* Complete Python Control.
* Extensibility/Customizability.
* High quality rendering both on screen and paper.
* Portability (at least UNIX/X11 and Win32, MacOS if feasible).
* Stealing good ideas others have had.
* Interactivity.
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From Daniel.Michelson@smhi.se Thu Jul 24 08:01:18 1997
From: Daniel.Michelson@smhi.se (Daniel Michelson)
Date: Thu, 24 Jul 1997 09:01:18 +0200
Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG
References: <Pine.SGI.3.95q.970723235541.25656A-100000@maigret>
Message-ID: <33D6FDBE.2781@smhi.se>
Hej,
RE: the forum for PLOT-SIG
I appreciate that some members of the MATRIX-SIG are intent on
establishing the necessary logistics to allow the (rapid) creation of
Python plotting functionality. In this context, the hospitality of the
Starship has been described as being potentially more accessable than
python.org, and this task would thus be underway faster if the informal
route is taken.
I don't want to rain on anyone's parade, but I feel obliged to voice a
concern about the communications between Python developers and users. I
am well aware that many developers on this SIG are also users, but there
are a number of users who are not and cannot be developers but who
appreciate the ability to incorporate new Python functionality ASAP.
Needless to say, I am one of these users, and plotting abilities
directly in Python are currently prioritized. (And I am encouraged by
this PLOT-SIG initiative!)
What this all boils down to is that, regardless of the PLOT-SIG's host,
the forum should be open. Konrad mentioned that he could easily start an
open mailing list on the Starship. Others have used words like
"informal" and "unofficial". I just want to make sure that the "mailing
list" and the latter two terms are not mutually exclusive and that the
PLOT-SIG doesn't risk turning into a private club.
best, -d
+------------------+--------------------------------------------------+
| | Daniel B. Michelson |
| /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute|
| \ | \/ | |__| | | S-601 76 Norrk=F6ping, Sweden |=
| \ | | | | | | Telephone: +46 11 - 15 84 94 |
| \/ | | | | | | Telefax: +46 11 - 17 02 07 |
| | Internet: Daniel.Michelson@smhi.se |
+------------------+--------------------------------------------------+
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From jbaddor@phy.ulaval.ca Thu Jul 24 15:30:03 1997
From: jbaddor@phy.ulaval.ca (Jean-Bernard ADDOR)
Date: Thu, 24 Jul 1997 10:30:03 -0400 (EDT)
Subject: [MATRIX-SIG] -3**(-1/3.), sgn()
Message-ID: <Pine.SUN.3.95.970724100850.8464B-100000@mills>
Hy !
a is an array with positive, negatives or null values
I want to compute a**(-1/3.). It works perfectly if (a) is not an array
but a scalar (function pow). The funcion power do not give fractional
power from negative number.
I would like to write sgn(a)*abs(a)**(-1/3.) but the sgn function seems
not to be defined in python (perhaps because it is not in c, but it is in
fortran). I can use greater_equal. Do you have a better solution ?
A bientot,
Jean-Bernard
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From jbaddor@phy.ulaval.ca Thu Jul 24 16:17:26 1997
From: jbaddor@phy.ulaval.ca (Jean-Bernard ADDOR)
Date: Thu, 24 Jul 1997 11:17:26 -0400 (EDT)
Subject: [MATRIX-SIG] abs vs absolute
Message-ID: <Pine.SUN.3.95.970724111527.8464C-100000@mills>
Hy !
a = array([-2, 3,0, 5])
What is the difference between:
abs(a) and absolute(a) ?
Jean-Bernard
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From barrett@compass.gsfc.nasa.gov Thu Jul 24 17:11:40 1997
From: barrett@compass.gsfc.nasa.gov (Paul Barrett)
Date: Thu, 24 Jul 1997 12:11:40 -0400
Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG
In-Reply-To: <Pine.SGI.3.95q.970723235541.25656A-100000@maigret>
References: <Pine.SGI.3.95q.970723235541.25656A-100000@maigret>
Message-ID: <199707241611.MAA24662@compass.>
I've been following along on this thread for awhile now, but have been
too busy doing programmatic work to take the time to respond. I'll
make some time and do so now.
Some of my current feelings are:
1) A PLOT-SIG should be created in spite of what happens to the
MATRIX-SIG. The creation of a SIG with narrow focus and specific
goals is more likely to reach its objective than being a splinter
group in a more general SIG like a DATA-ANALSYS-ENVIRONMENTS-SIG.
Note that the MATRIX-SIG was created for just this reason.
(As an aside, I don't think the MATRIX-SIG should be terminated, but
just moved to the back burner, so that the old-pros and newcomers can
revisit this topic from time to time without creating noise in newly
created or related SIGS.)
2) Now that the MATRIX-SIG is nearly complete, a PLOT-SIG (or
GRAPHICS-SIG, or DATA-DISPLAY-SIG) is of next highest priority for me.
Where I work, IDL and SuperMongo are still used by my coworkers
because of their plotting capabilities, mostly for their ease of use.
Note that they both have their limitations, mostly associated with
their inferior language and syntax and user interface. Many coworkers
would be willing to migrate to Numeric Python if there was good
plotting capabilities, eg. 2-D, contour plots, shaded-surfaces, etc.
An excellent plotting package able to do 3-D volume rendering and
animation alla Vis5D could be developed later, but should be included
in the initial design.
3) I also feel that we should try to develop a high level OO interface
to current packages and if deemed necessary a new Python developed
package. The former would help to get some good graphics capability
into Python quickly using pgplot, PLplot, etc., but would allow
migration to a more advanced plotting package in future. I don't see
why those of us wanting to continue to use one of the currrent
packages cannot add any addition functions to get full capabilities
from their package, knowing full well that these functions are not
platform and package independent.
4) My current dismay at many current packages is that they are
platform specific and do not really allow easy access to low level
features of the graph. I see a design with 3 or 4 levels. The
highest level would provide the creation of a simple plot from a
single command. Lower level commands would allow users to create
new standard plot types (eg. bar charts, etc.) using the low level
commands.
In addition a high level interface would allow us to develop a
platform independent package. IDL has managed to do this across X11,
Windows, and Mac interfaces. I don't see why we can't do something
similar.
6) I think we should create the SIG and start hashing out the design
issues ASAP. A table containing a consensus of each plotting packages
strengths and weaknesses might help immensely in our final design.
-- Paul
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hochberg@wwa.com Thu Jul 24 17:20:39 1997
From: hochberg@wwa.com (Timothy A. Hochberg)
Date: Thu, 24 Jul 1997 11:20:39 -0500 (CDT)
Subject: [MATRIX-SIG] -3**(-1/3.), sgn()
In-Reply-To: <Pine.SUN.3.95.970724100850.8464B-100000@mills>
Message-ID: <Pine.BSI.3.95.970724111050.20521A-100000@shoga.wwa.com>
On Thu, 24 Jul 1997, Jean-Bernard ADDOR wrote:
> I would like to write sgn(a)*abs(a)**(-1/3.) but the sgn function seems
> not to be defined in python (perhaps because it is not in c, but it is in
> fortran). I can use greater_equal. Do you have a better solution ?
For what it's worth, sgn(x) can be implemented as:
def sgn(x):
return greater_equal(x,0) - less_equal(x,0)
(or is that what you meant by use greater equal?)
____
/im
+------------------------------------------------+
|Tim Hochberg Research Assistant |
|hochberg <at> wwa.com University of Illinois |
| Electrical Engineering |
+------------------------------------------------+
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From jbaddor@phy.ulaval.ca Thu Jul 24 18:42:01 1997
From: jbaddor@phy.ulaval.ca (Jean-Bernard ADDOR)
Date: Thu, 24 Jul 1997 13:42:01 -0400 (EDT)
Subject: [MATRIX-SIG] -3**(-1/3.), sgn()
In-Reply-To: <Pine.BSI.3.95.970724111050.20521A-100000@shoga.wwa.com>
Message-ID: <Pine.SUN.3.95.970724133615.8464E-100000@mills>
What I meant was :
def sgn(a):
return 2*Numeric.greater_equal(a,0) - 1
I don't know which one is better (faster, without using too much memory).
Your give sgn(0)=0 what is better that sgn(0)=1 for mine, but for my
application it doesn't change anythink.
Jean-Bernard
On Thu, 24 Jul 1997, Timothy A. Hochberg wrote:
> On Thu, 24 Jul 1997, Jean-Bernard ADDOR wrote:
>
> > I would like to write sgn(a)*abs(a)**(-1/3.) but the sgn function seems
> > not to be defined in python (perhaps because it is not in c, but it is in
> > fortran). I can use greater_equal. Do you have a better solution ?
>
> For what it's worth, sgn(x) can be implemented as:
>
> def sgn(x):
> return greater_equal(x,0) - less_equal(x,0)
>
> (or is that what you meant by use greater equal?)
>
> ____
> /im
>
> +------------------------------------------------+
> |Tim Hochberg Research Assistant |
> |hochberg <at> wwa.com University of Illinois |
> | Electrical Engineering |
> +------------------------------------------------+
>
>
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From barrett@compass.gsfc.nasa.gov Thu Jul 24 19:14:57 1997
From: barrett@compass.gsfc.nasa.gov (Paul Barrett)
Date: Thu, 24 Jul 1997 14:14:57 -0400
Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG
In-Reply-To: <Pine.SGI.3.95q.970724135351.26570E-100000@maigret>
References: <199707241611.MAA24662@compass.>
<Pine.SGI.3.95q.970724135351.26570E-100000@maigret>
Message-ID: <199707241814.OAA25006@compass.>
David Ascher writes:
> > 2) Now that the MATRIX-SIG is nearly complete, a PLOT-SIG (or
> > GRAPHICS-SIG, or DATA-DISPLAY-SIG) is of next highest priority for me.
>
> I don't think that GRAPHICS-SIG is appropriate -- too many things qualify
> as graphics -- I think that displaying data is indeed the core issue. And
> DATA-DISPLAY-SIG is a bit verbose. Can you live with PLOT-SIG?
>
I can live with it. I just tossed out these alternative SIG names to
stimulate others to think about issues other than just 2-D plotting.
(I don't expect any of them to be adopted and used as the name of the
SIG.) I think we should consider how our work may possibly affect the
inevitable next step of developing a scientific visualization package
for Python.
-- Paul
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Thu Jul 24 19:46:41 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Thu, 24 Jul 1997 14:46:41 -0400 (EDT)
Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG
In-Reply-To: <199707241814.OAA25006@compass.>
Message-ID: <Pine.SGI.3.95q.970724144557.26720B-100000@maigret>
> I can live with it. I just tossed out these alternative SIG names to
> stimulate others to think about issues other than just 2-D plotting.
> (I don't expect any of them to be adopted and used as the name of the
> SIG.) I think we should consider how our work may possibly affect the
> inevitable next step of developing a scientific visualization package
> for Python.
Fair enough. Sure, 3d + animation + head-mounted displays and a VR cave
would be nice too (Brown will be getting one of those soon, but I'll be
gone =).
--da
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From ptaney@usgs.gov Fri Jul 25 01:04:02 1997
From: ptaney@usgs.gov (Paul Taney, Computer Specialist, Reston, VA )
Date: Thu, 24 Jul 1997 20:04:02 -0400
Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG
In-Reply-To: <Pine.SGI.3.95q.970724144557.26720B-100000@maigret>
Message-ID: <199707250004.UAA09316@mailrvares.er.usgs.gov>
On Thu, 24 Jul 1997 14:46:41 -0400 (EDT)
David Ascher said:
>
> > I can live with it. I just tossed out these alternative SIG names to
> > stimulate others to think about issues other than just 2-D plotting.
> > (I don't expect any of them to be adopted and used as the name of the
> > SIG.) I think we should consider how our work may possibly affect the
> > inevitable next step of developing a scientific visualization package
> > for Python.
>
> Fair enough. Sure, 3d + animation + head-mounted displays and a VR cave
> would be nice too
>
PLOT is a dull word, IMHO, like PLOD. If you call it SCIVIZ-SIG, VIZ-SIG
or
PYTHON-3D you'll have more participants, maybe even more help. This is
sexy
stuff :-)
paul
ptaney@usgs.gov @ Starbucks sells a Javachip. (It's
U.S.Geological Survey @ ice cream, but that shouldn't
reston, va 22092 @ be an obstacle for the Java
ph: 703-648-5598 @ marketing people.) -- Jeremy
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From barrett@compass.gsfc.nasa.gov Fri Jul 25 15:02:46 1997
From: barrett@compass.gsfc.nasa.gov (Paul Barrett)
Date: Fri, 25 Jul 1997 10:02:46 -0400
Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG
In-Reply-To: <199707250004.UAA09316@mailrvares.er.usgs.gov>
References: <Pine.SGI.3.95q.970724144557.26720B-100000@maigret>
<199707250004.UAA09316@mailrvares.er.usgs.gov>
Message-ID: <199707251402.KAA26770@compass.>
Paul Taney writes:
>
> PLOT is a dull word, IMHO, like PLOD. If you call it SCIVIZ-SIG, VIZ-SIG
> or
> PYTHON-3D you'll have more participants, maybe even more help. This is
> sexy
> stuff :-)
>
I also considered SCIVIZ-SIG as an alternative name. It has a nice
ring to it!
-- Paul
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hyoon@nyptsrv1.etsd.ml.com Fri Jul 25 15:41:38 1997
From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant)
Date: Fri, 25 Jul 1997 10:41:38 -0400
Subject: [MATRIX-SIG] hash() function w/ NumPy
Message-ID: <199707251441.KAA07971@ptq3.etsd>
Hello,
Taking a break from all the encouraging discussion about the Plots.
I am attempting to incorporate hash('STR') to an Array. Since hash turns
a String into uniq numeric sequence, I am trying to use it to sneak in
string id's into Numerical arrays. Now I know how to turn one into hash,
but how do I get it back to a string? There is supposed to be function,
but I remeber reading that I may not be able get the same string back
in reverse. If anyone remembers the function and the documentation in
Lutz or Web pages let me know. If anyone had done some of these work
already, please tip me off few tricks.
I guess this is not exactely recommended, but I cannot think of a good
way to use strings as row and/or column headings (as a list may be). Has
anyone done this?
Thanks much,
Hoon,
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From jim@ks.uiuc.edu Fri Jul 25 17:08:33 1997
From: jim@ks.uiuc.edu (Jim Phillips)
Date: Fri, 25 Jul 97 11:08:33 -0500
Subject: [MATRIX-SIG] Proposal: "Tables" in NumPy
In-Reply-To: <199707251441.KAA07971@ptq3.etsd>
References: <199707251441.KAA07971@ptq3.etsd>
Message-ID: <199707251608.AA151256914@london.ks.uiuc.edu>
Hoon Yoon wrote:
> I am attempting to incorporate hash('STR') to an Array. Since hash turns
> a String into uniq numeric sequence, I am trying to use it to sneak in
The has function does not return a unique number for every string. This
would be impossible! A hash function returns an integer given some data, but
collisions do occur and have to be dealt with. There is definitely not a 1-1
mapping between strings and hash values (there are fewer 10 digit numbers than
10 character strings).
As I see it, there are two possibilities. You can store the strings
separately in a list, dictionary, array, etc. and define your own id's based
on that, or you can write your own function to map (short!) strings uniquely
to integers.
I do sympathize with this problem. I would really love to see something
higher level than an array, perhaps called a "table", which could be indexed
as a dictionary (columns by heading) in some dimensions, and as an array (rows
by index) in others. This would really simplify data storage since one could
label data and access it without worrying about which column "pressure" is.
Pickled tables would also be self documenting!
I have written something like this in python, but only in two dimensions and
it is rather slow. I would love to see a full N-dimension implementation as a
part of NumPy. Perhaps a good interface would look something like this:
>>> a # a 2-D table
dogs cats birds
2 4 6
1 2 3
9 8 7
>>> a['dogs'] # returns array (reference!)
2 1 9
>>> a[:,1] # returns table (reference!)
dogs cats birds
1 2 3
>>> a[('dogs','birds'),0:2] # returns table (reference!)
dogs birds
2 6
1 3
This could really be a killer feature for data analysis since we would no
longer have to choose between C-style structures and FORTRAN-style arrays!
-Jim Phillips
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Fri Jul 25 17:30:50 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Fri, 25 Jul 1997 12:30:50 -0400 (EDT)
Subject: [MATRIX-SIG] RFP: PLOT-SIG
Message-ID: <Pine.SGI.3.95q.970725122247.28673B-100000@maigret>
After somewhat extensive discussion in the MATRIX-SIG, I am officially
submitting a proposal for the creation of a new sig. Here is the one line
comment and more extended docstring. I am volunteering as sigmeister.
Please comment.
This announcement is cc'ed to MATRIX-SIG, but I request that all followups
be directed at the META-SIG. Please check your cc: line before sending
comments.
--david ascher
--------------------------------
# PLOT-SIG: Development of Data Plotting Solutions for Python
"""
The purpose of this list is to develop, coddle together, adopt or
otherwise make available Python tools for plotting of scientific and
business plots of data.
Plotting needs vary greatly depending on each user's requirements.
Some users wish to see a better API between Python and existing
plotting libraries or programs. Others would rather see a new
framework which maximally leverages the OOP and dynamic strengths of
Python, at the cost of reinventing a few wheels. Both strategies will
be entertained by the SIG, with individual members contributing to
projects they wish to see furthered.
Principles guiding the development of all software will include:
* Ease of use.
* Integration with other Python packages (NumPy, PIL, etc.).
* Quality of the software.
* Quality of the output.
One possible goal for the API project is to develop a package-
independent API for plots, which would produce reasonably
similar results regardless of the plotting package used as a backend
(PLPlot, Gist/Yorick, Gnuplot, etc.), in the same spirit as the
interface defined by the DB-SIG. Package-specific extensions could
naturally be provided as well.
The goals for the new framework need to be further specified by the
SIG, but include:
* Complete Python Control.
* Extensibility/Customizability.
* High quality rendering both on screen and paper.
* Portability (at least UNIX/X11 and Win32, MacOS if feasible).
* Stealing good ideas others have had.
* Interactivity
The SIG is open, and membership is expected to include folks who wish to
contribute to the development efforts, folks who have lots of expertise
they wish to share, and novice users who wish to share their lists of
requirements, questions about the available software, etc.
"""
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Fri Jul 25 17:29:54 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Fri, 25 Jul 1997 12:29:54 -0400
Subject: [MATRIX-SIG] hash() function w/ NumPy
Message-ID: <199707251626.MAA27320@dante.mh.lucent.com>
The hash() function takes a string of arbitrary length
and transforms it into a 32 bit value. If the string has more
than 32 bits there is no way to get the string back in general.
However, if the hash values are unique you can use a ancillary
dictionary mapping hash(x) --> x, but you better be ready
to do something appropriate in the case that collisions occur.
Maybe a better strategy is to simply number the strings sequentially
in an external index structure.
class string_cache:
def __init__(self):
self.dict = {}
self.list = []
def insert(self, str):
d = self.dict
if d.has_key(str):
return d[str]
else:
result = d[str] = len(d)
self.list.append(str)
return result
def get_by_index(self, index):
return self.list[index]
Untested, and I hope you don't want to do deletions :).
If you want to get the strings back in aggregate
(ie, get a bunch of strings from the structure
in a couple interpreter ticks)
the above could be modified to use my favorite extension
module, kjbuckets :) but I'll spare you.
With Python 1.4+ string hash collisions are unlikely (with 1.3, sadly,
they were common, and still are for tuples, but this will
be fixed in 1.5). -- Aaron Watters
----------
> From: Hoon Yoon - IPT Quant <hyoon@nyptsrv1.etsd.ml.com>
> To: matrix-sig@python.org
> Subject: [MATRIX-SIG] hash() function w/ NumPy
> Date: Friday, July 25, 1997 10:41 AM
>
> Hello,
>
> Taking a break from all the encouraging discussion about the Plots.
> I am attempting to incorporate hash('STR') to an Array. Since hash turns
> a String into uniq numeric sequence, I am trying to use it to sneak in
> string id's into Numerical arrays. Now I know how to turn one into hash,
> but how do I get it back to a string? There is supposed to be function,
> but I remeber reading that I may not be able get the same string back
> in reverse. If anyone remembers the function and the documentation in
> Lutz or Web pages let me know. If anyone had done some of these work
> already, please tip me off few tricks.
> I guess this is not exactely recommended, but I cannot think of a good
> way to use strings as row and/or column headings (as a list may be). Has
> anyone done this?
> Thanks much,
>
> Hoon,
A sus ordenes :).
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Fri Jul 25 17:37:02 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Fri, 25 Jul 1997 12:37:02 -0400 (EDT)
Subject: [MATRIX-SIG] Proposal: "Tables" in NumPy
In-Reply-To: <199707251608.AA151256914@london.ks.uiuc.edu>
Message-ID: <Pine.SGI.3.95q.970725123124.28673C-100000@maigret>
On Fri, 25 Jul 1997, Jim Phillips wrote:
> I do sympathize with this problem. I would really love to see something
> higher level than an array, perhaps called a "table", which could be
> indexed as a dictionary (columns by heading) in some dimensions, and as
> an array (rows by index) in others. This would really simplify data
> storage since one could label data and access it without worrying about
> which column "pressure" is. Pickled tables would also be self
> documenting!
This is one of the features of the S+ data.frame object, which is the
topic of another thread on this SIG. For anyone interested in this, I
recommend finding and reading Chapter 3 of "Statistical Models in S", by
John M. Chambers and Trevor J. Hastie. You don't really need to know
anything about statistics, but the chapter will explain what one (IMHO,
very good) solution to this problem looks like. We can't do all of
data.frame in Python (due to a few general syntactic issues such as the
mechanism behind comparisons and the way indexing works), but we can get a
lot of it done w/ the right class wrappers.
It's yet another one of my long-term projects, but it's still in the
design phase.
--david
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hyoon@nyptsrv1.etsd.ml.com Fri Jul 25 17:54:47 1997
From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant)
Date: Fri, 25 Jul 1997 12:54:47 -0400
Subject: [MATRIX-SIG] hash() function w/ NumPy
Message-ID: <199707251654.MAA08227@ptq3.etsd>
div_prj =3D []
id_lst =3D []
for i in range(0,len(prj_divs),1):
div_spstr =3D string.split(prj_divs[i])
id =3D div_spstr[3]
=04 if not id_lst.count(id): id_lst.append(id)
=09
div_prj.append([id_lst.index(id),chkdate(div_spstr[0]),eval(div_spstr[1])=
])
div_prj =3D array(div_prj)
Thank Aaron:
Here's something I came up with. Basically a similar idea, but it =
keeps
list of id_lst, which I can be pointed right back by the array =
div_prj.
This is not horribly bad, since it does do what I wanted with out =
reverse
of hash. I guess, I need to come up with Qstrategy for comparing =
indicies
from 2 matricies and print function that grabs original names back, =
etc...
=20
I am still unused to the idea that Python could pretty much do =
anything.
I can probably class these things and get back something extremely =
similar,
but better than Gauss's way of handling stuff. Although result may be =
bit
slower than single purpose Gauss, I can be much more creative.
=20
'till I see you,
Hoon,
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From aaron_watters@msn.com Sun Jul 27 13:36:51 1997
From: aaron_watters@msn.com (aaron watters)
Date: Sun, 27 Jul 97 12:36:51 UT
Subject: [MATRIX-SIG] vrml for plotting
Message-ID: <UPMAIL01.199707271236410463@msn.com>
Hey kids! I love the Python list. It's quite obvious
to me (now that I've been remindided)
that for many forms of sci-viz vrml is the right
tool!
Maybe soon I'll have a closer look at this.
(and it'll pop up in your web browser too!)
I quote one of the esteemed mitchell brothers:
Subject: Re: VRML 2.0 modeling
From: John Mitchell <johnm@magnet.com>
Date: 1997/07/25
Message-Id: <Pine.SGI.3.96.970725104840.26170H-100000@lemur.magnet.com>
Newsgroups: comp.lang.python
[More Headers]
On Thu, 24 Jul 1997, Lance Ellinghaus wrote:
> Does anyone have any classes that can be used to build simple VRML 2.0
> scenes and output
> them to a web browser?
I have a bunch of classes, basically mapping VRML objects to Python
classes. It's cool in that you, with a little headscratching, can
translate VRML2 example "wrl" files into 100% python. The main problem
is that the classes are very "blobby", do no error checking, and lots of
things that are "obvious" wont work. VRML requires certain things in
certain orders and if they aint, you get nothing.
But, last time I checked, the code works. I translated a few examples to
Python classes, and wrote a VRML Clock.
Its at:
http://people.magnet.com/~johnm/VRML/
hope this helps.
- j
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Mon Jul 28 15:00:23 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Mon, 28 Jul 1997 10:00:23 -0400
Subject: [MATRIX-SIG] Proposal: "Tables" in NumPy
Message-ID: <199707281356.JAA26322@dante.mh.lucent.com>
I think you can approximate this, maybe, by cross
breeding kjbuckets.kjDict structures and arrays.
I'll have to have a think on this.
For a bit of a performance penalty (more loops in Python)
you could use standard python dictionaries.
On the matrix end I have this inkling that a "dual of
take" might come in handy here. In fact, I'm pretty sure
of it. Mind if I have a look at your code?
This is quite an exciting concept, I think.
-- Aaron Watters
ps: more generally, you'd like full integration of matrices
in relational algebra-style tables (ie, sets of tuples).
----------
> From: Jim Phillips <jim@ks.uiuc.edu>
> To: matrix-sig@python.org
> I have written something like this in python, but only in two dimensions and
> it is rather slow. I would love to see a full N-dimension implementation as a
> part of NumPy. Perhaps a good interface would look something like this:
>
> >>> a # a 2-D table
> dogs cats birds
> 2 4 6
> 1 2 3
> 9 8 7
> >>> a['dogs'] # returns array (reference!)
> 2 1 9
> >>> a[:,1] # returns table (reference!)
> dogs cats birds
> 1 2 3
> >>> a[('dogs','birds'),0:2] # returns table (reference!)
> dogs birds
> 2 6
> 1 3
>
> This could really be a killer feature for data analysis since we would no
> longer have to choose between C-style structures and FORTRAN-style arrays!
>
> -Jim Phillips
>
> _______________
> MATRIX-SIG - SIG on Matrix Math for Python
>
> send messages to: matrix-sig@python.org
> administrivia to: matrix-sig-request@python.org
> _______________
>
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From c9603085@zeus.hud.ac.uk Tue Jul 29 11:18:50 1997
From: c9603085@zeus.hud.ac.uk (MISS. K.L.COLLIER)
Date: Tue, 29 Jul 1997 11:18:50 +0100
Subject: [MATRIX-SIG] convolution
Message-ID: <199707291018.LAA04298@artemis.csm>
Hi,
I want to find the convolution of two images and am not sure how
to do it. There is a convolve function but its only 1D and there are
2D fft's but not an inverse 2D fft? How do I do it? Any ideas?
Thanks
Karen Collier c9603085@zeus.hud.ac.uk
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From Daniel.Michelson@smhi.se Tue Jul 29 14:37:45 1997
From: Daniel.Michelson@smhi.se (Daniel Michelson)
Date: Tue, 29 Jul 1997 15:37:45 +0200
Subject: [MATRIX-SIG] propaganda
Message-ID: <33DDF229.52BF@smhi.se>
Greetings!
Python is already being used at NASA and we want it to get a foot in the
door at EUMETSAT too! We're involved in a very large project designed to
prototype the next generation methods and systems for processing
meteorological satellite data for all EUMETSAT members. We're part of a
consortium of European weather services of which we're the only ones
with any knowledge of and experience with Python. =
We're now faced with the task of convincing our partners that Python is
worth using. So far, the attitude seems to be that Python is assumed
"guilty" until proven "innocent"; the preferences among our collegues
are ANSI C and Fortran 77 administered by sh or ksh... :-(
There is a lot of good information at python.org but: =
If anyone has written a page or two (or more) previously on the benefits
of using Python when building a large distributed system, would you
please share it with me?
Please help! We don't want to be caught in a time warp and forced to use
old, slow methods!
best, -d
+------------------+--------------------------------------------------+
| | Daniel B. Michelson |
| /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute|
| \ | \/ | |__| | | S-601 76 Norrk=F6ping, Sweden |=
| \ | | | | | | Telephone: +46 11 - 15 84 94 |
| \/ | | | | | | Telefax: +46 11 - 17 02 07 |
| | Internet: Daniel.Michelson@smhi.se |
+------------------+--------------------------------------------------+
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Tue Jul 29 16:05:28 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Tue, 29 Jul 1997 11:05:28 -0400
Subject: [MATRIX-SIG] propaganda
Message-ID: <199707291501.LAA08323@dante.mh.lucent.com>
Is there any chance you can convince them
that Python is just another C library with which
to build tools? Maybe they'd be more receptive
if it isn't a black and white situation.
If you do, of course, I expect the other tools will
be able to sit back and let Python do most of the
work, however.
I may send you some email addresses privately.
The following come to mind:
infoseek
worldnet (att)
industrial light and magic (ILM)
ILU (xerox PARC)
gstein@microsoft.com (this one I'll send publicly :) ).
There is a "who uses python" page at http://www.python.org
too. -- Aaron Watters
----------
From: Daniel Michelson <Daniel.Michelson@smhi.se>
To: guido@CNRI.Reston.Va.US
Cc: matrix-sig@python.org; image-sig@python.org
Subject: [MATRIX-SIG] propaganda
Date: Tuesday, July 29, 1997 9:37 AM
Greetings!
Python is already being used at NASA and we want it to get a foot in the
door at EUMETSAT too! We're involved in a very large project designed to
prototype the next generation methods and systems for processing
meteorological satellite data for all EUMETSAT members. We're part of a
consortium of European weather services of which we're the only ones
with any knowledge of and experience with Python.
We're now faced with the task of convincing our partners that Python is
worth using. So far, the attitude seems to be that Python is assumed
"guilty" until proven "innocent"; the preferences among our collegues
are ANSI C and Fortran 77 administered by sh or ksh... :-(
There is a lot of good information at python.org but:
If anyone has written a page or two (or more) previously on the benefits
of using Python when building a large distributed system, would you
please share it with me?
Please help! We don't want to be caught in a time warp and forced to use
old, slow methods!
best, -d
+------------------+--------------------------------------------------+
| | Daniel B. Michelson |
| /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute|
| \ | \/ | |__| | | S-601 76 Norrköping, Sweden |
| \ | | | | | | Telephone: +46 11 - 15 84 94 |
| \/ | | | | | | Telefax: +46 11 - 17 02 07 |
| | Internet: Daniel.Michelson@smhi.se |
+------------------+--------------------------------------------------+
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
----------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From hochberg@wwa.com Tue Jul 29 16:12:27 1997
From: hochberg@wwa.com (Timothy A. Hochberg)
Date: Tue, 29 Jul 1997 10:12:27 -0500 (CDT)
Subject: [MATRIX-SIG] convolution
In-Reply-To: <199707291018.LAA04298@artemis.csm>
Message-ID: <Pine.BSI.3.95.970729100841.15195A-100000@shoga.wwa.com>
On Tue, 29 Jul 1997, MISS. K.L.COLLIER wrote:
> I want to find the convolution of two images and am not sure how
> to do it. There is a convolve function but its only 1D and there are
> 2D fft's but not an inverse 2D fft? How do I do it? Any ideas?
Looking at FFT.py, fft2d is implementd as:
def fft2d(a, s=None, axes=(-2,-1)):
return _raw_fft2d(a,s,axes,fft)
So, I'd try defining inverse_fft2d as:
def inverse_fft2d(a, s=None, axes=(-2,-1)):
return _raw_fft2d(a,s,axes,inverse_fft)
(Note that you have to explicitly import _raw_fft2d, import * won't pull
it in.)
Make sure you test this - it worked for one test case, but I wouldn't call
it tested.
____
/im
+------------------------------------------------+
|Tim Hochberg Research Assistant |
|hochberg <at> wwa.com University of Illinois |
| Electrical Engineering |
+------------------------------------------------+
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From Dale_Bertrand@brown.edu Tue Jul 29 22:30:24 1997
From: Dale_Bertrand@brown.edu (Dale Bertrand)
Date: Tue, 29 Jul 1997 17:30:24 -0400 (EDT)
Subject: [MATRIX-SIG] PyArrayObjects
In-Reply-To: <199707291501.LAA08323@dante.mh.lucent.com>
Message-ID: <XFMail.970729174336.Dale_Bertrand@brown.edu>
I am writing a small python module in c which needs to access the internals of a
PyObject (which is really a PyArrayObject). I've been using the function:
PyArray_ContiguousFromObject to extract the PyArrayObject from the PyObject and
then modify the data via arrayobject->data.
Unfortunately, it seems to me that PyArray_ContiguousFromObject makes a copy of
the PyObject which I _must_ return to see my changes. I would like my c functio
n to simply access and modify the array data without making a copy.
How do I access the array data without copying the Object?
A snip of code:
static PyObject *
do_array(PyObject *self,PyObject *args)
{
PyObject *op; /* <- I would like to edit this */
PyArrayObject *mp; /* <-without making this */
int h,w;
int ubound[1];
int lbound[1];
TRY(PyArg_ParseTuple(args, "Oii", &op, &lbound , &ubound));
TRY(mp = (PyArrayObject *)PyArray_ContiguousFromObject(op,PyArray_CHAR, 2, 2
));
h = mp->dimensions[0];
w = mp->dimensions[1];
threshold((unsigned char *)mp->data, (int *)lbound , (int*)ubound, 1, w*h);
Py_DECREF(mp);
return (PyObject *)mp; /* I do not want to have to return this */
}
Thanks
-Dale Bertrand
----------------------------------
E-Mail: Dale Bertrand <Dale_Bertrand@brown.edu>
Date: 29-Jul-97
----------------------------------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From pas@xis.xerox.com Wed Jul 30 00:28:12 1997
From: pas@xis.xerox.com (Perry A. Stoll)
Date: Tue, 29 Jul 1997 16:28:12 PDT
Subject: [MATRIX-SIG] PyArrayObjects
In-Reply-To: <XFMail.970729174336.Dale_Bertrand@brown.edu>
Message-ID: <Pine.SUN.3.96.970729191854.11649B-100000@fantod>
On Tue, 29 Jul 1997, Dale Bertrand wrote:
Unfortunately, it seems to me that PyArray_ContiguousFromObject makes a copy of
the PyObject which I _must_ return to see my changes. I would like my c functio
n to simply access and modify the array data without making a copy.
I think the more python way of doing things is to have functions return a
copy and leave the original unchanged. This accounts for the behaviour
you correctly identified for PyArray_ContiguousFromObject. But if you have
to modify the original, you have to...
How do I access the array data without copying the Object?
How about something like this (untested)?
static PyObject *
do_array(PyObject *self,PyObject *args)
{
PyObject *op; /* <- I would like to edit this */
PyArrayObject *mp; /* <-without making this */
int h,w;
int ubound;
int lbound;
TRY(PyArg_ParseTuple(args, "Oii", &op, &lbound , &ubound));
/* check that it really is an array object. I believe this
will forbid inheriting in Python from C objects. Oh well. */
TRY(PyArray_Check(op));
/* a simple cast now that you know it really is an array object */
mp = (PyArrayObject *)op;
h = mp->dimensions[0];
w = mp->dimensions[1];
threshold((unsigned char *)mp->data, &lbound , &ubound, 1,w*h);
return op;
}
Hope that helps.
-Perry
<pas@xis.xerox.com>
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Wed Jul 30 19:49:17 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Wed, 30 Jul 1997 14:49:17 -0400
Subject: [MATRIX-SIG] simplex algorithm?
Message-ID: <199707301933.PAA26929@dante.mh.lucent.com>
Anybody got a simplex algorithm
(simple linear optimization)
or is this something else to add
to my todo list :)? -- Aaron Watters
ps: I ask because a friend and I
just determined that laying out
html (with tables and frames)
can be formulated as a fairly sparse
linear maximization problem.
Threw me back to my fond memories
of beautiful England, where I once
studied such stuff, too many years
ago... (sigh)
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From amullhau@ix.netcom.com Wed Jul 30 23:11:34 1997
From: amullhau@ix.netcom.com (Andrew P. Mullhaupt)
Date: Wed, 30 Jul 1997 18:11:34 -0400
Subject: [MATRIX-SIG] simplex algorithm?
In-Reply-To: <199707301933.PAA26929@dante.mh.lucent.com>
Message-ID: <3.0.1.32.19970730181134.00838ea0@popd.netcruiser>
At 02:49 PM 7/30/97 -0400, Aaron Watters wrote:
>Anybody got a simplex algorithm
>(simple linear optimization)
>or is this something else to add
>to my todo list :)? -- Aaron Watters
>
>ps: I ask because a friend and I
> just determined that laying out
> html (with tables and frames)
> can be formulated as a fairly sparse
> linear maximization problem.
Stop! Do not implement the simplex algorithm from eight billion years ago
until you look at the new 'interior point' and primal-dual methods.
A good book is Steven Wright's book on primal-dual methods. Get this book,
which has pointers to code you can get on the net.
Then, if you want to write code, wrap up this code in a nice module.
Later,
Andrew Mullhaupt
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From da@maigret.cog.brown.edu Thu Jul 31 03:24:36 1997
From: da@maigret.cog.brown.edu (David Ascher)
Date: Wed, 30 Jul 1997 22:24:36 -0400 (EDT)
Subject: [MATRIX-SIG] simplex algorithm?
In-Reply-To: <3.0.1.32.19970730181134.00838ea0@popd.netcruiser>
Message-ID: <Pine.SGI.3.95q.970730222122.7869C-100000@maigret>
Aaron:
> Anybody got a simplex algorithm (simple linear optimization) or is this
> something else to add to my todo list :)? -- Aaron Watters
>
> ps: I ask because a friend and I just determined that laying out html
> (with tables and frames) can be formulated as a fairly sparse linear
> maximization problem. Threw me back to my fond memories of beautiful
> England, where I once studied such stuff, too many years ago... (sigh)
Andrew:
> Stop! Do not implement the simplex algorithm from eight billion years ago
> until you look at the new 'interior point' and primal-dual methods.
I guess now we know how old Aaron is, +/- a few years.
--david
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From jrwebb@goodnet.com Thu Jul 31 04:41:52 1997
From: jrwebb@goodnet.com (James R. Webb)
Date: Wed, 30 Jul 1997 20:41:52 -0700
Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG
Message-ID: <199707310339.UAA14455@mail.goodnet.com>
The programmers out there may want to look at what the developers of PerlDL
are doing. They have created a nice plotting interface to PGPlot and
OpenGL/MesaGL. Might want to check out
http://www.aao.gov.au/local/www/kgb/perldl/ to stimulate the discussion.
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From Daniel.Michelson@smhi.se Tue Jul 29 14:37:45 1997
From: Daniel.Michelson@smhi.se (Daniel Michelson)
Date: Tue, 29 Jul 1997 15:37:45 +0200
Subject: [MATRIX-SIG] propaganda
Message-ID: <33DDF229.52BF@smhi.se>
Greetings!
Python is already being used at NASA and we want it to get a foot in the
door at EUMETSAT too! We're involved in a very large project designed to
prototype the next generation methods and systems for processing
meteorological satellite data for all EUMETSAT members. We're part of a
consortium of European weather services of which we're the only ones
with any knowledge of and experience with Python. =
We're now faced with the task of convincing our partners that Python is
worth using. So far, the attitude seems to be that Python is assumed
"guilty" until proven "innocent"; the preferences among our collegues
are ANSI C and Fortran 77 administered by sh or ksh... :-(
There is a lot of good information at python.org but: =
If anyone has written a page or two (or more) previously on the benefits
of using Python when building a large distributed system, would you
please share it with me?
Please help! We don't want to be caught in a time warp and forced to use
old, slow methods!
best, -d
+------------------+--------------------------------------------------+
| | Daniel B. Michelson |
| /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute|
| \ | \/ | |__| | | S-601 76 Norrk=F6ping, Sweden |=
| \ | | | | | | Telephone: +46 11 - 15 84 94 |
| \/ | | | | | | Telefax: +46 11 - 17 02 07 |
| | Internet: Daniel.Michelson@smhi.se |
+------------------+--------------------------------------------------+
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
From arw@dante.mh.lucent.com Tue Jul 29 16:05:28 1997
From: arw@dante.mh.lucent.com (Aaron Watters)
Date: Tue, 29 Jul 1997 11:05:28 -0400
Subject: [MATRIX-SIG] propaganda
Message-ID: <199707291501.LAA08323@dante.mh.lucent.com>
Is there any chance you can convince them
that Python is just another C library with which
to build tools? Maybe they'd be more receptive
if it isn't a black and white situation.
If you do, of course, I expect the other tools will
be able to sit back and let Python do most of the
work, however.
I may send you some email addresses privately.
The following come to mind:
infoseek
worldnet (att)
industrial light and magic (ILM)
ILU (xerox PARC)
gstein@microsoft.com (this one I'll send publicly :) ).
There is a "who uses python" page at http://www.python.org
too. -- Aaron Watters
----------
From: Daniel Michelson <Daniel.Michelson@smhi.se>
To: guido@CNRI.Reston.Va.US
Cc: matrix-sig@python.org; image-sig@python.org
Subject: [MATRIX-SIG] propaganda
Date: Tuesday, July 29, 1997 9:37 AM
Greetings!
Python is already being used at NASA and we want it to get a foot in the
door at EUMETSAT too! We're involved in a very large project designed to
prototype the next generation methods and systems for processing
meteorological satellite data for all EUMETSAT members. We're part of a
consortium of European weather services of which we're the only ones
with any knowledge of and experience with Python.
We're now faced with the task of convincing our partners that Python is
worth using. So far, the attitude seems to be that Python is assumed
"guilty" until proven "innocent"; the preferences among our collegues
are ANSI C and Fortran 77 administered by sh or ksh... :-(
There is a lot of good information at python.org but:
If anyone has written a page or two (or more) previously on the benefits
of using Python when building a large distributed system, would you
please share it with me?
Please help! We don't want to be caught in a time warp and forced to use
old, slow methods!
best, -d
+------------------+--------------------------------------------------+
| | Daniel B. Michelson |
| /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute|
| \ | \/ | |__| | | S-601 76 Norrköping, Sweden |
| \ | | | | | | Telephone: +46 11 - 15 84 94 |
| \/ | | | | | | Telefax: +46 11 - 17 02 07 |
| | Internet: Daniel.Michelson@smhi.se |
+------------------+--------------------------------------------------+
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________
----------
_______________
MATRIX-SIG - SIG on Matrix Math for Python
send messages to: matrix-sig@python.org
administrivia to: matrix-sig-request@python.org
_______________