From eagle at newdream.net  Mon Oct  1 12:32:33 2007
From: eagle at newdream.net (Eagle Jones)
Date: Mon, 1 Oct 2007 09:32:33 -0700
Subject: [Numpy-discussion] Naming a slice index?
Message-ID: <f33199570710010932o32be40d9u5a502167f5fbc1c3@mail.gmail.com>

New to python and numpy; hopefully I'm missing something obvious. I'd
like to be able to slice an array with a name. For example:

_T = 6:10
_R = 10:15
A = identity(20)
foo = A[_T, _R]

This doesn't work. Nor does _T=range(6:10); _R = range(10:15). Any ideas?

Thanks,
Eagle


From tim.hochberg at ieee.org  Mon Oct  1 12:35:28 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Mon, 1 Oct 2007 09:35:28 -0700
Subject: [Numpy-discussion] Naming a slice index?
In-Reply-To: <f33199570710010932o32be40d9u5a502167f5fbc1c3@mail.gmail.com>
References: <f33199570710010932o32be40d9u5a502167f5fbc1c3@mail.gmail.com>
Message-ID: <e4412d6b0710010935g6df0d2a9mc5c789642742b633@mail.gmail.com>

On 10/1/07, Eagle Jones <eagle at newdream.net> wrote:
>
> New to python and numpy; hopefully I'm missing something obvious. I'd
> like to be able to slice an array with a name. For example:
>
> _T = 6:10
> _R = 10:15
> A = identity(20)
> foo = A[_T, _R]
>
> This doesn't work. Nor does _T=range(6:10); _R = range(10:15). Any ideas?


Try slice(10,15)


Thanks,
> Eagle
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071001/51e9ca9c/attachment.html>

From cimrman3 at ntc.zcu.cz  Mon Oct  1 12:35:28 2007
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Mon, 01 Oct 2007 18:35:28 +0200
Subject: [Numpy-discussion] Naming a slice index?
In-Reply-To: <f33199570710010932o32be40d9u5a502167f5fbc1c3@mail.gmail.com>
References: <f33199570710010932o32be40d9u5a502167f5fbc1c3@mail.gmail.com>
Message-ID: <470121D0.3010707@ntc.zcu.cz>

Eagle Jones wrote:
> New to python and numpy; hopefully I'm missing something obvious. I'd
> like to be able to slice an array with a name. For example:
> 
> _T = 6:10

_T = slice( 6, 10 )

...



From aisaac at american.edu  Mon Oct  1 21:40:26 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Mon, 1 Oct 2007 21:40:26 -0400
Subject: [Numpy-discussion] numpy.distutils.cpuinfo bugs?
Message-ID: <Mahogany-0.67.0-1068-20071001-214026.00@american.edu>

Two problems noted::

    >>> from numpy.distutils import cpuinfo
    0
    >>> cpu = cpuinfo.cpuinfo()
    >>> print "Has SSE2? %s"%cpu._has_sse2()
    Has SSE2? False
    >>>

1. The printed '0' traces to an undesirable print statement.
(I've reported this before.)

2. The 'False" report is on a Pentium M.  Should that not be 
True? <URL:http://en.wikipedia.org/wiki/Pentium_M>
Or am I misusing the test?

Thank you,
Alan Isaac






From millman at berkeley.edu  Tue Oct  2 01:37:58 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 1 Oct 2007 22:37:58 -0700
Subject: [Numpy-discussion] adopting Python Style Guide for classes
Message-ID: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>

Hello,

NumPy and SciPy should conform with Guido's style guide as closely as possible:
http://www.python.org/dev/peps/pep-0008/
The only serious divergence that I am aware of between the NumPy and
SciPy codebase and the Python recommended standards is in class
naming.  According to Guido,  class names should use the CapWords
convention.  Most Python projects (eg, ETS, matploltlib) adhere to the
Python naming conventions and it is confusing that NumPy and SciPy
don't.

Currently, both NumPy and SciPy use either lower_underscore_separated
or CapWords for class names.  By my rough count, NumPy has about 752
classes using CapWords and 440 using  lower_underscore_separated.
While SciPy has about 256 classes using CapWords and 815 using
lower_underscore_separated.  Moreover, a far amount of the
lower_underscore_separated classes are tests.  That a number of
classes currently use CapWords despite the NumPy convention to use
lower_underscore_separated names may indicate it would make more sense
to switch to the Python convention.

I am hoping that most of you agree with the general principle of
bringing NumPy and SciPy into compliance with the standard naming
conventions.  I have already talked to a few people including Travis
about this switch and so far everyone supports the change in general.
Please let me know if you have any major objections to adopting the
Python class naming convention.

Once we have agreed to using CapWords for classes, we will need to
decide what to do about our existing class names.  Obviously, it is
important that we are careful to not break a lot of code just to bring
our class names up to standards.  So at the very least, we probably
won't be able to just change every class name until NumPy 1.1.0 is
released.

Here is what I propose for NumPy:
1.  Change the names of the TestCase class names to use CapWords.  I
just checked in a change to allow TestCase classes to be prefixed with
either 'test' or 'Test':
http://projects.scipy.org/scipy/numpy/changeset/4144
If no one objects, I plan to go through all the NumPy unit tests and
change their names to CapWords.  Ideally, I would like to get at least
this change in before NumPy 1.0.4.
2.  Any one adding a new class to NumPy would use CapWords.
3.  When we release NumPy 1.1, we will convert all (or almost all)
class names to CapWords.  There is no reason to worry about the exact
details of this conversion at this time.  I just would like to get a
sense whether, in general, this seems like a good direction to move
in.  If so, then after we get steps 1 and 2 completed we can start
discussing how to handle step 3.

Cheers,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From pearu at cens.ioc.ee  Tue Oct  2 03:03:03 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Tue, 02 Oct 2007 09:03:03 +0200
Subject: [Numpy-discussion] numpy.distutils.cpuinfo bugs?
In-Reply-To: <Mahogany-0.67.0-1068-20071001-214026.00@american.edu>
References: <Mahogany-0.67.0-1068-20071001-214026.00@american.edu>
Message-ID: <4701ED27.9060800@cens.ioc.ee>



Alan G Isaac wrote:
> Two problems noted::
> 
>     >>> from numpy.distutils import cpuinfo
>     0
>     >>> cpu = cpuinfo.cpuinfo()
>     >>> print "Has SSE2? %s"%cpu._has_sse2()
>     Has SSE2? False
>     >>>
> 
> 1. The printed '0' traces to an undesirable print statement.
> (I've reported this before.)

Travis seemed to fix this about two weeks ago.

> 2. The 'False" report is on a Pentium M.  Should that not be 
> True? <URL:http://en.wikipedia.org/wiki/Pentium_M>
> Or am I misusing the test?

What OS are you using? If Linux, then can you send the content
of /proc/cpuinfo?

Pearu


From david at ar.media.kyoto-u.ac.jp  Tue Oct  2 03:00:48 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 02 Oct 2007 16:00:48 +0900
Subject: [Numpy-discussion] numpy.distutils.cpuinfo bugs?
In-Reply-To: <4701ED27.9060800@cens.ioc.ee>
References: <Mahogany-0.67.0-1068-20071001-214026.00@american.edu>
	<4701ED27.9060800@cens.ioc.ee>
Message-ID: <4701ECA0.3030408@ar.media.kyoto-u.ac.jp>

Pearu Peterson wrote:
>
> Alan G Isaac wrote:
>
>> 2. The 'False" report is on a Pentium M.  Should that not be 
>> True? <URL:http://en.wikipedia.org/wiki/Pentium_M>
>> Or am I misusing the test?
>
> What OS are you using? If Linux, then can you send the content
> of /proc/cpuinfo?
On Windows, you can use this software, I think:

http://www.intel.com/support/processors/tools/frequencyid/

which is essentially the same method (getting the info from the cpuid 
instruction provided on x86).

cheers,

David


From pearu at cens.ioc.ee  Tue Oct  2 03:12:43 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Tue, 02 Oct 2007 09:12:43 +0200
Subject: [Numpy-discussion] adopting Python Style Guide for classes
In-Reply-To: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>
References: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>
Message-ID: <4701EF6B.4090509@cens.ioc.ee>



Jarrod Millman wrote:
> Hello,
> 
..
> Please let me know if you have any major objections to adopting the
> Python class naming convention.

I don't object.

> Once we have agreed to using CapWords for classes, we will need to
> decide what to do about our existing class names.  Obviously, it is
> important that we are careful to not break a lot of code just to bring
> our class names up to standards.  So at the very least, we probably
> won't be able to just change every class name until NumPy 1.1.0 is
> released.
> 
> Here is what I propose for NumPy:
> 1.  Change the names of the TestCase class names to use CapWords.  I
> just checked in a change to allow TestCase classes to be prefixed with
> either 'test' or 'Test':
> http://projects.scipy.org/scipy/numpy/changeset/4144
> If no one objects, I plan to go through all the NumPy unit tests and
> change their names to CapWords.  Ideally, I would like to get at least
> this change in before NumPy 1.0.4.

It is safe to change all classes in tests to CamelCase.

> 2.  Any one adding a new class to NumPy would use CapWords.
> 3.  When we release NumPy 1.1, we will convert all (or almost all)
> class names to CapWords.  There is no reason to worry about the exact
> details of this conversion at this time.  I just would like to get a
> sense whether, in general, this seems like a good direction to move
> in.  If so, then after we get steps 1 and 2 completed we can start
> discussing how to handle step 3.

After fixing the class names in tests then how many classes use
camelcase style in numpy/distutils? How many of them are implementation
specific and how many of them are exposed to users? I think having this
statistics would be essential to make any decisions. Eg would we
need to introduce warnings for the few following releases of numpy/scipy
when camelcase class is used by user code, or not?

Pearu


From cookedm at physics.mcmaster.ca  Tue Oct  2 05:22:16 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Tue, 2 Oct 2007 05:22:16 -0400
Subject: [Numpy-discussion] adopting Python Style Guide for classes
In-Reply-To: <4701EF6B.4090509@cens.ioc.ee>
References: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>
	<4701EF6B.4090509@cens.ioc.ee>
Message-ID: <20071002092216.GA26977@arbutus.physics.mcmaster.ca>

On Tue, Oct 02, 2007 at 09:12:43AM +0200, Pearu Peterson wrote:
> 
> 
> Jarrod Millman wrote:
> > Hello,
> > 
> ..
> > Please let me know if you have any major objections to adopting the
> > Python class naming convention.
> 
> I don't object.

Me either.

> > 2.  Any one adding a new class to NumPy would use CapWords.
> > 3.  When we release NumPy 1.1, we will convert all (or almost all)
> > class names to CapWords.  There is no reason to worry about the exact
> > details of this conversion at this time.  I just would like to get a
> > sense whether, in general, this seems like a good direction to move
> > in.  If so, then after we get steps 1 and 2 completed we can start
> > discussing how to handle step 3.
> 
> After fixing the class names in tests then how many classes use
> camelcase style in numpy/distutils? How many of them are implementation
> specific and how many of them are exposed to users? I think having this
> statistics would be essential to make any decisions. Eg would we
> need to introduce warnings for the few following releases of numpy/scipy
> when camelcase class is used by user code, or not?

In numpy/distutils, there's the classes in command/* modules (but note
that distutils uses the same lower_case convention, so I'd say keep
them), cpu_info (none of which are user accessible; I'm working in there
now), and system_info (which are documented as user accessible). Poking
through the rest, it looks like only the system_info classes are ones
that we would expect users to subclass. We could document the lower_case
names as deprecated, and alias them to CamlCase versions.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca


From pearu at cens.ioc.ee  Tue Oct  2 05:40:00 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Tue, 02 Oct 2007 11:40:00 +0200
Subject: [Numpy-discussion] adopting Python Style Guide for classes
In-Reply-To: <4701EF6B.4090509@cens.ioc.ee>
References: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>
	<4701EF6B.4090509@cens.ioc.ee>
Message-ID: <470211F0.50209@cens.ioc.ee>



Pearu Peterson wrote:
..
> After fixing the class names in tests then how many classes use
> camelcase style in numpy/distutils? How many of them are implementation
..                   ^^^^^^^^^^^^^^^
Btw, I meant numpy/scipy here.

Pearu


From b3i4old02 at sneakemail.com  Tue Oct  2 08:27:05 2007
From: b3i4old02 at sneakemail.com (Michael Hoffman)
Date: Tue, 02 Oct 2007 13:27:05 +0100
Subject: [Numpy-discussion] Naming a slice index?
In-Reply-To: <f33199570710010932o32be40d9u5a502167f5fbc1c3@mail.gmail.com>
References: <f33199570710010932o32be40d9u5a502167f5fbc1c3@mail.gmail.com>
Message-ID: <fdtdfd$ij2$1@sea.gmane.org>

Eagle Jones wrote:
> New to python and numpy; hopefully I'm missing something obvious. I'd
> like to be able to slice an array with a name. For example:
> 
> _T = 6:10
> _R = 10:15

from numpy import index_exp

_T = index_exp[6:10]
_R = index_exp[10:15]

> A = identity(20)
> foo = A[_T, _R]



From b3i4old02 at sneakemail.com  Tue Oct  2 08:36:02 2007
From: b3i4old02 at sneakemail.com (Michael Hoffman)
Date: Tue, 02 Oct 2007 13:36:02 +0100
Subject: [Numpy-discussion] Run length encoding of an ndarray
Message-ID: <fdte08$kr0$1@sea.gmane.org>

I am trying to do a type of run-length encoding of a 2D array by axis. I 
have an array of values arranged along two axes, state and position. 
These are many (180, 30000) uint8 arrays.

I would like to have a list of tuples like

(state, start_pos, end_pos, values)

only separating out a set of values into a new tuple if they are all the 
same value in a run of at least 10 cells.

Is there a clever way to do this in NumPy? I was thinking of using 
itertools.groupby() but it would be nicer to have something faster.



From b3i4old02 at sneakemail.com  Tue Oct  2 09:13:48 2007
From: b3i4old02 at sneakemail.com (Michael Hoffman)
Date: Tue, 02 Oct 2007 14:13:48 +0100
Subject: [Numpy-discussion] Run length encoding of an ndarray
In-Reply-To: <fdte08$kr0$1@sea.gmane.org>
References: <fdte08$kr0$1@sea.gmane.org>
Message-ID: <fdtg70$sl6$1@sea.gmane.org>

Michael Hoffman wrote:
> I am trying to do a type of run-length encoding of a 2D array by axis. I 
> have an array of values arranged along two axes, state and position. 
> These are many (180, 30000) uint8 arrays.
> 
> I would like to have a list of tuples like
> 
> (state, start_pos, end_pos, values)
> 
> only separating out a set of values into a new tuple if they are all the 
> same value in a run of at least 10 cells.
> 
> Is there a clever way to do this in NumPy? I was thinking of using 
> itertools.groupby() but it would be nicer to have something faster.

I just realized I could convert this to a string and use something like 
re.compile(r"(.)\1{9,}").finditer() but I'm still wondering if there's 
something more NumPythonic.



From stefan at sun.ac.za  Tue Oct  2 09:36:42 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Tue, 2 Oct 2007 15:36:42 +0200
Subject: [Numpy-discussion] Run length encoding of an ndarray
In-Reply-To: <fdte08$kr0$1@sea.gmane.org>
References: <fdte08$kr0$1@sea.gmane.org>
Message-ID: <20071002133642.GJ6930@mentat.za.net>

On Tue, Oct 02, 2007 at 01:36:02PM +0100, Michael Hoffman wrote:
> I am trying to do a type of run-length encoding of a 2D array by axis. I 
> have an array of values arranged along two axes, state and position. 
> These are many (180, 30000) uint8 arrays.
> 
> I would like to have a list of tuples like
> 
> (state, start_pos, end_pos, values)
> 
> only separating out a set of values into a new tuple if they are all the 
> same value in a run of at least 10 cells.

This snippet does run-length encoding for one row.

x = N.array([1,1,1,1,1,1,1,1,1,1,0,0,2,2,9,9,9,9,9,9,9,9,9,9])

pos, = N.where(N.diff(x) != 0)
pos = N.concatenate(([0],pos+1,[len(x)]))
rle = [(a,b,x[a]) for (a,b) in zip(pos[:-1],pos[1:])]

[(0, 10, 1), (10, 12, 0), (12, 14, 2), (14, 24, 9)]

Maybe you can use that as a starting point.

Cheers
St?fan


From aisaac at american.edu  Tue Oct  2 10:05:06 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 2 Oct 2007 10:05:06 -0400
Subject: [Numpy-discussion] numpy.distutils.cpuinfo bugs?
In-Reply-To: <4701ED27.9060800@cens.ioc.ee>
References: <Mahogany-0.67.0-1068-20071001-214026.00@american.edu><4701ED27.9060800@cens.ioc.ee>
Message-ID: <Mahogany-0.67.0-1576-20071002-100506.00@american.edu>

On Tue, 02 Oct 2007, Pearu Peterson apparently wrote:
>> 1. The printed '0' traces to an undesirable print statement. 
>> (I've reported this before.)

> Travis seemed to fix this about two weeks ago. 

Sorry for the noise.

>> 2. The 'False" report is on a Pentium M.  Should that not be 
>> True? <URL:http://en.wikipedia.org/wiki/Pentium_M> 
>> Or am I misusing the test? 

> What OS are you using? If Linux, then can you send the content 
> of /proc/cpuinfo? 

It is a student's machine, running Windows XP.
I did check the system information to confirm that it is
really a Pentium M.  I will be happy to convey any suggested
diagnostics to the student.  (I have sent David's 
suggestion.)

Thank you,
Alan




From aisaac at american.edu  Tue Oct  2 10:58:33 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 2 Oct 2007 10:58:33 -0400
Subject: [Numpy-discussion] numpy.distutils.cpuinfo bugs?
In-Reply-To: <Mahogany-0.67.0-1068-20071001-214026.00@american.edu>
References: <Mahogany-0.67.0-1068-20071001-214026.00@american.edu>
Message-ID: <Mahogany-0.67.0-1576-20071002-105833.00@american.edu>

Here is the processor information using the Intel utility.

Cheers,
Alan Isaac


Intel(R) Processor Identification Utility
Version: 3.7.20070907
Time Stamp: 2007/10/02 14:21:29
Number of processors in system: 1
Current processor: #1
Cores per processor: 1
Disabled cores per processor: 0
Processor Name: Intel(R) Pentium(R) M processor 735   1.70GHz
Type: 0
Family: 6
Model: D
Stepping: 6
Revision: 18
L1 Instruction Cache: 32 KB
L1 Data Cache: 32 KB
L2 Cache: 2 MB
Packaging: ?FCPGA/?FCBGA
EIST: Yes
MMX(TM): Yes
SSE: Yes
SSE2: Yes
SSE3: No
SSE4: No
Enhanced Halt State: No
Execute Disable Bit: No
Hyper-Threading Technology: No
Intel(R) 64 Architecture: No
Intel(R) Virtualization Technology: No
Expected Processor Frequency: 1.70 GHz
Reported Processor Frequency: 1.70 GHz
Expected System Bus Frequency: 400 MHz
Reported System Bus Frequency: 400 MHz
*************************************************************





From Chris.Barker at noaa.gov  Tue Oct  2 12:52:14 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 02 Oct 2007 09:52:14 -0700
Subject: [Numpy-discussion] adopting Python Style Guide for classes
In-Reply-To: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>
References: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>
Message-ID: <4702773E.4070500@noaa.gov>

Jarrod Millman wrote:
> I am hoping that most of you agree with the general principle of
> bringing NumPy and SciPy into compliance with the standard naming
> conventions. 

+1

> 3.  When we release NumPy 1.1, we will convert all (or almost all)
> class names to CapWords.  

What's the backwards-compatible plan?

- keep the old names as aliases?
- raise deprecation warnings?

What about factory functions that kind of look like they might be 
classes -- numpy.array() comes to mind. Though maybe using CamelCase for 
the real classes will help folks understand the difference.

What is a "class" in this case -- with new-style classes, there is no 
distinction between types and classes, so I guess they are all classes, 
which means lots of things like:

numpy.float32

etc. etc. etc. are classes. should they be CamelCase too?

NOTE:
for i in dir(numpy):
      if type(getattr(numpy, i)) == type(numpy.ndarray): print i

Yields 86 type objects.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From matthew.brett at gmail.com  Tue Oct  2 12:59:03 2007
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 2 Oct 2007 12:59:03 -0400
Subject: [Numpy-discussion] adopting Python Style Guide for classes
In-Reply-To: <4702773E.4070500@noaa.gov>
References: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>
	<4702773E.4070500@noaa.gov>
Message-ID: <1e2af89e0710020959q73bcd78u5d4e8c3fc782a601@mail.gmail.com>

On 10/2/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> Jarrod Millman wrote:
> > I am hoping that most of you agree with the general principle of
> > bringing NumPy and SciPy into compliance with the standard naming
> > conventions.

Excellent plan - and I think it will make the code considerably more
readable (and writeable).

> > 3.  When we release NumPy 1.1, we will convert all (or almost all)
> > class names to CapWords.
>
> What's the backwards-compatible plan?
>
> - keep the old names as aliases?
> - raise deprecation warnings?

Both seem good.  How about implementing both for the next minor
release, with the ability to turn the deprecation warnings off?

> What about factory functions that kind of look like they might be
> classes -- numpy.array() comes to mind. Though maybe using CamelCase for
> the real classes will help folks understand the difference.

Sounds right to me - factory function as function, class as class.

> What is a "class" in this case -- with new-style classes, there is no
> distinction between types and classes, so I guess they are all classes,
> which means lots of things like:
>
> numpy.float32
>
> etc. etc. etc. are classes. should they be CamelCase too?

I would vote for CamelCase in this case too.

Matthew


From robert.kern at gmail.com  Tue Oct  2 13:14:13 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 02 Oct 2007 12:14:13 -0500
Subject: [Numpy-discussion] adopting Python Style Guide for classes
In-Reply-To: <1e2af89e0710020959q73bcd78u5d4e8c3fc782a601@mail.gmail.com>
References: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>	<4702773E.4070500@noaa.gov>
	<1e2af89e0710020959q73bcd78u5d4e8c3fc782a601@mail.gmail.com>
Message-ID: <47027C65.8050905@gmail.com>

Matthew Brett wrote:
> On 10/2/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:

>> What is a "class" in this case -- with new-style classes, there is no
>> distinction between types and classes, so I guess they are all classes,
>> which means lots of things like:
>>
>> numpy.float32
>>
>> etc. etc. etc. are classes. should they be CamelCase too?
> 
> I would vote for CamelCase in this case too.

I would prefer to leave them as they are. While they are implemented as classes,
they're usually used as data. Also, they are more similar to builtin types than
classes one might write, and Python itself has a convention of leaving these
lowercase (e.g. int, float, etc.).

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From tim.hochberg at ieee.org  Tue Oct  2 13:20:25 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Tue, 2 Oct 2007 10:20:25 -0700
Subject: [Numpy-discussion] adopting Python Style Guide for classes
In-Reply-To: <4702773E.4070500@noaa.gov>
References: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>
	<4702773E.4070500@noaa.gov>
Message-ID: <e4412d6b0710021020r3a0d7f1j51c99237b457e40a@mail.gmail.com>

On 10/2/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
>
> Jarrod Millman wrote:
> > I am hoping that most of you agree with the general principle of
> > bringing NumPy and SciPy into compliance with the standard naming
> > conventions.
>
> +1
>
> > 3.  When we release NumPy 1.1, we will convert all (or almost all)
> > class names to CapWords.
>
> What's the backwards-compatible plan?
>
> - keep the old names as aliases?
> - raise deprecation warnings?
>
> What about factory functions that kind of look like they might be
> classes -- numpy.array() comes to mind. Though maybe using CamelCase for
> the real classes will help folks understand the difference.


I'm not a big fan of this kind of distinction distinction between factory
functions and "real" classes based on the concrete types of the objects.. In
some cases whether an object is a class or a factory function is simply an
implementation detail. The real distinction, as I see it, is whether the
object in question is meant to be subclassed. Thus an object is conceptually
a class if it can be called to create an instance and it can be usefully
subclassed.  A factory function, on the other hand is only meant to be
called to get an instance, regardless whether it is implemented as a class
or a function.

Of course, in the case of ndarray/array the distinction is clear since array
cannot be subclassed, so the minor rant above doesn't apply.


What is a "class" in this case -- with new-style classes, there is no
> distinction between types and classes, so I guess they are all classes,
> which means lots of things like:
>
> numpy.float32
>
> etc. etc. etc. are classes. should they be CamelCase too?


Core Python makes an additional distinction between types built into the
core (float, int, etc...) which are all lower case and library classes,
which generally use CapWords. So I guess there are two questions, IMO:

   1. Can float32 and friends be usefull subclassed? I suspect the answer
   is no. And in my head at least, these are conceptually mostly marker objects
   that can also be used as coercion functions, not classes. FWIW.
   2. Are these enough like builtin types to leave them alone in any
   case?

One approach would be CapWords the superclasses of these that are
subclassable, but leave the leaf types alone. For example, looking at
float32 and its bases :

   - numpy.generic -> numpy.Generic
   - numpy.number -> numpy.Number
   - numpy.inexact -> numpy.Inexact
   - numpy.floating -> numpy.Floating
   - numpy.float32 stays the same

This is probably a lot less painful in terms of backwards compatibility.

My $0.02.

NOTE:
> for i in dir(numpy):
>       if type(getattr(numpy, i)) == type(numpy.ndarray): print i
>
> Yields 86 type objects.
>
> -Chris
>
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071002/0d6ab426/attachment.html>

From faltet at carabos.com  Tue Oct  2 13:22:41 2007
From: faltet at carabos.com (Francesc Altet)
Date: Tue, 2 Oct 2007 19:22:41 +0200
Subject: [Numpy-discussion] adopting Python Style Guide for classes
In-Reply-To: <47027C65.8050905@gmail.com>
References: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>
	<1e2af89e0710020959q73bcd78u5d4e8c3fc782a601@mail.gmail.com>
	<47027C65.8050905@gmail.com>
Message-ID: <200710021922.41414.faltet@carabos.com>

A Tuesday 02 October 2007, Robert Kern escrigu?:
> Matthew Brett wrote:
> > On 10/2/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> >> What is a "class" in this case -- with new-style classes, there is
> >> no distinction between types and classes, so I guess they are all
> >> classes, which means lots of things like:
> >>
> >> numpy.float32
> >>
> >> etc. etc. etc. are classes. should they be CamelCase too?
> >
> > I would vote for CamelCase in this case too.
>
> I would prefer to leave them as they are. While they are implemented
> as classes, they're usually used as data. Also, they are more similar
> to builtin types than classes one might write, and Python itself has
> a convention of leaving these lowercase (e.g. int, float, etc.).

I would prefer to leave them as they are now too.  In addition to what 
Robert is saying, they are very heavily used in regular NumPy programs, 
and changing everything (both in code and docs) would be rather messy.

Cheers,

-- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"


From pgmdevlist at gmail.com  Tue Oct  2 13:23:43 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 2 Oct 2007 13:23:43 -0400
Subject: [Numpy-discussion] Default value in documentation
Message-ID: <200710021323.44022.pgmdevlist@gmail.com>

All,
I'm starting to update the documentation of maskedarray to the latest 
standard.
How should I represent the default value of an optional parameter ?
I was thinking something like

def function(a, value=None)
    """Does something
*Parameters*:
    a : {ndarray}
        Input array.
    value : {float} *[None]*
        Some value. If None, a default based on the dtype of a is used.
    """"

Suggestions/ideas are welcome.
Thanks a lot in advance
P.


From faltet at carabos.com  Tue Oct  2 13:28:55 2007
From: faltet at carabos.com (Francesc Altet)
Date: Tue, 2 Oct 2007 19:28:55 +0200
Subject: [Numpy-discussion] adopting Python Style Guide for classes
In-Reply-To: <e4412d6b0710021020r3a0d7f1j51c99237b457e40a@mail.gmail.com>
References: <c7009a550710012237g5f1cb779p8c25968de1de9e83@mail.gmail.com>
	<4702773E.4070500@noaa.gov>
	<e4412d6b0710021020r3a0d7f1j51c99237b457e40a@mail.gmail.com>
Message-ID: <200710021928.55751.faltet@carabos.com>

A Tuesday 02 October 2007, Timothy Hochberg escrigu?:
> One approach would be CapWords the superclasses of these that are
> subclassable, but leave the leaf types alone. For example, looking at
> float32 and its bases :
>
>    - numpy.generic -> numpy.Generic
>    - numpy.number -> numpy.Number
>    - numpy.inexact -> numpy.Inexact
>    - numpy.floating -> numpy.Floating
>    - numpy.float32 stays the same
>
> This is probably a lot less painful in terms of backwards
> compatibility.

Yeah.  I second this also.

-- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"


From robert.kern at gmail.com  Tue Oct  2 13:40:28 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 02 Oct 2007 12:40:28 -0500
Subject: [Numpy-discussion] Default value in documentation
In-Reply-To: <200710021323.44022.pgmdevlist@gmail.com>
References: <200710021323.44022.pgmdevlist@gmail.com>
Message-ID: <4702828C.1010507@gmail.com>

Pierre GM wrote:
> All,
> I'm starting to update the documentation of maskedarray to the latest 
> standard.
> How should I represent the default value of an optional parameter ?
> I was thinking something like
> 
> def function(a, value=None)
>     """Does something
> *Parameters*:
>     a : {ndarray}
>         Input array.
>     value : {float} *[None]*
>         Some value. If None, a default based on the dtype of a is used.
>     """"

I prefer this:

    value : {float}, optional
        Some value. If not provided, a default based on the dtype of a is used.

So label it optional and describe the default in prose in the parameter
description if necessary. While this is less descriptive for meaningful defaults
(axis=-1), the case you show above is precisely suited to it. None is not really
the default value, it's really just being used as a marker for "argument not
provided". And for the meaningful defaults, the function signature is more than
adequate to provide the necessary information.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From pgmdevlist at gmail.com  Tue Oct  2 14:10:48 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 2 Oct 2007 14:10:48 -0400
Subject: [Numpy-discussion] Default value in documentation
In-Reply-To: <4702828C.1010507@gmail.com>
References: <200710021323.44022.pgmdevlist@gmail.com>
	<4702828C.1010507@gmail.com>
Message-ID: <200710021410.50354.pgmdevlist@gmail.com>

On Tuesday 02 October 2007 13:40:28 Robert Kern wrote:
> I prefer this:
>
>     value : {float}, optional
>         Some value. If not provided, a default based on the dtype of a is
> used.
>...
>And for the meaningful defaults, the function 
> signature is more than adequate to provide the necessary information.

OK, thanks a lot.
Additional questions: is there any kind of standard to describe the attributes 
of a class, a la :IVariables: in epydoc ?


From charlesr.harris at gmail.com  Tue Oct  2 14:25:58 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 2 Oct 2007 12:25:58 -0600
Subject: [Numpy-discussion] Default value in documentation
In-Reply-To: <4702828C.1010507@gmail.com>
References: <200710021323.44022.pgmdevlist@gmail.com>
	<4702828C.1010507@gmail.com>
Message-ID: <e06186140710021125l4ce2159hf221af8d1fe1733f@mail.gmail.com>

On 10/2/07, Robert Kern <robert.kern at gmail.com> wrote:
>
> Pierre GM wrote:
> > All,
> > I'm starting to update the documentation of maskedarray to the latest
> > standard.
> > How should I represent the default value of an optional parameter ?
> > I was thinking something like
> >
> > def function(a, value=None)
> >     """Does something
> > *Parameters*:
> >     a : {ndarray}
> >         Input array.
> >     value : {float} *[None]*
> >         Some value. If None, a default based on the dtype of a is used.
> >     """"
>
> I prefer this:
>
>     value : {float}, optional
>         Some value. If not provided, a default based on the dtype of a is
> used.
>
> So label it optional and describe the default in prose in the parameter
> description if necessary. While this is less descriptive for meaningful
> defaults
> (axis=-1), the case you show above is precisely suited to it. None is not
> really
> the default value, it's really just being used as a marker for "argument
> not
> provided". And for the meaningful defaults, the function signature is more
> than
> adequate to provide the necessary information.


I've been putting the default value first among the options. Thus

    axis : {-1, integer}, optional

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071002/9e920743/attachment.html>

From stefan at sun.ac.za  Tue Oct  2 15:05:38 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Tue, 2 Oct 2007 21:05:38 +0200
Subject: [Numpy-discussion] Default value in documentation
In-Reply-To: <e06186140710021125l4ce2159hf221af8d1fe1733f@mail.gmail.com>
References: <200710021323.44022.pgmdevlist@gmail.com>
	<4702828C.1010507@gmail.com>
	<e06186140710021125l4ce2159hf221af8d1fe1733f@mail.gmail.com>
Message-ID: <20071002190538.GP6930@mentat.za.net>

On Tue, Oct 02, 2007 at 12:25:58PM -0600, Charles R Harris wrote:
>     So label it optional and describe the default in prose in the parameter
>     description if necessary. While this is less descriptive for meaningful
>     defaults
>     (axis=-1), the case you show above is precisely suited to it. None is not
>     really
>     the default value, it's really just being used as a marker for "argument
>     not
>     provided". And for the meaningful defaults, the function signature is more
>     than
>     adequate to provide the necessary information.
> 
> 
> I've been putting the default value first among the options. Thus
> 
>     axis : {-1, integer}, optional

Since the default is already listed in the function signature, I would
rather not duplicate it in the documentation.

St?fan


From cookedm at physics.mcmaster.ca  Tue Oct  2 15:20:54 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Tue, 2 Oct 2007 15:20:54 -0400
Subject: [Numpy-discussion] numpy.distutils.cpuinfo bugs?
In-Reply-To: <Mahogany-0.67.0-1068-20071001-214026.00@american.edu>
References: <Mahogany-0.67.0-1068-20071001-214026.00@american.edu>
Message-ID: <20071002192054.GA30238@arbutus.physics.mcmaster.ca>

On Mon, Oct 01, 2007 at 09:40:26PM -0400, Alan G Isaac wrote:
> Two problems noted::
> 
>     >>> from numpy.distutils import cpuinfo
>     0
>     >>> cpu = cpuinfo.cpuinfo()
>     >>> print "Has SSE2? %s"%cpu._has_sse2()
>     Has SSE2? False
>     >>>
> 
> 1. The printed '0' traces to an undesirable print statement.
> (I've reported this before.)
> 
> 2. The 'False" report is on a Pentium M.  Should that not be 
> True? <URL:http://en.wikipedia.org/wiki/Pentium_M>
> Or am I misusing the test?

Should be fixed now. There wasn't a test for the Pentium M, so it was
being caught by the Pentium III test (of which the M is based on).

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca


From aisaac at american.edu  Tue Oct  2 15:36:03 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Tue, 2 Oct 2007 15:36:03 -0400
Subject: [Numpy-discussion] Default value in documentation
In-Reply-To: <200710021410.50354.pgmdevlist@gmail.com>
References: <200710021323.44022.pgmdevlist@gmail.com><4702828C.1010507@gmail.com><200710021410.50354.pgmdevlist@gmail.com>
Message-ID: <Mahogany-0.67.0-216-20071002-153603.00@american.edu>

On Tue, 2 Oct 2007, Pierre GM apparently wrote:

> is there any kind of standard to describe the attributes 
> of a class, a la :IVariables: in epydoc ? 

I thought it was ...  :IVariables:
i.e., I thought the standard was reST as handled by epydoc.
<URL:http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines>
<URL:http://epydoc.sourceforge.net/api/epydoc.markup.restructuredtext-module.html#CONSOLIDATED_DEFLIST_FIELDS>

Cheers,
Alan Isaac





From charlesr.harris at gmail.com  Tue Oct  2 15:44:20 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 2 Oct 2007 13:44:20 -0600
Subject: [Numpy-discussion] Default value in documentation
In-Reply-To: <Mahogany-0.67.0-216-20071002-153603.00@american.edu>
References: <200710021323.44022.pgmdevlist@gmail.com>
	<4702828C.1010507@gmail.com> <200710021410.50354.pgmdevlist@gmail.com>
	<Mahogany-0.67.0-216-20071002-153603.00@american.edu>
Message-ID: <e06186140710021244mae0e49due80d654f320404c@mail.gmail.com>

On 10/2/07, Alan G Isaac <aisaac at american.edu> wrote:
>
> On Tue, 2 Oct 2007, Pierre GM apparently wrote:
>
> > is there any kind of standard to describe the attributes
> > of a class, a la :IVariables: in epydoc ?
>
> I thought it was ...  :IVariables:
> i.e., I thought the standard was reST as handled by epydoc.
> <URL:http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines>
> <URL:
> http://epydoc.sourceforge.net/api/epydoc.markup.restructuredtext-module.html#CONSOLIDATED_DEFLIST_FIELDS
> >


We're avoiding consolidated fields because they behave badly.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071002/caac20a1/attachment.html>

From pgmdevlist at gmail.com  Tue Oct  2 15:46:24 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 2 Oct 2007 15:46:24 -0400
Subject: [Numpy-discussion] Default value in documentation
In-Reply-To: <Mahogany-0.67.0-216-20071002-153603.00@american.edu>
References: <200710021323.44022.pgmdevlist@gmail.com>
	<200710021410.50354.pgmdevlist@gmail.com>
	<Mahogany-0.67.0-216-20071002-153603.00@american.edu>
Message-ID: <200710021546.24756.pgmdevlist@gmail.com>

On Tuesday 02 October 2007 15:36:03 Alan G Isaac wrote:
> On Tue, 2 Oct 2007, Pierre GM apparently wrote:
> > is there any kind of standard to describe the attributes
> > of a class, a la :IVariables: in epydoc ?
>
> I thought it was ...  :IVariables:
> i.e., I thought the standard was reST as handled by epydoc.
> <URL:http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines>
> <URL:http://epydoc.sourceforge.net/api/epydoc.markup.restructuredtext-modul
>e.html#CONSOLIDATED_DEFLIST_FIELDS>

Mmh... That makes sense on one side, but it doesn't seem very consistent with 
the rest of the Numpy/Scipy standards... I'll stick to :IVariables: till a 
consensus is reached, then.
Thanks again


From charlesr.harris at gmail.com  Tue Oct  2 17:22:53 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 2 Oct 2007 15:22:53 -0600
Subject: [Numpy-discussion] Default value in documentation
In-Reply-To: <200710021546.24756.pgmdevlist@gmail.com>
References: <200710021323.44022.pgmdevlist@gmail.com>
	<200710021410.50354.pgmdevlist@gmail.com>
	<Mahogany-0.67.0-216-20071002-153603.00@american.edu>
	<200710021546.24756.pgmdevlist@gmail.com>
Message-ID: <e06186140710021422l5950a8ahba73e68f9d97110c@mail.gmail.com>

Hi Pierre,

On 10/2/07, Pierre GM <pgmdevlist at gmail.com> wrote:
>
> On Tuesday 02 October 2007 15:36:03 Alan G Isaac wrote:
> > On Tue, 2 Oct 2007, Pierre GM apparently wrote:
> > > is there any kind of standard to describe the attributes
> > > of a class, a la :IVariables: in epydoc ?
> >
> > I thought it was ...  :IVariables:
> > i.e., I thought the standard was reST as handled by epydoc.
> > <URL:http://projects.scipy.org/scipy/numpy/wiki/CodingStyleGuidelines>
> > <URL:
> http://epydoc.sourceforge.net/api/epydoc.markup.restructuredtext-modul
> >e.html#CONSOLIDATED_DEFLIST_FIELDS>
>
> Mmh... That makes sense on one side, but it doesn't seem very consistent
> with
> the rest of the Numpy/Scipy standards... I'll stick to :IVariables: till a
> consensus is reached, then.
> Thanks again
> ______


Umm, no, we don't use them in the standards anymore. You should look at the
markup and run some examples. The main problem with the consolidated fields
is that they are all put together as item lists in a definition list and
moved to the end of the docstring when it is rendered. There is also little
control through the .css style sheet, they are all bulleted, and some of the
types are missing. This has been discussed on the list but few have shown
much interest in the topic. The rendering of the current markup can be seen
at http://www.scipy.org/doc/example/

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071002/5e8840ef/attachment.html>

From perry at stsci.edu  Wed Oct  3 11:12:07 2007
From: perry at stsci.edu (Perry Greenfield)
Date: Wed, 3 Oct 2007 11:12:07 -0400
Subject: [Numpy-discussion] indexing bug?
In-Reply-To: <20070928202342.GB32704@mentat.za.net>
References: <235015.97371.qm@web34405.mail.mud.yahoo.com>
	<46FD50F2.6050609@stsci.edu> <20070928202342.GB32704@mentat.za.net>
Message-ID: <23421265-8907-46D3-8B9D-E2C85F4188FE@stsci.edu>


On Sep 28, 2007, at 4:23 PM, Stefan van der Walt wrote:

> On Fri, Sep 28, 2007 at 03:07:30PM -0400, Nadia Dencheva wrote:
>> This should return an error and not silently truncate to int.
>
> Why do you say that?  The current behaviour is consistent and well
> defined:
>
> a[x] == a[int(x)]
>
I disagree. It is neither consistent nor well defined.

It is not consistent with Python list indexing behavior.

It is not consistent with numpy behavior:

 >>> x = arange(10)
 >>> x[array(2.99)]

raises an exception

It didn't work in Numeric
It didn't work in numarray

> We certainly can't change it now (just imagine all the code out there
> that will break); but I personally don't think it is a big problem.
>
I disagree. If people are willing to change the Class API of numpy to  
be consistent with Python, they certainly should be willing to change  
this. This behavior is new with numpy, so there should not be a lot  
of code that depends on it (and shame on those that do :-).

> On a somewhat related note, you may also be interested in the PEP at
>
> http://docs.python.org/whatsnew/pep-357.html
>
I believe that this PEP is irrelevant. It's purpose is to allow numpy  
rank-0 arrays (what numpy returns on indexing single values in  
arrays) to be used as an index in Python lists. It does not suggest  
that float values should be used as indices for sequences. It does  
provide a mechanism so that numpy rank-0 float arrays could be used  
as indices to lists, but doesn't mean that they should. And  
currently, they don't work as indices to lists (try it).

So I can't see any reason not to treat the current behavior as  
undesirable. It is not long standing (it wasn't available in numpy,  
and only in odd ways in numarray--and it was a bug there [I feel I  
can say that without contradiction :-)]. Now that the compatibility  
can of worms is open, fixing this is a small thing compared to the  
other issue being raised.

Perry Greenfield


From charlesr.harris at gmail.com  Wed Oct  3 11:26:57 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 3 Oct 2007 09:26:57 -0600
Subject: [Numpy-discussion] indexing bug?
In-Reply-To: <23421265-8907-46D3-8B9D-E2C85F4188FE@stsci.edu>
References: <235015.97371.qm@web34405.mail.mud.yahoo.com>
	<46FD50F2.6050609@stsci.edu> <20070928202342.GB32704@mentat.za.net>
	<23421265-8907-46D3-8B9D-E2C85F4188FE@stsci.edu>
Message-ID: <e06186140710030826t2bfce322x8a75fd64fa0f39f4@mail.gmail.com>

On 10/3/07, Perry Greenfield <perry at stsci.edu> wrote:
>
>
> On Sep 28, 2007, at 4:23 PM, Stefan van der Walt wrote:
>
> > On Fri, Sep 28, 2007 at 03:07:30PM -0400, Nadia Dencheva wrote:
> >> This should return an error and not silently truncate to int.
> >
> > Why do you say that?  The current behaviour is consistent and well
> > defined:
> >
> > a[x] == a[int(x)]
> >
> I disagree. It is neither consistent nor well defined.
>
> It is not consistent with Python list indexing behavior.
>
> It is not consistent with numpy behavior:
>
> >>> x = arange(10)
> >>> x[array(2.99)]
>
> raises an exception
>
> It didn't work in Numeric
> It didn't work in numarray
>
> > We certainly can't change it now (just imagine all the code out there
> > that will break); but I personally don't think it is a big problem.
> >
> I disagree. If people are willing to change the Class API of numpy to
> be consistent with Python, they certainly should be willing to change
> this. This behavior is new with numpy, so there should not be a lot
> of code that depends on it (and shame on those that do :-).
>
> > On a somewhat related note, you may also be interested in the PEP at
> >
> > http://docs.python.org/whatsnew/pep-357.html
> >
> I believe that this PEP is irrelevant. It's purpose is to allow numpy
> rank-0 arrays (what numpy returns on indexing single values in
> arrays) to be used as an index in Python lists. It does not suggest
> that float values should be used as indices for sequences. It does
> provide a mechanism so that numpy rank-0 float arrays could be used
> as indices to lists, but doesn't mean that they should. And
> currently, they don't work as indices to lists (try it).
>
> So I can't see any reason not to treat the current behavior as
> undesirable. It is not long standing (it wasn't available in numpy,
> and only in odd ways in numarray--and it was a bug there [I feel I
> can say that without contradiction :-)]. Now that the compatibility
> can of worms is open, fixing this is a small thing compared to the
> other issue being raised.


I agree that this should be changed, and the sooner the better. Look at how
many problems arange with float arguments causes, and the often expressed
wish that it had never allowed floats in the first place. I suspect using
floats as indexes is almost always a bug, and where it isn't the code would
be much clearer if the conversion to integer was made explicit.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071003/cfde2b6a/attachment.html>

From perry at stsci.edu  Wed Oct  3 11:43:04 2007
From: perry at stsci.edu (Perry Greenfield)
Date: Wed, 3 Oct 2007 11:43:04 -0400
Subject: [Numpy-discussion] [SciPy-dev] adopting Python Style Guide for
	classes
In-Reply-To: <c7009a550710021024y3a0eb3d6y1e9a64a87c71e3d@mail.gmail.com>
References: <c7009a550710021024y3a0eb3d6y1e9a64a87c71e3d@mail.gmail.com>
Message-ID: <92C0EE9C-0998-4A82-9A63-A7A3AE92D5C9@stsci.edu>

To follow on to my previous posting on this topic given Robert's  
response.

As I said previously, I was never strongly committed to one approach  
or the other. But since the v1 release has been made, I think more  
care needs to be given to consideration of proposals like this before  
actually charging off to make the changes.

1) Even though Robert is speaking for Travis, I think given Travis's  
role in numpy, it is important for Travis to speak directly to this  
when he gets the chance.

2) API changes should only be made in major releases, not minor  
releases (IMHO).

3) Greater time should be provided to accommodate the transition. For  
example, there should not be deprecation warnings in the first  
version that this API appears in. The first release of this should  
not lead to nuisance messages for those that have other software that  
depends on this. (A tool that allows conditional messages would be  
good, but the default should be no message). The next release, sure.  
As a result, it means that the old API can't be removed until at  
least two releases after that.

4) More information should be provided as to what actually will  
change in the public interface. I suspect that it isn't obvious to  
many what will change. From the mailing list discussions there  
doesn't even seem to be consensus on the factory functions or type  
objects (more on these later). Many of the remaining objects are  
probably used internally (essentially private) and will affect few  
outside of the numpy developers. Since users typically deal mostly  
with factory functions, and other functions, they may not deal with  
classes much (outside of types). So listing the public classes so  
affected will help people understand what changes typical users will  
see, and what changes advanced users will see. While this is  
annoying, I think someone needs to write up an explicit list of those  
public classes that will be changed (and those that won't) so we all  
know what we will face. It may be a very small list and thus  
alleviate concern about the process. It may show some surprises that  
people hadn't thought about. Not doing this before making the changes  
seems very unwise.

5) In my humble opinion, we would be nuts--absolutely nuts--to change  
either the type classes or the factory functions. This would be  
foolish consistency at it's worst. We *just* went through the  
exercise of changing Int32 to int32 and so forth and we would have to  
change back again? This cannot be seriously considered.

Perry Greenfield

On Oct 2, 2007, at 1:24 PM, Jarrod Millman wrote:

> Hello,
>
> For those of you not on the Numpy developer's list, we have been
> talking about adopting the Python class naming convention to conform
> with Guido's style guide as closely as possible:
> http://www.python.org/dev/peps/pep-0008/
> According to Guido,  class names should use the CapWords convention.
> Most Python projects (eg, ETS, matploltlib) adhere to the Python
> naming conventions and it is confusing that NumPy and SciPy don't.
>
> Currently, both NumPy and SciPy use either lower_underscore_separated
> or CapWords for class names.
>
> NumPy
> ======
> $ grep -r '^class [A-Z]' --include "*.py" * | wc -l
> 1014
> $ grep -r  '^class' --include "*.py" * | grep -v 'class [A-Z]' | wc -l
> 207
>
> SciPy
> =====
> $ grep -r '^class [A-Z]' --include "*.py" * | wc -l
> 587
> $ grep -r  '^class' --include "*.py" * | grep -v 'class [A-Z]' | wc -l
> 565
>
> So far, there is Universal support for the basic idea of conforming to
> the Python Style Guide.  Since it should effect the user API, we have
> already updated the TestCase classes.  First, we checked in a change
> to allow TestCase classes to be prefixed with
> either 'test' or 'Test':
> http://projects.scipy.org/scipy/numpy/changeset/4144
> Then, we updated both SciPy and NumPy to use CapWords for TestCase  
> classes:
> http://projects.scipy.org/scipy/numpy/changeset/4151
> http://projects.scipy.org/scipy/scipy/changeset/3388
> In order to run the SciPy tests on the trunk you will need to install
> NumPy from svn revision 4144 or greater.  Before SciPy 0.7.0 is
> released, we will release NumPy 1.0.4.
>
> Also, if you are adding a new class to NumPy or SciPy, please use
> CapWords.  Now we need to decide what to do about the remaining
> lower_underscore_separated class names.  Obviously, it is important
> that we are careful to not break a lot of code just to bring our class
> names up to standards.
>
> Cheers,
>
> -- 
> Jarrod Millman
> Computational Infrastructure for Research Labs
> 10 Giannini Hall, UC Berkeley
> phone: 510.643.4014
> http://cirl.berkeley.edu/
> _______________________________________________
> Scipy-dev mailing list
> Scipy-dev at scipy.org
> http://projects.scipy.org/mailman/listinfo/scipy-dev



From aisaac at american.edu  Wed Oct  3 12:03:33 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 3 Oct 2007 12:03:33 -0400
Subject: [Numpy-discussion] boolean matrix bug?
Message-ID: <Mahogany-0.67.0-1744-20071003-120333.00@american.edu>

Shouldn't x**0 be boolean for a boolean matrix?

Cheers,
Alan Isaac

>>> import numpy
>>> numpy.__version__
'1.0.3.1'
>>> x = numpy.mat('1 1;1 0',dtype='bool')
>>> x**0
matrix([[ 1.,  0.],
        [ 0.,  1.]])
>>> x**1
matrix([[ True,  True],
        [ True, False]], dtype=bool)
>>> x**2
matrix([[ True,  True],
        [ True,  True]], dtype=bool)
>>>





From stefan at sun.ac.za  Wed Oct  3 13:35:46 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Wed, 3 Oct 2007 19:35:46 +0200
Subject: [Numpy-discussion] indexing bug?
In-Reply-To: <23421265-8907-46D3-8B9D-E2C85F4188FE@stsci.edu>
References: <235015.97371.qm@web34405.mail.mud.yahoo.com>
	<46FD50F2.6050609@stsci.edu> <20070928202342.GB32704@mentat.za.net>
	<23421265-8907-46D3-8B9D-E2C85F4188FE@stsci.edu>
Message-ID: <20071003173546.GI7241@mentat.za.net>

On Wed, Oct 03, 2007 at 11:12:07AM -0400, Perry Greenfield wrote:
>
> On Sep 28, 2007, at 4:23 PM, Stefan van der Walt wrote:
>
>> On Fri, Sep 28, 2007 at 03:07:30PM -0400, Nadia Dencheva wrote:
>>> This should return an error and not silently truncate to int.
>>
>> Why do you say that?  The current behaviour is consistent and well
>> defined:
>>
>> a[x] == a[int(x)]
>>
> I disagree. It is neither consistent nor well defined.

It works for other objects too:

class MyIndex(object):
    def __int__(self):
        return 1

m = M()

x = N.array([1,2,3]

x[m] == x[1]

> It is not consistent with Python list indexing behavior.

Neither are most other forms of ndarray indexing:

x = [1,2,3]
x[[1,2]]

yields an error.

> It is not consistent with numpy behavior:
>
> >>> x = arange(10)
> >>> x[array(2.99)]
>
> raises an exception

That seems to be an exception to the rule.  I agree that both

x[2.99] and x[array(2.99)]

should behave the same way.

>> We certainly can't change it now (just imagine all the code out there
>> that will break); but I personally don't think it is a big problem.
>>
> I disagree. If people are willing to change the Class API of numpy to be 
> consistent with Python, they certainly should be willing to change this. 
> This behavior is new with numpy, so there should not be a lot of code that 
> depends on it (and shame on those that do :-).

Let me rephrase: we cannot change the API until 1.1, unless this is
seen as a bug.  To which other API changes are you referring?  The
style changes is a different matter entirely.

My point was that the current behaviour is easy to predict, but I am
not especially partial on the matter.

Regards
St?fan


From perry at stsci.edu  Wed Oct  3 13:50:01 2007
From: perry at stsci.edu (Perry Greenfield)
Date: Wed, 3 Oct 2007 13:50:01 -0400
Subject: [Numpy-discussion] indexing bug?
In-Reply-To: <20071003173546.GI7241@mentat.za.net>
References: <235015.97371.qm@web34405.mail.mud.yahoo.com>
	<46FD50F2.6050609@stsci.edu> <20070928202342.GB32704@mentat.za.net>
	<23421265-8907-46D3-8B9D-E2C85F4188FE@stsci.edu>
	<20071003173546.GI7241@mentat.za.net>
Message-ID: <3F6D275A-B490-417E-B1B4-ABD716164753@stsci.edu>


On Oct 3, 2007, at 1:35 PM, Stefan van der Walt wrote:

>
>>> We certainly can't change it now (just imagine all the code out  
>>> there
>>> that will break); but I personally don't think it is a big problem.
>>>
>> I disagree. If people are willing to change the Class API of numpy  
>> to be
>> consistent with Python, they certainly should be willing to change  
>> this.
>> This behavior is new with numpy, so there should not be a lot of  
>> code that
>> depends on it (and shame on those that do :-).
>
> Let me rephrase: we cannot change the API until 1.1, unless this is
> seen as a bug.  To which other API changes are you referring?  The
> style changes is a different matter entirely.

The recent numpy and scipy threads on "adopting Python Style Guide  
for classes". If this is what you mean by style changes and it ends  
up changing array to Array and int32 to Int32, it's not just a style  
change, it will break lots of code ("Since it should effect the user  
API..."). These are API changes even if you consider case of the  
class name just style.

Perry Greenfield


From Chris.Barker at noaa.gov  Wed Oct  3 13:52:58 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 03 Oct 2007 10:52:58 -0700
Subject: [Numpy-discussion] indexing bug?
In-Reply-To: <20071003173546.GI7241@mentat.za.net>
References: <235015.97371.qm@web34405.mail.mud.yahoo.com>
	<46FD50F2.6050609@stsci.edu> <20070928202342.GB32704@mentat.za.net>
	<23421265-8907-46D3-8B9D-E2C85F4188FE@stsci.edu>
	<20071003173546.GI7241@mentat.za.net>
Message-ID: <4703D6FA.6020402@noaa.gov>

Stefan van der Walt wrote:
>> The current behavior is consistent and well
>>> defined:
>>> a[x] == a[int(x)]

This is all possible because of PEP 357:

http://www.python.org/dev/peps/pep-0357/

However, when I read this from the PEP:

"""
It is not possible to use the nb_int (and __int__ special method)
for this purpose because that method is used to *coerce* objects
to integers.  It would be inappropriate to allow every object that
can be coerced to an integer to be used as an integer everywhere
Python expects a true integer.  For example, if __int__ were used
to convert an object to an integer in slicing, then float objects
would be allowed in slicing and x[3.2:5.8] would not raise an error
as it should.
"""

It seems pretty clear that only integer types were intended to by used 
as indexes. Does that make this a bug? I'll defer that to others more in 
the know than I.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From tim.hochberg at ieee.org  Wed Oct  3 14:01:42 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 3 Oct 2007 11:01:42 -0700
Subject: [Numpy-discussion] indexing bug?
In-Reply-To: <4703D6FA.6020402@noaa.gov>
References: <235015.97371.qm@web34405.mail.mud.yahoo.com>
	<46FD50F2.6050609@stsci.edu> <20070928202342.GB32704@mentat.za.net>
	<23421265-8907-46D3-8B9D-E2C85F4188FE@stsci.edu>
	<20071003173546.GI7241@mentat.za.net> <4703D6FA.6020402@noaa.gov>
Message-ID: <e4412d6b0710031101n33f3e30oe582ca07974f592d@mail.gmail.com>

On 10/3/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
>
> Stefan van der Walt wrote:
> >> The current behavior is consistent and well
> >>> defined:
> >>> a[x] == a[int(x)]
>
> This is all possible because of PEP 357:


I think that the current behavior has always been possible; arbitrary
objects can be passed as indices and coercing to int was always possible.

http://www.python.org/dev/peps/pep-0357/
>
> However, when I read this from the PEP:
>
> """
> It is not possible to use the nb_int (and __int__ special method)
> for this purpose because that method is used to *coerce* objects
> to integers.  It would be inappropriate to allow every object that
> can be coerced to an integer to be used as an integer everywhere
> Python expects a true integer.  For example, if __int__ were used
> to convert an object to an integer in slicing, then float objects
> would be allowed in slicing and x[3.2:5.8] would not raise an error
> as it should.
> """
>
> It seems pretty clear that only integer types were intended to by used
> as indexes. Does that make this a bug? I'll defer that to others more in
> the know than I.


As I understand it, the whole point of PEP-357 was to allow the coercion of
int-like things (numpy.int32 say, or your own private integerish class) to
be used as indices without also allowing things that aren't integers, but
that can be coerced to integers (floats for instance) to slip through. So
yes, I'd say this is a bug. Probably someplace that should be using
__index__ or the C equivalent is still using __int__, but I haven't had time
to look.

On the other hand, this may well be a bug that people rely on and it's not
doing drastic immediate harm. So, while I think it should get fixed
eventually, it's probably fine to wait till 1.1 or whenever the next
convenient time is.

-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071003/ecd8919d/attachment.html>

From robert.kern at gmail.com  Wed Oct  3 14:18:01 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 03 Oct 2007 13:18:01 -0500
Subject: [Numpy-discussion] indexing bug?
In-Reply-To: <e4412d6b0710031101n33f3e30oe582ca07974f592d@mail.gmail.com>
References: <235015.97371.qm@web34405.mail.mud.yahoo.com>	<46FD50F2.6050609@stsci.edu>
	<20070928202342.GB32704@mentat.za.net>	<23421265-8907-46D3-8B9D-E2C85F4188FE@stsci.edu>	<20071003173546.GI7241@mentat.za.net>
	<4703D6FA.6020402@noaa.gov>
	<e4412d6b0710031101n33f3e30oe582ca07974f592d@mail.gmail.com>
Message-ID: <4703DCD9.6080805@gmail.com>

Timothy Hochberg wrote:

> As I understand it, the whole point of PEP-357 was to allow the coercion
> of int-like things (numpy.int32 say, or your own private integerish
> class) to be used as indices without also allowing things that aren't
> integers, but that can be coerced to integers (floats for instance) to
> slip through. So yes, I'd say this is a bug. Probably someplace that
> should be using __index__ or the C equivalent is still using __int__,
> but I haven't had time to look.

I believe the problem is in numpy/core/src/arrayobject.c:

 2635 static PyObject *
 2636 array_subscript_simple(PyArrayObject *self, PyObject *op)
 2637 {
 2638     intp dimensions[MAX_DIMS], strides[MAX_DIMS];
 2639     intp offset;
 2640     int nd;
 2641     PyArrayObject *other;
 2642     intp value;
 2643
 2644     value = PyArray_PyIntAsIntp(op);

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From millman at berkeley.edu  Wed Oct  3 14:26:29 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 3 Oct 2007 11:26:29 -0700
Subject: [Numpy-discussion] [SciPy-dev] adopting Python Style Guide for
	classes
In-Reply-To: <92C0EE9C-0998-4A82-9A63-A7A3AE92D5C9@stsci.edu>
References: <c7009a550710021024y3a0eb3d6y1e9a64a87c71e3d@mail.gmail.com>
	<92C0EE9C-0998-4A82-9A63-A7A3AE92D5C9@stsci.edu>
Message-ID: <c7009a550710031126j3f10870bsc76d1b40421b868a@mail.gmail.com>

On 10/3/07, Perry Greenfield <perry at stsci.edu> wrote:
> 2) API changes should only be made in major releases, not minor
> releases (IMHO).

+1

> 3) Greater time should be provided to accommodate the transition. For
> example, there should not be deprecation warnings in the first
> version that this API appears in. The first release of this should
> not lead to nuisance messages for those that have other software that
> depends on this. (A tool that allows conditional messages would be
> good, but the default should be no message). The next release, sure.
> As a result, it means that the old API can't be removed until at
> least two releases after that.

I am not sure I agree with this.  For example, I think it would be
acceptable for NumPy 1.1.0 to have deprecation warning about changed
APIs.  Perhaps you were saying that NumPy 1.0.4 could use the new
class names in addition to the old names without complaint.  That
sounds reasonable to me.  Then when NumPy 1.1.0 comes out the old
style names would raise deprecation warnings.

> 4) More information should be provided as to what actually will
> change in the public interface. I suspect that it isn't obvious to
> many what will change. From the mailing list discussions there
> doesn't even seem to be consensus on the factory functions or type
> objects (more on these later). Many of the remaining objects are
> probably used internally (essentially private) and will affect few
> outside of the numpy developers. Since users typically deal mostly
> with factory functions, and other functions, they may not deal with
> classes much (outside of types). So listing the public classes so
> affected will help people understand what changes typical users will
> see, and what changes advanced users will see. While this is
> annoying, I think someone needs to write up an explicit list of those
> public classes that will be changed (and those that won't) so we all
> know what we will face. It may be a very small list and thus
> alleviate concern about the process. It may show some surprises that
> people hadn't thought about. Not doing this before making the changes
> seems very unwise.

As long as we agree in principle how we want classes named, I think
there is little need to rush to change existing class names.

> 5) In my humble opinion, we would be nuts--absolutely nuts--to change
> either the type classes or the factory functions. This would be
> foolish consistency at it's worst. We *just* went through the
> exercise of changing Int32 to int32 and so forth and we would have to
> change back again? This cannot be seriously considered.

I think that the general consensus is that we should keep int32,
rather than switch to Int32.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Wed Oct  3 14:35:49 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Wed, 3 Oct 2007 11:35:49 -0700
Subject: [Numpy-discussion] indexing bug?
In-Reply-To: <e4412d6b0710031101n33f3e30oe582ca07974f592d@mail.gmail.com>
References: <235015.97371.qm@web34405.mail.mud.yahoo.com>
	<46FD50F2.6050609@stsci.edu> <20070928202342.GB32704@mentat.za.net>
	<23421265-8907-46D3-8B9D-E2C85F4188FE@stsci.edu>
	<20071003173546.GI7241@mentat.za.net> <4703D6FA.6020402@noaa.gov>
	<e4412d6b0710031101n33f3e30oe582ca07974f592d@mail.gmail.com>
Message-ID: <c7009a550710031135v6e9df8d8n42a021fd48cef7b6@mail.gmail.com>

On 10/3/07, Timothy Hochberg <tim.hochberg at ieee.org> wrote:
> As I understand it, the whole point of PEP-357 was to allow the coercion of
> int-like things (numpy.int32 say, or your own private integerish class) to
> be used as indices without also allowing things that aren't integers, but
> that can be coerced to integers (floats for instance) to slip through. So
> yes, I'd say this is a bug. Probably someplace that should be using
> __index__ or the C equivalent is still using __int__, but I haven't had time
> to look.

It seems like a bug to me as well.

> On the other hand, this may well be a bug that people rely on and it's not
> doing drastic immediate harm. So, while I think it should get fixed
> eventually, it's probably fine to wait till 1.1 or whenever the next
> convenient time is.

I don't know that it is essential to fix in the 1.0.x series, but 1.1
is still several months away.  And the longer this "bug" is out there,
the greater the chance that people will write code that depends on it.
 So I would tentatively vote to fix it sooner rather than later.

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From stefan at sun.ac.za  Wed Oct  3 15:04:10 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Wed, 3 Oct 2007 21:04:10 +0200
Subject: [Numpy-discussion] indexing bug?
In-Reply-To: <3F6D275A-B490-417E-B1B4-ABD716164753@stsci.edu>
References: <235015.97371.qm@web34405.mail.mud.yahoo.com>
	<46FD50F2.6050609@stsci.edu> <20070928202342.GB32704@mentat.za.net>
	<23421265-8907-46D3-8B9D-E2C85F4188FE@stsci.edu>
	<20071003173546.GI7241@mentat.za.net>
	<3F6D275A-B490-417E-B1B4-ABD716164753@stsci.edu>
Message-ID: <20071003190410.GK7241@mentat.za.net>

On Wed, Oct 03, 2007 at 01:50:01PM -0400, Perry Greenfield wrote:
> > Let me rephrase: we cannot change the API until 1.1, unless this is
> > seen as a bug.  To which other API changes are you referring?  The
> > style changes is a different matter entirely.
> 
> The recent numpy and scipy threads on "adopting Python Style Guide  
> for classes". If this is what you mean by style changes and it ends  
> up changing array to Array and int32 to Int32, it's not just a style  
> change, it will break lots of code ("Since it should effect the user  
> API..."). These are API changes even if you consider case of the  
> class name just style.

Don't worry, nothing is being rushed through.  Only internal
representations are currently being changed (e.g. unit tests).  No
changes to the API will be made until 1.1, and then only after the
discussion is over.

I think your quote above refers to the e-mail Jarrod Millman wrote,
but that was simply a typo, and should have read "since it shouldn't
affect the user API".

Regards
St?fan


From perry at stsci.edu  Wed Oct  3 15:08:01 2007
From: perry at stsci.edu (Perry Greenfield)
Date: Wed, 3 Oct 2007 15:08:01 -0400
Subject: [Numpy-discussion] [SciPy-dev] adopting Python Style Guide for
	classes
In-Reply-To: <c7009a550710031126j3f10870bsc76d1b40421b868a@mail.gmail.com>
References: <c7009a550710021024y3a0eb3d6y1e9a64a87c71e3d@mail.gmail.com>
	<92C0EE9C-0998-4A82-9A63-A7A3AE92D5C9@stsci.edu>
	<c7009a550710031126j3f10870bsc76d1b40421b868a@mail.gmail.com>
Message-ID: <41A7993F-ED30-45CD-987B-9C488452A59B@stsci.edu>


On Oct 3, 2007, at 2:26 PM, Jarrod Millman wrote:

>
>> 3) Greater time should be provided to accommodate the transition. For
>> example, there should not be deprecation warnings in the first
>> version that this API appears in. The first release of this should
>> not lead to nuisance messages for those that have other software that
>> depends on this. (A tool that allows conditional messages would be
>> good, but the default should be no message). The next release, sure.
>> As a result, it means that the old API can't be removed until at
>> least two releases after that.
>
> I am not sure I agree with this.  For example, I think it would be
> acceptable for NumPy 1.1.0 to have deprecation warning about changed
> APIs.  Perhaps you were saying that NumPy 1.0.4 could use the new
> class names in addition to the old names without complaint.  That
> sounds reasonable to me.  Then when NumPy 1.1.0 comes out the old
> style names would raise deprecation warnings.
>
The situation I'm trying to avoid is a too tight coupling between  
numpy changes and client applications that use numpy. Suppose we  
distribute an application that uses numpy. We could make the changes  
to our application before the api-change release comes out (from svn)  
and then when we release our new version (very soon after the api- 
changed numpy comes out) we effectively force all of our users to  
update immediately. The problem is that they may not want to update  
on our schedule. They become annoyed at us.

So we take the other tack, we wait for a while before changing our  
code to require the new numpy. This give the user community time to  
switch their stuff too. But, now our code generates annoying  
deprecation warnings that are useless to the people we distribute  
applications to if they update to the new numpy before we do. Here's  
where I display some ignorance. If the warnings use the standard lib  
warning module, I'm guessing that we can add warnings filters to  
suppress any warnings that arise from our code (but not having much  
experience with it, it isn't clear to me whether the filter  
suppresses all warnings arising from numpy or whether one can  
suppress only those associated with ones that are from the  
applications use). But it's good to clarify this point. If they are  
present by default, an application needs to be able to suppress them.


>> 5) In my humble opinion, we would be nuts--absolutely nuts--to change
>> either the type classes or the factory functions. This would be
>> foolish consistency at it's worst. We *just* went through the
>> exercise of changing Int32 to int32 and so forth and we would have to
>> change back again? This cannot be seriously considered.
>
> I think that the general consensus is that we should keep int32,
> rather than switch to Int32.

That's good. But what about array(), zeros(), ones(), arange(), etc.?

Perry



From haase at msg.ucsf.edu  Wed Oct  3 15:34:10 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Wed, 3 Oct 2007 21:34:10 +0200
Subject: [Numpy-discussion] [SciPy-dev] adopting Python Style Guide for
	classes
In-Reply-To: <41A7993F-ED30-45CD-987B-9C488452A59B@stsci.edu>
References: <c7009a550710021024y3a0eb3d6y1e9a64a87c71e3d@mail.gmail.com>
	<92C0EE9C-0998-4A82-9A63-A7A3AE92D5C9@stsci.edu>
	<c7009a550710031126j3f10870bsc76d1b40421b868a@mail.gmail.com>
	<41A7993F-ED30-45CD-987B-9C488452A59B@stsci.edu>
Message-ID: <bc657ead0710031234r646df945q5260d12731da201@mail.gmail.com>

On 10/3/07, Perry Greenfield <perry at stsci.edu> wrote:
>
> On Oct 3, 2007, at 2:26 PM, Jarrod Millman wrote:
>
> >
> >> 3) Greater time should be provided to accommodate the transition. For
> >> example, there should not be deprecation warnings in the first
> >> version that this API appears in. The first release of this should
> >> not lead to nuisance messages for those that have other software that
> >> depends on this. (A tool that allows conditional messages would be
> >> good, but the default should be no message). The next release, sure.
> >> As a result, it means that the old API can't be removed until at
> >> least two releases after that.
> >
> > I am not sure I agree with this.  For example, I think it would be
> > acceptable for NumPy 1.1.0 to have deprecation warning about changed
> > APIs.  Perhaps you were saying that NumPy 1.0.4 could use the new
> > class names in addition to the old names without complaint.  That
> > sounds reasonable to me.  Then when NumPy 1.1.0 comes out the old
> > style names would raise deprecation warnings.
> >
> The situation I'm trying to avoid is a too tight coupling between
> numpy changes and client applications that use numpy. Suppose we
> distribute an application that uses numpy. We could make the changes
> to our application before the api-change release comes out (from svn)
> and then when we release our new version (very soon after the api-
> changed numpy comes out) we effectively force all of our users to
> update immediately. The problem is that they may not want to update
> on our schedule. They become annoyed at us.
>
> So we take the other tack, we wait for a while before changing our
> code to require the new numpy. This give the user community time to
> switch their stuff too. But, now our code generates annoying
> deprecation warnings that are useless to the people we distribute
> applications to if they update to the new numpy before we do. Here's
> where I display some ignorance. If the warnings use the standard lib
> warning module, I'm guessing that we can add warnings filters to
> suppress any warnings that arise from our code (but not having much
> experience with it, it isn't clear to me whether the filter
> suppresses all warnings arising from numpy or whether one can
> suppress only those associated with ones that are from the
> applications use). But it's good to clarify this point. If they are
> present by default, an application needs to be able to suppress them.
>
>
> >> 5) In my humble opinion, we would be nuts--absolutely nuts--to change
> >> either the type classes or the factory functions. This would be
> >> foolish consistency at it's worst. We *just* went through the
> >> exercise of changing Int32 to int32 and so forth and we would have to
> >> change back again? This cannot be seriously considered.
> >
> > I think that the general consensus is that we should keep int32,
> > rather than switch to Int32.
>
> That's good. But what about array(), zeros(), ones(), arange(), etc.?
>
Just to express my worries:  I hope that numpy stays "easily typable"
as a numerical and interactive (!)  environment.
What I'm trying to say:
IMHO,  "arange" should stay lower case, like "range", even if it might
be (internally) implemented as a class.
Similarly, many functions/objects/classes have a functional "feeling"
to them, even tough they be implemented as classes.

My 2 cents.
-Sebastian Haase


From aisaac at american.edu  Wed Oct  3 15:54:16 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 3 Oct 2007 15:54:16 -0400
Subject: [Numpy-discussion] [SciPy-dev] adopting Python Style Guide for
	classes
In-Reply-To: <41A7993F-ED30-45CD-987B-9C488452A59B@stsci.edu>
References: <c7009a550710021024y3a0eb3d6y1e9a64a87c71e3d@mail.gmail.com><92C0EE9C-0998-4A82-9A63-A7A3AE92D5C9@stsci.edu><c7009a550710031126j3f10870bsc76d1b40421b868a@mail.gmail.com><41A7993F-ED30-45CD-987B-9C488452A59B@stsci.edu>
Message-ID: <Mahogany-0.67.0-1744-20071003-155416.00@american.edu>

To help me understand, might someone offer some examples of
NumPy names that really should be changed?

Thank you,
Alan Isaac





From steve at shrogers.com  Wed Oct  3 23:09:55 2007
From: steve at shrogers.com (Steven H. Rogers)
Date: Wed, 03 Oct 2007 21:09:55 -0600
Subject: [Numpy-discussion] APL2007 Update
Message-ID: <47045983.3000302@shrogers.com>

APL 2007 in Montreal (only 2 1/2 weeks away, Oct 20-22).

Summary program  information is now available
on the APL2007 home page

        http://www.sigapl.org/apl2007.html

with a link to the comprehensive program description at

        http://www.sigapl.org/apl2007-program.html#a2

Registration for APL2007 is at

        http://www.regmaster.com/conf/oopsla2007.html



From drfredkin at ucsd.edu  Wed Oct  3 21:48:18 2007
From: drfredkin at ucsd.edu (Donald Fredkin)
Date: Thu, 4 Oct 2007 01:48:18 +0000 (UTC)
Subject: [Numpy-discussion] Possible bug in numpy.trace()
Message-ID: <fe1gp1$h9a$1@sea.gmane.org>

It appears that something peculiar is going on with the numpy function
trace(). The docs say that trace(A,...) is identical to A.trace(...).
Here is a test:

A = arange(8).reshape((2,2,2))
A.trace(0,1,2)
#[Out]# array([ 3, 11])
# which is correct
trace(A,0,1,2)
#[Out]# array([6, 8])
#which is wrong
#Since trace() is computed by summing the result of diagonal()
A.diagonal(0,1,2)
#[Out]# array([[0, 3],
#[Out]#        [4, 7]])
#which is correct
diagonal(A,0,1,2)
#[Out]# array([[0, 3],
#[Out]#        [4, 7]])
#which is the same correct result
A.trace(0,0,1)
#[Out]# array([6, 8])
#this is the erroneous answer we found earlier for trace(A,0,1,2)

-- 
Donald R. Fredkin
drfredkin at ucsd.edu



From stefan at sun.ac.za  Thu Oct  4 07:42:22 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 4 Oct 2007 13:42:22 +0200
Subject: [Numpy-discussion] Possible bug in numpy.trace()
In-Reply-To: <fe1gp1$h9a$1@sea.gmane.org>
References: <fe1gp1$h9a$1@sea.gmane.org>
Message-ID: <20071004114221.GA27995@mentat.za.net>

Hi Donald

On Thu, Oct 04, 2007 at 01:48:18AM +0000, Donald Fredkin wrote:
> It appears that something peculiar is going on with the numpy function
> trace(). The docs say that trace(A,...) is identical to A.trace(...).
> Here is a test:
> 
> A = arange(8).reshape((2,2,2))
> A.trace(0,1,2)
> #[Out]# array([ 3, 11])
> # which is correct
> trace(A,0,1,2)
> #[Out]# array([6, 8])
> #which is wrong
> #Since trace() is computed by summing the result of diagonal()

Thanks for the report.  This is already fixed in the latest version:

In [11]: A.trace(0,1,2)
Out[11]: array([ 3, 11])

In [12]: N.trace(A,0,1,2)
Out[12]: array([ 3, 11])

In [13]: N.__version__
Out[13]: '1.0.4.dev4149'

Regards
St?fan




From e.tadeu at gmail.com  Thu Oct  4 08:55:08 2007
From: e.tadeu at gmail.com (Edson Tadeu)
Date: Thu, 4 Oct 2007 09:55:08 -0300
Subject: [Numpy-discussion] Best representation for array of points, or,
	how to distinguish a Nx1 array of points from a Nx3 array of scalars
Message-ID: <a1b1282f0710040555g713e31d3w1052038866d8dfaa@mail.gmail.com>

I've found that there are two ways to represent an array of 3D points:

# 01:

N = 4
a = numpy.zeros((N, 3), dtype='f8')


# 02:

N = 4
b = numpy.zeros(N, dtype='3f8')


The second representation seems "conceptually" better, it is really an 1D
array of N elements ( i.e., b.shape == (4,) ), but, "in practice", it is not
very useful, because I can't directly add it to scalar, or to another array
of points, and I can't multiply it by a transformation matrix. With the
first representation, I can use expressions like this:

c = (a+1) * numpy.matrix([[1,0,0],[0,0,3],[0,2,0]]) - numpy.array([1, 5.5,
0])


So, I chose the second representation for my application; the problem is
that its shape is (N, 3), and I haven't found a way to distinguish it from a
Nx3 array of scalars, for example. This distinction is needed, because I
must convert it to other array types (from other libraries), I must make
proper I/O of the array, etc. Also, my application uses 2D and 3D arrays,
and they can be arrays of scalars, of points or of small matrices... so, for
example, a 10x7 array of 2x2 matrices of double is represented in numpy as a
10x7x2x2 array of f8 (without the proper information, it could be viewed
also as a 10x7x2 array of 2D points or a 10x7x2x2 array of doubles, and I
DON'T want this). My question is:

* Is there any field in the NumPy object where I can keep this information
(the shape of the "element"), without creeping it with the dtype='(M,N)f8'
representation I explained above?


  Thanks,
Edson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071004/3cff9091/attachment.html>

From aisaac at american.edu  Thu Oct  4 09:41:31 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 4 Oct 2007 09:41:31 -0400
Subject: [Numpy-discussion] Best representation for array of points, or,
	how to distinguish a Nx1 array of points from a Nx3 array of scalars
In-Reply-To: <a1b1282f0710040555g713e31d3w1052038866d8dfaa@mail.gmail.com>
References: <a1b1282f0710040555g713e31d3w1052038866d8dfaa@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-1576-20071004-094131.00@american.edu>

On Thu, 4 Oct 2007, Edson Tadeu apparently wrote:
> Is there any field in the NumPy object where I can keep 
> this information (the shape of the "element"), without 
> creeping it with the dtype='(M,N)f8'
> representation I explained above? 

Perhaps you want a recarray?

Cheers,
Alan Isaac





From robert.kern at gmail.com  Thu Oct  4 12:41:25 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 04 Oct 2007 11:41:25 -0500
Subject: [Numpy-discussion] Best representation for array of points, or,
 how to distinguish a Nx1 array of points from a Nx3 array of scalars
In-Reply-To: <Mahogany-0.67.0-1576-20071004-094131.00@american.edu>
References: <a1b1282f0710040555g713e31d3w1052038866d8dfaa@mail.gmail.com>
	<Mahogany-0.67.0-1576-20071004-094131.00@american.edu>
Message-ID: <470517B5.80103@gmail.com>

Alan G Isaac wrote:
> On Thu, 4 Oct 2007, Edson Tadeu apparently wrote:
>> Is there any field in the NumPy object where I can keep 
>> this information (the shape of the "element"), without 
>> creeping it with the dtype='(M,N)f8'
>> representation I explained above? 
> 
> Perhaps you want a recarray?

No, that's what the dtype='3f8' gets him. He gave reasons why he doesn't want that.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robert.kern at gmail.com  Thu Oct  4 12:47:40 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 04 Oct 2007 11:47:40 -0500
Subject: [Numpy-discussion] Best representation for array of points, or,
 how to distinguish a Nx1 array of points from a Nx3 array of scalars
In-Reply-To: <a1b1282f0710040555g713e31d3w1052038866d8dfaa@mail.gmail.com>
References: <a1b1282f0710040555g713e31d3w1052038866d8dfaa@mail.gmail.com>
Message-ID: <4705192C.50607@gmail.com>

Edson Tadeu wrote:

> * Is there any field in the NumPy object where I can keep this
> information (the shape of the "element"), without creeping it with the
> dtype='(M,N)f8' representation I explained above?

There isn't. You could make a subclass that tracks this, but you would have to
implement it carefully to maintain the information.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From cookedm at physics.mcmaster.ca  Thu Oct  4 13:08:08 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Thu, 04 Oct 2007 13:08:08 -0400
Subject: [Numpy-discussion] [SciPy-dev] adopting Python Style Guide for
	classes
In-Reply-To: <Mahogany-0.67.0-1744-20071003-155416.00@american.edu> (Alan G.
	Isaac's message of "Wed\, 3 Oct 2007 15\:54\:16 -0400")
References: <c7009a550710021024y3a0eb3d6y1e9a64a87c71e3d@mail.gmail.com>
	<92C0EE9C-0998-4A82-9A63-A7A3AE92D5C9@stsci.edu>
	<c7009a550710031126j3f10870bsc76d1b40421b868a@mail.gmail.com>
	<41A7993F-ED30-45CD-987B-9C488452A59B@stsci.edu>
	<Mahogany-0.67.0-1744-20071003-155416.00@american.edu>
Message-ID: <qnkhcl6st8n.fsf@arbutus.physics.mcmaster.ca>

Alan G Isaac <aisaac at american.edu> writes:

> To help me understand, might someone offer some examples of
> NumPy names that really should be changed?

Internal classes, like:
- nd_grid, etc. in numpy/lib/index_tricks.py
- masked_unary_operation, etc. in numpy/core/ma.py

Things we probably wouldn't change:
- array-like things, such as numpy.lib.polynomial.poly1d
- ufuncs implemented in Python like vectorize
- distributions from scipy.stats

In numpy, outside of tests, distutils, and f2py, there's actually not
that many classes defined in Python (as opposed to C). In scipy, most
non-testing classes are already CamelCase (one exception is weave).

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca


From stefan at sun.ac.za  Thu Oct  4 13:33:01 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 4 Oct 2007 19:33:01 +0200
Subject: [Numpy-discussion] Best representation for array of points, or,
	how to distinguish a Nx1 array of points from a Nx3 array of	scalars
In-Reply-To: <4705192C.50607@gmail.com>
References: <a1b1282f0710040555g713e31d3w1052038866d8dfaa@mail.gmail.com>
	<4705192C.50607@gmail.com>
Message-ID: <20071004173301.GH27995@mentat.za.net>

On Thu, Oct 04, 2007 at 11:47:40AM -0500, Robert Kern wrote:
> Edson Tadeu wrote:
> 
> > * Is there any field in the NumPy object where I can keep this
> > information (the shape of the "element"), without creeping it with the
> > dtype='(M,N)f8' representation I explained above?
> 
> There isn't. You could make a subclass that tracks this, but you would have to
> implement it carefully to maintain the information.

There is an example of how to do that here:

http://www.scipy.org/Subclasses

Regards
St?fan


From pgmdevlist at gmail.com  Thu Oct  4 13:46:01 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Thu, 4 Oct 2007 13:46:01 -0400
Subject: [Numpy-discussion] [SciPy-dev] adopting Python Style Guide for
	classes
In-Reply-To: <qnkhcl6st8n.fsf@arbutus.physics.mcmaster.ca>
References: <c7009a550710021024y3a0eb3d6y1e9a64a87c71e3d@mail.gmail.com>
	<Mahogany-0.67.0-1744-20071003-155416.00@american.edu>
	<qnkhcl6st8n.fsf@arbutus.physics.mcmaster.ca>
Message-ID: <200710041346.03134.pgmdevlist@gmail.com>

On Thursday 04 October 2007 13:08:08 David M. Cooke wrote:
> Alan G Isaac <aisaac at american.edu> writes:
> > To help me understand, might someone offer some examples of
> > NumPy names that really should be changed?
>
> Internal classes, like:
> - masked_unary_operation, etc. in numpy/core/ma.py

FYI, the new style has been followed in maskedarray (and a couple of recent 
bugs fixed in the SVN).


From e.tadeu at gmail.com  Thu Oct  4 15:14:00 2007
From: e.tadeu at gmail.com (Edson Tadeu)
Date: Thu, 4 Oct 2007 16:14:00 -0300
Subject: [Numpy-discussion] Best representation for array of points, or,
	how to distinguish a Nx1 array of points from a Nx3 array of scalars
In-Reply-To: <20071004173301.GH27995@mentat.za.net>
References: <a1b1282f0710040555g713e31d3w1052038866d8dfaa@mail.gmail.com>
	<4705192C.50607@gmail.com> <20071004173301.GH27995@mentat.za.net>
Message-ID: <a1b1282f0710041214v3ff4d9bdg4791437843758fd5@mail.gmail.com>

For now, I solved it using a view with a different type:

from numpy import *
N = 5
a=zeros(N,'3f8')
b=a.view()
b.dtype='f8'
b.shape = N,3


 I'll try to put this mechanism in a subclass.

  Thanks for the answers,
Edson


On 10/4/07, Stefan van der Walt <stefan at sun.ac.za> wrote:
>
> On Thu, Oct 04, 2007 at 11:47:40AM -0500, Robert Kern wrote:
> > Edson Tadeu wrote:
> >
> > > * Is there any field in the NumPy object where I can keep this
> > > information (the shape of the "element"), without creeping it with the
> > > dtype='(M,N)f8' representation I explained above?
> >
> > There isn't. You could make a subclass that tracks this, but you would
> have to
> > implement it carefully to maintain the information.
>
> There is an example of how to do that here:
>
> http://www.scipy.org/Subclasses
>
> Regards
> St?fan
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071004/b1f5884b/attachment.html>

From robert.kern at gmail.com  Thu Oct  4 15:51:09 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 04 Oct 2007 14:51:09 -0500
Subject: [Numpy-discussion] Best representation for array of points, or,
 how to distinguish a Nx1 array of points from a Nx3 array of scalars
In-Reply-To: <a1b1282f0710041214v3ff4d9bdg4791437843758fd5@mail.gmail.com>
References: <a1b1282f0710040555g713e31d3w1052038866d8dfaa@mail.gmail.com>	<4705192C.50607@gmail.com>
	<20071004173301.GH27995@mentat.za.net>
	<a1b1282f0710041214v3ff4d9bdg4791437843758fd5@mail.gmail.com>
Message-ID: <4705442D.1070801@gmail.com>

Edson Tadeu wrote:
> For now, I solved it using a view with a different type:
> 
> from numpy import *
> N = 5
> a=zeros(N,'3f8')
> b=a.view()
> b.dtype='f8'
> b.shape = N,3

Note that this doesn't quite do what you want. I was wrong about the dtype='3f8'
 syntax: it just gives you a regular float array with the shape changed
appropriately (at least in recent version of numpy; if you are getting something
different, what version are you using?).


In [24]: from numpy import *

In [25]: N = 5

In [26]: zeros(N, dtype='3f8')
Out[26]:
array([[ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.]])


If you want a shape-(N,) record array, you need a different dtype:


In [28]: zeros(N, dtype='f8,f8,f8')
Out[28]:
array([(0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0),
       (0.0, 0.0, 0.0)],
      dtype=[('f0', '<f8'), ('f1', '<f8'), ('f2', '<f8')])

In [31]: zeros(N, dtype=dtype([('x', float), ('y', float), ('z', float)]))
Out[31]:
array([(0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0), (0.0, 0.0, 0.0),
       (0.0, 0.0, 0.0)],
      dtype=[('x', '<f8'), ('y', '<f8'), ('z', '<f8')])


To go back, you can give .view() an argument:


In [30]: zeros(N, dtype='f8,f8,f8').view(dtype((float, 3)))
Out[30]:
array([[ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.]])


-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From aisaac at american.edu  Thu Oct  4 20:29:57 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 4 Oct 2007 20:29:57 -0400
Subject: [Numpy-discussion] Default value in documentation
In-Reply-To: <e06186140710021244mae0e49due80d654f320404c@mail.gmail.com>
References: <200710021323.44022.pgmdevlist@gmail.com><4702828C.1010507@gmail.com>
	<200710021410.50354.pgmdevlist@gmail.com><Mahogany-0.67.0-216-20071002-153603.00@american.edu><e06186140710021244mae0e49due80d654f320404c@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-1576-20071004-202957.00@american.edu>

On Tue, 2 Oct 2007, Charles R Harris apparently wrote:
> We're avoiding consolidated fields because they behave 
> badly.  ... The main problem with the consolidated fields 
> is that they are all put together as item lists in 
> a definition list and moved to the end of the docstring 
> when it is rendered. There is also little control through 
> the .css style sheet, they are all bulleted, and some of 
> the types are missing.


Is Ed Loper aware of this complaint?
If so, has he indicated an interest (or lack) in addressing this complaint?

Consolidated fields are great:
it is a shame to do without them.

Cheers,
Alan Isaac




From matthieu.brucher at gmail.com  Fri Oct  5 04:51:03 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 5 Oct 2007 10:51:03 +0200
Subject: [Numpy-discussion] Casting a float array into a string array
Message-ID: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>

Hi,

I'm trying to cast a float array into a string array (for instance
transforming [[2., 3.], [4., 5.]] into [['2.', '3.'], ['4.', '5.']]), I
tried with astype(str) and every variation (str_, string, string_, string0),
but not luck.
Is there a function or a method of the array class that can fulfill my needs
? And if there is a way to add a formatting option ('1.1f' for instance), it
would be even better.

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/669e51e0/attachment.html>

From dmitrey.kroshko at scipy.org  Fri Oct  5 07:56:25 2007
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Fri, 05 Oct 2007 14:56:25 +0300
Subject: [Numpy-discussion] howto convert float array to array of integers
Message-ID: <47062669.3000900@scipy.org>

hi all,
I have an array like
array([ 1.,  0.,  2.,  -10.])
what is most efficient (i.e. fastest) way to convert the one to array of 
integers?
array([ 1,  0,  2,  -10])

Thx in advance, D.


From chanley at stsci.edu  Fri Oct  5 08:20:39 2007
From: chanley at stsci.edu (Christopher Hanley)
Date: Fri, 05 Oct 2007 08:20:39 -0400
Subject: [Numpy-discussion] howto convert float array to array of
	integers
In-Reply-To: <47062669.3000900@scipy.org>
References: <47062669.3000900@scipy.org>
Message-ID: <47062C17.6080508@stsci.edu>

I usually use the astype method.

 >>> import numpy as n
 >>> a = n.array([1.,0.,2.,-10.])
 >>> a.dtype
dtype('float64')
 >>> print a
[  1.   0.   2. -10.]
 >>> b = a.astype(n.integer)
 >>> b.dtype
dtype('int32')
 >>> print b
[  1   0   2 -10]


dmitrey wrote:
> hi all,
> I have an array like
> array([ 1.,  0.,  2.,  -10.])
> what is most efficient (i.e. fastest) way to convert the one to array of 
> integers?
> array([ 1,  0,  2,  -10])
>
> Thx in advance, D.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   


-- 
Christopher Hanley
Systems Software Engineer
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21218
(410) 338-4338





From lbolla at gmail.com  Fri Oct  5 08:21:33 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Fri, 5 Oct 2007 14:21:33 +0200
Subject: [Numpy-discussion] howto convert float array to array of
	integers
In-Reply-To: <47062669.3000900@scipy.org>
References: <47062669.3000900@scipy.org>
Message-ID: <80c99e790710050521n3807fa96k66ecb840eb0de842@mail.gmail.com>

what about astype?

a.astype(t) -> Copy of array cast to type t.

    Cast array m to type t.  t can be either a string representing a
typecode,
    or a python type object of type int, float, or complex.

L.


On 10/5/07, dmitrey <dmitrey.kroshko at scipy.org> wrote:
>
> hi all,
> I have an array like
> array([ 1.,  0.,  2.,  -10.])
> what is most efficient (i.e. fastest) way to convert the one to array of
> integers?
> array([ 1,  0,  2,  -10])
>
> Thx in advance, D.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/580c285a/attachment.html>

From lbolla at gmail.com  Fri Oct  5 09:34:11 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Fri, 5 Oct 2007 15:34:11 +0200
Subject: [Numpy-discussion] Casting a float array into a string array
In-Reply-To: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
Message-ID: <80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>

what's wrong with astype?

In [3]: x = numpy.array([[2.,3.],[4.,5.]])

In [4]: x.astype(str)
Out[4]:
array([['2', '3'],
       ['4', '5']],
      dtype='|S1')

and if you want a list:

In [5]: x.astype(str).tolist()
Out[5]: [['2', '3'], ['4', '5']]


L.


On 10/5/07, Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
>
> Hi,
>
> I'm trying to cast a float array into a string array (for instance
> transforming [[2., 3.], [4., 5.]] into [['2.', '3.'], ['4.', '5.']]), I
> tried with astype(str) and every variation (str_, string, string_, string0),
> but not luck.
> Is there a function or a method of the array class that can fulfill my
> needs ? And if there is a way to add a formatting option ('1.1f' for
> instance), it would be even better.
>
> Matthieu
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/da665a8e/attachment.html>

From ndbecker2 at gmail.com  Fri Oct  5 09:33:28 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 05 Oct 2007 09:33:28 -0400
Subject: [Numpy-discussion] More complex data types
Message-ID: <fe5ef9$qpp$1@sea.gmane.org>

I'm thinking (again) about using numpy for signal processing applications. 
One issue is that there are more data types that are commonly used in
signal processing that are not available in numpy (or python). 
Specifically, it is frequently required to convert floating point
algorithms into integer algorithms.  numpy is fine for arrays of integers
(of various sizes), but it is also very useful to have arrays of
complex<integers>.  While numpy has complex<double,float>, it doesn't have
complex<int,int_64...>  Has anyone thought about this?



From matthieu.brucher at gmail.com  Fri Oct  5 10:19:00 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 5 Oct 2007 16:19:00 +0200
Subject: [Numpy-discussion] Casting a float array into a string array
In-Reply-To: <80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>
Message-ID: <e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>

I'd like to have the '2.', because if the number is negative, only '-' is
returned, not the real value.

Matthieu

2007/10/5, lorenzo bolla <lbolla at gmail.com>:
>
> what's wrong with astype?
>
> In [3]: x = numpy.array([[2.,3.],[4.,5.]])
>
> In [4]: x.astype(str)
> Out[4]:
> array([['2', '3'],
>        ['4', '5']],
>       dtype='|S1')
>
> and if you want a list:
>
> In [5]: x.astype(str).tolist()
> Out[5]: [['2', '3'], ['4', '5']]
>
>
> L.
>
>
> On 10/5/07, Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
>
> > Hi,
> >
> > I'm trying to cast a float array into a string array (for instance
> > transforming [[2., 3.], [4., 5.]] into [['2.', '3.'], ['4.', '5.']]), I
> > tried with astype(str) and every variation (str_, string, string_, string0),
> > but not luck.
> > Is there a function or a method of the array class that can fulfill my
> > needs ? And if there is a way to add a formatting option ('1.1f' for
> > instance), it would be even better.
> >
> > Matthieu
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/ea705ae8/attachment.html>

From simone.marras at bsc.es  Fri Oct  5 10:38:18 2007
From: simone.marras at bsc.es (Simone Marras)
Date: Fri, 05 Oct 2007 16:38:18 +0200
Subject: [Numpy-discussion] ERROR in installation of NumPy
Message-ID: <47064C5A.5010604@bsc.es>

Hello everyone,

I am trying to install numpy on my Suse 10.2 using Python 2.5
Python is correctly installed and when I launch > python setup.py
install, I get the following error:

    numpy/core/src/multiarraymodule.c:7604: fatal error: error writing
to /tmp/ccNImg9Q.s: No space left on devicetee: /tmp/tmpLmEe5Y:     No
space left on device

    compilation terminated.
    _exec_command_posix failed (status=256)




Can anyone help me, pelase?

thanks in advance,

simone



From peridot.faceted at gmail.com  Fri Oct  5 10:48:41 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 5 Oct 2007 10:48:41 -0400
Subject: [Numpy-discussion] Casting a float array into a string array
In-Reply-To: <e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>
	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>
Message-ID: <ce557a360710050748k74e3876dpa92e9610dca5f301@mail.gmail.com>

On 05/10/2007, Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
> I'd like to have the '2.', because if the number is negative, only '-' is
> returned, not the real value.

For string arrays you need to specify the length of the string as part
of the data type (and it defaults to length 1):

In [11]: random.normal(size=(2,3)).astype("|S20")
Out[11]:
array([['-0.223407464804', '1.1952765837', '-2.24073805089'],
       ['1.36604738338', '-0.197059880309', '2.15648625075']],
      dtype='|S20')

In [12]: random.normal(size=(2,3)).astype("|S2")
Out[12]:
array([['-0', '1.', '-0'],
       ['0.', '0.', '1.']],
      dtype='|S2')

In [13]: random.normal(size=(2,3)).astype("str")
Out[13]:
array([['-', '1', '1'],
       ['-', '-', '0']],
      dtype='|S1')

Anne


From lbolla at gmail.com  Fri Oct  5 10:53:33 2007
From: lbolla at gmail.com (lorenzo bolla)
Date: Fri, 5 Oct 2007 16:53:33 +0200
Subject: [Numpy-discussion] Casting a float array into a string array
In-Reply-To: <e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>
	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>
Message-ID: <80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>

gotcha. specify the number of bytes, then.

In [20]: x
Out[20]:
array([[-2.,  3.],
       [ 4.,  5.]])

In [21]: x.astype(numpy.dtype('S10'))
Out[21]:
array([['-2.0', '3.0'],
       ['4.0', '5.0']],
      dtype='|S10')

L.

On 10/5/07, Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
>
> I'd like to have the '2.', because if the number is negative, only '-' is
> returned, not the real value.
>
> Matthieu
>
> 2007/10/5, lorenzo bolla < lbolla at gmail.com>:
> >
> > what's wrong with astype?
> >
> > In [3]: x = numpy.array([[2.,3.],[4.,5.]])
> >
> > In [4]: x.astype(str)
> > Out[4]:
> > array([['2', '3'],
> >        ['4', '5']],
> >       dtype='|S1')
> >
> > and if you want a list:
> >
> > In [5]: x.astype(str).tolist()
> > Out[5]: [['2', '3'], ['4', '5']]
> >
> >
> > L.
> >
> >
> >  On 10/5/07, Matthieu Brucher < matthieu.brucher at gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I'm trying to cast a float array into a string array (for instance
> > > transforming [[2., 3.], [4., 5.]] into [['2.', '3.'], ['4.', '5.']]), I
> > > tried with astype(str) and every variation (str_, string, string_, string0),
> > > but not luck.
> > > Is there a function or a method of the array class that can fulfill my
> > > needs ? And if there is a way to add a formatting option ('1.1f' for
> > > instance), it would be even better.
> > >
> > > Matthieu
> > >
> > > _______________________________________________
> > > Numpy-discussion mailing list
> > > Numpy-discussion at scipy.org
> > > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> > >
> > >
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/ace9ba33/attachment.html>

From emanuele at relativita.com  Fri Oct  5 10:57:48 2007
From: emanuele at relativita.com (Emanuele Olivetti)
Date: Fri, 05 Oct 2007 16:57:48 +0200
Subject: [Numpy-discussion] ERROR in installation of NumPy
In-Reply-To: <47064C5A.5010604@bsc.es>
References: <47064C5A.5010604@bsc.es>
Message-ID: <470650EC.7070005@relativita.com>

Simone Marras wrote:
> Hello everyone,
>
> I am trying to install numpy on my Suse 10.2 using Python 2.5
> Python is correctly installed and when I launch > python setup.py
> install, I get the following error:
>
>     numpy/core/src/multiarraymodule.c:7604: fatal error: error writing
> to /tmp/ccNImg9Q.s: No space left on devicetee: /tmp/tmpLmEe5Y:     No
> space left on device
>
>     compilation terminated.
>     _exec_command_posix failed (status=256)
>
>
>
>
> Can anyone help me, pelase?
>
> thanks in advance,
>
>   

As the error message says, you have to free some space in
the partition where /tmp is. Your disk (or partition) is full
and intermediate/temporary files - needed by the installation
step - cannot be created. That's why installation fails.

Cheers,

Emanuele



From matthieu.brucher at gmail.com  Fri Oct  5 11:06:07 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 5 Oct 2007 17:06:07 +0200
Subject: [Numpy-discussion] Casting a float array into a string array
In-Reply-To: <80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>
	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>
	<80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>
Message-ID: <e76aa17f0710050806p79241f3al27655e6240c94c9b@mail.gmail.com>

Thank you for the precision, I didn't thought of using 'Sxx' directly :(

Matthieu

2007/10/5, lorenzo bolla <lbolla at gmail.com>:
>
> gotcha. specify the number of bytes, then.
>
> In [20]: x
> Out[20]:
> array([[-2.,  3.],
>        [ 4.,  5.]])
>
> In [21]: x.astype(numpy.dtype('S10'))
> Out[21]:
> array([['-2.0', '3.0'],
>        ['4.0', '5.0']],
>       dtype='|S10')
>
> L.
>
> On 10/5/07, Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
> >
> > I'd like to have the '2.', because if the number is negative, only '-'
> > is returned, not the real value.
> >
> > Matthieu
> >
> > 2007/10/5, lorenzo bolla < lbolla at gmail.com>:
> > >
> > > what's wrong with astype?
> > >
> > > In [3]: x = numpy.array([[2.,3.],[4.,5.]])
> > >
> > > In [4]: x.astype(str)
> > > Out[4]:
> > > array([['2', '3'],
> > >        ['4', '5']],
> > >       dtype='|S1')
> > >
> > > and if you want a list:
> > >
> > > In [5]: x.astype(str).tolist()
> > > Out[5]: [['2', '3'], ['4', '5']]
> > >
> > >
> > > L.
> > >
> > >
> > >  On 10/5/07, Matthieu Brucher < matthieu.brucher at gmail.com > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm trying to cast a float array into a string array (for instance
> > > > transforming [[2., 3.], [4., 5.]] into [['2.', '3.'], ['4.', '5.']]), I
> > > > tried with astype(str) and every variation (str_, string, string_, string0),
> > > > but not luck.
> > > > Is there a function or a method of the array class that can fulfill
> > > > my needs ? And if there is a way to add a formatting option ('1.1f'
> > > > for instance), it would be even better.
> > > >
> > > > Matthieu
> > > >
> > > > _______________________________________________
> > > > Numpy-discussion mailing list
> > > > Numpy-discussion at scipy.org
> > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> > > >
> > > >
> > >
> > > _______________________________________________
> > > Numpy-discussion mailing list
> > > Numpy-discussion at scipy.org
> > > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> > >
> > >
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/f205b2c8/attachment.html>

From charlesr.harris at gmail.com  Fri Oct  5 11:09:53 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 5 Oct 2007 09:09:53 -0600
Subject: [Numpy-discussion] More complex data types
In-Reply-To: <fe5ef9$qpp$1@sea.gmane.org>
References: <fe5ef9$qpp$1@sea.gmane.org>
Message-ID: <e06186140710050809y68df406u95a099411f5fd0f0@mail.gmail.com>

On 10/5/07, Neal Becker <ndbecker2 at gmail.com> wrote:
>
> I'm thinking (again) about using numpy for signal processing applications.
> One issue is that there are more data types that are commonly used in
> signal processing that are not available in numpy (or python).
> Specifically, it is frequently required to convert floating point
> algorithms into integer algorithms.  numpy is fine for arrays of integers
> (of various sizes), but it is also very useful to have arrays of
> complex<integers>.  While numpy has complex<double,float>, it doesn't have
> complex<int,int_64...>  Has anyone thought about this?


A bit. Multiplication begins to be a problem, though. Would you also want
fixed point multiplication with scaling, a la PPC with altivec? What about
division? So on and so forth. I think something like this would best be
implemented in a specialized signal processing package but I am not sure of
the best way to do it.

Chuck

_______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/b92e3453/attachment.html>

From simone.marras at bsc.es  Fri Oct  5 11:11:25 2007
From: simone.marras at bsc.es (Simone Marras)
Date: Fri, 05 Oct 2007 17:11:25 +0200
Subject: [Numpy-discussion] ERROR in installation of NumPy
In-Reply-To: <470650EC.7070005@relativita.com>
References: <47064C5A.5010604@bsc.es> <470650EC.7070005@relativita.com>
Message-ID: <4706541D.9050902@bsc.es>

Hi Emanuele,

thanks a lot for replying. Look, actually my machine is bare new, so I
dont understand why my /tmp gets full for every small operation I do.
Now I couldnt even send out an email. do you know what this problem
could be?

Thank you,

All the best

Simone

Emanuele Olivetti wrote:
> Simone Marras wrote:
>   
>> Hello everyone,
>>
>> I am trying to install numpy on my Suse 10.2 using Python 2.5
>> Python is correctly installed and when I launch > python setup.py
>> install, I get the following error:
>>
>>     numpy/core/src/multiarraymodule.c:7604: fatal error: error writing
>> to /tmp/ccNImg9Q.s: No space left on devicetee: /tmp/tmpLmEe5Y:     No
>> space left on device
>>
>>     compilation terminated.
>>     _exec_command_posix failed (status=256)
>>
>>
>>
>>
>> Can anyone help me, pelase?
>>
>> thanks in advance,
>>
>>   
>>     
>
> As the error message says, you have to free some space in
> the partition where /tmp is. Your disk (or partition) is full
> and intermediate/temporary files - needed by the installation
> step - cannot be created. That's why installation fails.
>
> Cheers,
>
> Emanuele
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/da5b5aba/attachment.html>

From gael.varoquaux at normalesup.org  Fri Oct  5 11:29:08 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 5 Oct 2007 17:29:08 +0200
Subject: [Numpy-discussion] ERROR in installation of NumPy
In-Reply-To: <4706541D.9050902@bsc.es>
References: <47064C5A.5010604@bsc.es> <470650EC.7070005@relativita.com>
	<4706541D.9050902@bsc.es>
Message-ID: <20071005152908.GA30895@clipper.ens.fr>

On Fri, Oct 05, 2007 at 05:11:25PM +0200, Simone Marras wrote:
>    thanks a lot for replying. Look, actually my machine is bare new, so I
>    dont understand why my /tmp gets full for every small operation I do. Now
>    I couldnt even send out an email. do you know what this problem could be?

Looks like an installation problem. Can you give us the output of "df
-h".

Ga?l


From simone.marras at bsc.es  Fri Oct  5 11:35:46 2007
From: simone.marras at bsc.es (Simone Marras)
Date: Fri, 05 Oct 2007 17:35:46 +0200
Subject: [Numpy-discussion] ERROR in installation of NumPy
In-Reply-To: <20071005152908.GA30895@clipper.ens.fr>
References: <47064C5A.5010604@bsc.es>
	<470650EC.7070005@relativita.com>	<4706541D.9050902@bsc.es>
	<20071005152908.GA30895@clipper.ens.fr>
Message-ID: <470659D2.1080306@bsc.es>

Hi there, here the output,
thanks

simone at linux-bstu:/home> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             7.4G  7.0G   14M 100% /
udev                  506M  112K  506M   1% /dev
/dev/sda6              69G   14G   52G  22% /home
/dev/sdb1             459G  295G  141G  68% /media/disk


Simone







Gael Varoquaux wrote:
> On Fri, Oct 05, 2007 at 05:11:25PM +0200, Simone Marras wrote:
>   
>>    thanks a lot for replying. Look, actually my machine is bare new, so I
>>    dont understand why my /tmp gets full for every small operation I do. Now
>>    I couldnt even send out an email. do you know what this problem could be?
>>     
>
> Looks like an installation problem. Can you give us the output of "df
> -h".
>
> Ga?l
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/16dd695f/attachment.html>

From gael.varoquaux at normalesup.org  Fri Oct  5 11:40:57 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 5 Oct 2007 17:40:57 +0200
Subject: [Numpy-discussion] ERROR in installation of NumPy
In-Reply-To: <470659D2.1080306@bsc.es>
References: <47064C5A.5010604@bsc.es> <470650EC.7070005@relativita.com>
	<4706541D.9050902@bsc.es> <20071005152908.GA30895@clipper.ens.fr>
	<470659D2.1080306@bsc.es>
Message-ID: <20071005154057.GB30895@clipper.ens.fr>

On Fri, Oct 05, 2007 at 05:35:46PM +0200, Simone Marras wrote:
>    simone at linux-bstu:/home> df -h
>    Filesystem            Size  Used Avail Use% Mounted on
>    /dev/sda2             7.4G  7.0G   14M 100% /
>    udev                  506M  112K  506M   1% /dev
>    /dev/sda6              69G   14G   52G  22% /home
>    /dev/sdb1             459G  295G  141G  68% /media/disk

OK, well the answer is simple, you do not have enough space on your "/"
drive. 7.0G seems a lot for a linux install, but is not totally
ridiculous. I would say whoever installed the box didn't allocate enough
space for "/". I certainly would not put less than 10G, not because I
can't run with less, but because it forces me to do more maintainance.

Workaround : resize the partitions.

Ga?l


From simone.marras at bsc.es  Fri Oct  5 11:45:29 2007
From: simone.marras at bsc.es (Simone Marras)
Date: Fri, 05 Oct 2007 17:45:29 +0200
Subject: [Numpy-discussion] ERROR in installation of NumPy
In-Reply-To: <20071005154057.GB30895@clipper.ens.fr>
References: <47064C5A.5010604@bsc.es>
	<470650EC.7070005@relativita.com>	<4706541D.9050902@bsc.es>
	<20071005152908.GA30895@clipper.ens.fr>	<470659D2.1080306@bsc.es>
	<20071005154057.GB30895@clipper.ens.fr>
Message-ID: <47065C19.4040606@bsc.es>

Hi Gael,

I will ask the guy who installed the system yesterday, so that I hope he
can fix it on monday without me having to re-install all the libraries I
installed already

Thanks a lot for the help

All the best

Simone

Gael Varoquaux wrote:
> On Fri, Oct 05, 2007 at 05:35:46PM +0200, Simone Marras wrote:
>   
>>    simone at linux-bstu:/home> df -h
>>    Filesystem            Size  Used Avail Use% Mounted on
>>    /dev/sda2             7.4G  7.0G   14M 100% /
>>    udev                  506M  112K  506M   1% /dev
>>    /dev/sda6              69G   14G   52G  22% /home
>>    /dev/sdb1             459G  295G  141G  68% /media/disk
>>     
>
> OK, well the answer is simple, you do not have enough space on your "/"
> drive. 7.0G seems a lot for a linux install, but is not totally
> ridiculous. I would say whoever installed the box didn't allocate enough
> space for "/". I certainly would not put less than 10G, not because I
> can't run with less, but because it forces me to do more maintainance.
>
> Workaround : resize the partitions.
>
> Ga?l
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/34f9b6a5/attachment.html>

From ndbecker2 at gmail.com  Fri Oct  5 11:48:39 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 05 Oct 2007 11:48:39 -0400
Subject: [Numpy-discussion] More complex data types
References: <fe5ef9$qpp$1@sea.gmane.org>
	<e06186140710050809y68df406u95a099411f5fd0f0@mail.gmail.com>
Message-ID: <fe5mco$qcf$1@sea.gmane.org>

Charles R Harris wrote:

> On 10/5/07, Neal Becker <ndbecker2 at gmail.com> wrote:
>>
>> I'm thinking (again) about using numpy for signal processing
>> applications. One issue is that there are more data types that are
>> commonly used in signal processing that are not available in numpy (or
>> python). Specifically, it is frequently required to convert floating
>> point
>> algorithms into integer algorithms.  numpy is fine for arrays of integers
>> (of various sizes), but it is also very useful to have arrays of
>> complex<integers>.  While numpy has complex<double,float>, it doesn't
>> have
>> complex<int,int_64...>  Has anyone thought about this?
> 
> 
> A bit. Multiplication begins to be a problem, though. Would you also want
> fixed point multiplication with scaling, a la PPC with altivec? What about
> division? So on and so forth. I think something like this would best be
> implemented in a specialized signal processing package but I am not sure
> of the best way to do it.
> 

I'd keep things as simple as possible.  No fixed point/scaling.  It's simple
enough to explictly rescale things as you wish.

That is (using c++ syntax):
complex<int> a, b;
complex<int> c = a * b;
complex<int> d = d >> 4;

Complicating life is interoperability (conversion) of types.

I've used this concept for some years with c++/python - but not with numpy. 
It's pretty trivial to make a complex<int> type as a C extension to python. 
Adding this to numpy would be really useful.



From broman at spawar.navy.mil  Fri Oct  5 11:54:29 2007
From: broman at spawar.navy.mil (Vincent Broman)
Date: Fri, 5 Oct 2007 08:54:29 -0700
Subject: [Numpy-discussion] Numpy-discussion Digest, Vol 13, Issue 11
In-Reply-To: <mailman.98678.1191596770.30507.numpy-discussion@scipy.org>
References: <mailman.98678.1191596770.30507.numpy-discussion@scipy.org>
Message-ID: <200710050854.29329@b00d61a8cecf8b2266f81358fd170621.navy.mil>

On Friday, 05 October 2007, Neal Becker wrote:
> I'm thinking (again) about using numpy for signal processing applications. 
> One issue is that there are more data types that are commonly used in
> signal processing that are not available in numpy (or python). 
> Specifically, it is frequently required to convert floating point
> algorithms into integer algorithms. ?numpy is fine for arrays of integers
> (of various sizes), but it is also very useful to have arrays of
> complex<integers>. ?While numpy has complex<double,float>, it doesn't have
> complex<int,int_64...> ?Has anyone thought about this?

Complex ints (or Gaussian integers) are algebraically interesting
but they're just a Ring, not a Field. Before you know it,
you're going to want to divide something.  In signal processing,
int measurements are usually not counters, they're just fixed point numbers
with an implicit scale, and you only want to do integer arithmetic
to save time/space, not because it is a good conceptual fit to the problem.
In an interpreted language like Python, fixed point seems like
a strange bird. I just use complex floats and a fast library.

Vincent Broman


From charlesr.harris at gmail.com  Fri Oct  5 12:08:45 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 5 Oct 2007 10:08:45 -0600
Subject: [Numpy-discussion] More complex data types
In-Reply-To: <fe5mco$qcf$1@sea.gmane.org>
References: <fe5ef9$qpp$1@sea.gmane.org>
	<e06186140710050809y68df406u95a099411f5fd0f0@mail.gmail.com>
	<fe5mco$qcf$1@sea.gmane.org>
Message-ID: <e06186140710050908y51892d0aue4c1813314484bd4@mail.gmail.com>

On 10/5/07, Neal Becker <ndbecker2 at gmail.com> wrote:
>
> Charles R Harris wrote:
>
> > On 10/5/07, Neal Becker <ndbecker2 at gmail.com> wrote:
> >>
> >> I'm thinking (again) about using numpy for signal processing
> >> applications. One issue is that there are more data types that are
> >> commonly used in signal processing that are not available in numpy (or
> >> python). Specifically, it is frequently required to convert floating
> >> point
> >> algorithms into integer algorithms.  numpy is fine for arrays of
> integers
> >> (of various sizes), but it is also very useful to have arrays of
> >> complex<integers>.  While numpy has complex<double,float>, it doesn't
> >> have
> >> complex<int,int_64...>  Has anyone thought about this?
> >
> >
> > A bit. Multiplication begins to be a problem, though. Would you also
> want
> > fixed point multiplication with scaling, a la PPC with altivec? What
> about
> > division? So on and so forth. I think something like this would best be
> > implemented in a specialized signal processing package but I am not sure
> > of the best way to do it.
> >
>
> I'd keep things as simple as possible.  No fixed point/scaling.  It's
> simple
> enough to explictly rescale things as you wish.
>
> That is (using c++ syntax):
> complex<int> a, b;
> complex<int> c = a * b;
> complex<int> d = d >> 4;
>
> Complicating life is interoperability (conversion) of types.
>
> I've used this concept for some years with c++/python - but not with
> numpy.
> It's pretty trivial to make a complex<int> type as a C extension to
> python.
> Adding this to numpy would be really useful.


How about fiddling with floating point to emulate integers by subclassing
ndarray? That wouldn't buy you the speed and size advantage of true fixed
point but would make a flexible emulator. Which raises the question, what
are your main goals in using such a data type? Not that I don't see the
natural utility of having complex integer numbers (Gaussian integers), but
if you are trying to emulate hardware something more flexible might be
appropriate.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/664fd3c0/attachment.html>

From ndbecker2 at gmail.com  Fri Oct  5 12:50:21 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 05 Oct 2007 12:50:21 -0400
Subject: [Numpy-discussion] More complex data types
References: <fe5ef9$qpp$1@sea.gmane.org>
	<e06186140710050809y68df406u95a099411f5fd0f0@mail.gmail.com>
	<fe5mco$qcf$1@sea.gmane.org>
	<e06186140710050908y51892d0aue4c1813314484bd4@mail.gmail.com>
Message-ID: <fe5q0d$775$1@sea.gmane.org>

Charles R Harris wrote:

> On 10/5/07, Neal Becker <ndbecker2 at gmail.com> wrote:
>>
>> Charles R Harris wrote:
>>
>> > On 10/5/07, Neal Becker <ndbecker2 at gmail.com> wrote:
>> >>
>> >> I'm thinking (again) about using numpy for signal processing
>> >> applications. One issue is that there are more data types that are
>> >> commonly used in signal processing that are not available in numpy (or
>> >> python). Specifically, it is frequently required to convert floating
>> >> point
>> >> algorithms into integer algorithms.  numpy is fine for arrays of
>> integers
>> >> (of various sizes), but it is also very useful to have arrays of
>> >> complex<integers>.  While numpy has complex<double,float>, it doesn't
>> >> have
>> >> complex<int,int_64...>  Has anyone thought about this?
>> >
>> >
>> > A bit. Multiplication begins to be a problem, though. Would you also
>> want
>> > fixed point multiplication with scaling, a la PPC with altivec? What
>> about
>> > division? So on and so forth. I think something like this would best be
>> > implemented in a specialized signal processing package but I am not
>> > sure of the best way to do it.
>> >
>>
>> I'd keep things as simple as possible.  No fixed point/scaling.  It's
>> simple
>> enough to explictly rescale things as you wish.
>>
>> That is (using c++ syntax):
>> complex<int> a, b;
>> complex<int> c = a * b;
>> complex<int> d = d >> 4;
>>
>> Complicating life is interoperability (conversion) of types.
>>
>> I've used this concept for some years with c++/python - but not with
>> numpy.
>> It's pretty trivial to make a complex<int> type as a C extension to
>> python.
>> Adding this to numpy would be really useful.
> 
> 
> How about fiddling with floating point to emulate integers by subclassing
> ndarray? That wouldn't buy you the speed and size advantage of true fixed
> point but would make a flexible emulator. Which raises the question, what
> are your main goals in using such a data type? Not that I don't see the
> natural utility of having complex integer numbers (Gaussian integers), but
> if you are trying to emulate hardware something more flexible might be
> appropriate.
> 
> Chuck

Yes, this is intended for modelling hardware.  I don't know what you mean
by "more flexible".  I design my hardware algorithms to use integer
arithmetic.  What did you have in mind?



From charlesr.harris at gmail.com  Fri Oct  5 13:16:24 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 5 Oct 2007 11:16:24 -0600
Subject: [Numpy-discussion] More complex data types
In-Reply-To: <fe5q0d$775$1@sea.gmane.org>
References: <fe5ef9$qpp$1@sea.gmane.org>
	<e06186140710050809y68df406u95a099411f5fd0f0@mail.gmail.com>
	<fe5mco$qcf$1@sea.gmane.org>
	<e06186140710050908y51892d0aue4c1813314484bd4@mail.gmail.com>
	<fe5q0d$775$1@sea.gmane.org>
Message-ID: <e06186140710051016i5c2fc4fay3d11652dec545077@mail.gmail.com>

On 10/5/07, Neal Becker <ndbecker2 at gmail.com> wrote:
>
> Charles R Harris wrote:
>
> > On 10/5/07, Neal Becker <ndbecker2 at gmail.com> wrote:
> >>
> >> Charles R Harris wrote:
> >>
> >> > On 10/5/07, Neal Becker <ndbecker2 at gmail.com> wrote:
> >> >>
> >> >> I'm thinking (again) about using numpy for signal processing
> >> >> applications. One issue is that there are more data types that are
> >> >> commonly used in signal processing that are not available in numpy
> (or
> >> >> python). Specifically, it is frequently required to convert floating
> >> >> point
> >> >> algorithms into integer algorithms.  numpy is fine for arrays of
> >> integers
> >> >> (of various sizes), but it is also very useful to have arrays of
> >> >> complex<integers>.  While numpy has complex<double,float>, it
> doesn't
> >> >> have
> >> >> complex<int,int_64...>  Has anyone thought about this?
> >> >
> >> >
> >> > A bit. Multiplication begins to be a problem, though. Would you also
> >> want
> >> > fixed point multiplication with scaling, a la PPC with altivec? What
> >> about
> >> > division? So on and so forth. I think something like this would best
> be
> >> > implemented in a specialized signal processing package but I am not
> >> > sure of the best way to do it.
> >> >
> >>
> >> I'd keep things as simple as possible.  No fixed point/scaling.  It's
> >> simple
> >> enough to explictly rescale things as you wish.
> >>
> >> That is (using c++ syntax):
> >> complex<int> a, b;
> >> complex<int> c = a * b;
> >> complex<int> d = d >> 4;
> >>
> >> Complicating life is interoperability (conversion) of types.
> >>
> >> I've used this concept for some years with c++/python - but not with
> >> numpy.
> >> It's pretty trivial to make a complex<int> type as a C extension to
> >> python.
> >> Adding this to numpy would be really useful.
> >
> >
> > How about fiddling with floating point to emulate integers by
> subclassing
> > ndarray? That wouldn't buy you the speed and size advantage of true
> fixed
> > point but would make a flexible emulator. Which raises the question,
> what
> > are your main goals in using such a data type? Not that I don't see the
> > natural utility of having complex integer numbers (Gaussian integers),
> but
> > if you are trying to emulate hardware something more flexible might be
> > appropriate.
> >
> > Chuck
>
> Yes, this is intended for modelling hardware.  I don't know what you mean
> by "more flexible".  I design my hardware algorithms to use integer
> arithmetic.  What did you have in mind?


Well, you could do bounds and overflow checking, use mixed integer
precisions, and also deal with oddball sizes, such as 12 bits. You might
also support 16 bit floats, something that is not available as a native type
on most platforms. I'm just tossing some ideas out there. As I say, I don't
see any reason not to have integer complex arrays, if only for the data
type. Then again, maybe you could define your own data type and just
overload the operators, something like

In [5]: complex32 = dtype('int16,int16')

In [6]: zeros(2, dtype=complex32)
Out[6]:
array([(0, 0), (0, 0)],
      dtype=[('f0', '<i2'), ('f1', '<i2')])

I suspect that the tools you want for emulation and debugging  would be hard
to get with exact replication.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071005/4032b951/attachment.html>

From jdh2358 at gmail.com  Fri Oct  5 14:02:00 2007
From: jdh2358 at gmail.com (John Hunter)
Date: Fri, 5 Oct 2007 13:02:00 -0500
Subject: [Numpy-discussion] adding field to rec array
In-Reply-To: <46FA9FF5.7070108@gmail.com>
References: <88e473830709261040g65cfa145w464c94b891c16591@mail.gmail.com>
	<46FA9FF5.7070108@gmail.com>
Message-ID: <88e473830710051102g69e709adu7e914e63b2c9faa5@mail.gmail.com>

On 9/26/07, Robert Kern <robert.kern at gmail.com> wrote:

> Here is the straightforward way:
>
> In [15]: import numpy as np
>
> In [16]: dt = np.dtype([('foo', int), ('bar', float)])
>
> In [17]: r = np.zeros((3,3), dtype=dt)

Here is a (hopefully) simple question.  If I create an array like
this, how can I efficiently convert it to a record array which lets me
do r.attr in addition to r['attr'].  I'm pretty addicted to the former
syntax.

In [114]: dt = np.dtype([('foo', int), ('bar', float)])

In [115]: r = np.zeros((3,3), dtype=dt)

In [116]: r.dtype
Out[116]: dtype([('foo', '<i4'), ('bar', '<f8')])

In [117]: r['foo']
Out[117]:
array([[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]])

In [118]: r.foo
------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in ?
AttributeError: 'numpy.ndarray' object has no attribute 'foo'


Thanks,
JDH


From barrywark at gmail.com  Fri Oct  5 14:05:56 2007
From: barrywark at gmail.com (Barry Wark)
Date: Fri, 5 Oct 2007 11:05:56 -0700
Subject: [Numpy-discussion] adding field to rec array
In-Reply-To: <88e473830710051102g69e709adu7e914e63b2c9faa5@mail.gmail.com>
References: <88e473830709261040g65cfa145w464c94b891c16591@mail.gmail.com>
	<46FA9FF5.7070108@gmail.com>
	<88e473830710051102g69e709adu7e914e63b2c9faa5@mail.gmail.com>
Message-ID: <cd7634ce0710051105w29f53b68s126f97e391f76d09@mail.gmail.com>

Try

r = r.view(numpy.recarray)

barry

On 10/5/07, John Hunter <jdh2358 at gmail.com> wrote:
> On 9/26/07, Robert Kern <robert.kern at gmail.com> wrote:
>
> > Here is the straightforward way:
> >
> > In [15]: import numpy as np
> >
> > In [16]: dt = np.dtype([('foo', int), ('bar', float)])
> >
> > In [17]: r = np.zeros((3,3), dtype=dt)
>
> Here is a (hopefully) simple question.  If I create an array like
> this, how can I efficiently convert it to a record array which lets me
> do r.attr in addition to r['attr'].  I'm pretty addicted to the former
> syntax.
>
> In [114]: dt = np.dtype([('foo', int), ('bar', float)])
>
> In [115]: r = np.zeros((3,3), dtype=dt)
>
> In [116]: r.dtype
> Out[116]: dtype([('foo', '<i4'), ('bar', '<f8')])
>
> In [117]: r['foo']
> Out[117]:
> array([[0, 0, 0],
>        [0, 0, 0],
>        [0, 0, 0]])
>
> In [118]: r.foo
> ------------------------------------------------------------
> Traceback (most recent call last):
>   File "<ipython console>", line 1, in ?
> AttributeError: 'numpy.ndarray' object has no attribute 'foo'
>
>
> Thanks,
> JDH
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From robert.kern at gmail.com  Fri Oct  5 14:05:56 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 05 Oct 2007 13:05:56 -0500
Subject: [Numpy-discussion] adding field to rec array
In-Reply-To: <88e473830710051102g69e709adu7e914e63b2c9faa5@mail.gmail.com>
References: <88e473830709261040g65cfa145w464c94b891c16591@mail.gmail.com>	<46FA9FF5.7070108@gmail.com>
	<88e473830710051102g69e709adu7e914e63b2c9faa5@mail.gmail.com>
Message-ID: <47067D04.7030906@gmail.com>

John Hunter wrote:
> On 9/26/07, Robert Kern <robert.kern at gmail.com> wrote:
> 
>> Here is the straightforward way:
>>
>> In [15]: import numpy as np
>>
>> In [16]: dt = np.dtype([('foo', int), ('bar', float)])
>>
>> In [17]: r = np.zeros((3,3), dtype=dt)
> 
> Here is a (hopefully) simple question.  If I create an array like
> this, how can I efficiently convert it to a record array which lets me
> do r.attr in addition to r['attr'].  I'm pretty addicted to the former
> syntax.

In [32]: r2 = r.view(np.recarray)

In [33]: r2.foo
Out[33]:
array([[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]])

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From aisaac at american.edu  Fri Oct  5 15:21:31 2007
From: aisaac at american.edu (Alan Isaac)
Date: Fri, 5 Oct 2007 15:21:31 -0400
Subject: [Numpy-discussion] howto convert float array to array of
	integers
In-Reply-To: <47062669.3000900@scipy.org>
References: <47062669.3000900@scipy.org>
Message-ID: <Mahogany-0.67.0-3696-20071005-152131.00@CASC64043305.american.edu>

On Fri, 05 Oct 2007, dmitrey wrote:
> I have an array like array([ 1.,  0.,  2.,  -10.])
> what is most efficient (i.e. fastest) way to convert the one to array of 
> integers? 
> array([ 1,  0,  2,  -10])

Use ``astype``.

Cheers,
Alan Isaac



>>> import numpy as N
>>> x = N.array([1,2,3],dtype='float')
>>> x
array([ 1.,  2.,  3.])
>>> x.astype('int')
array([1, 2, 3])
>>>




From Chris.Barker at noaa.gov  Fri Oct  5 17:18:40 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 05 Oct 2007 14:18:40 -0700
Subject: [Numpy-discussion] Casting a float array into a string array
In-Reply-To: <e76aa17f0710050806p79241f3al27655e6240c94c9b@mail.gmail.com>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>
	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>
	<80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>
	<e76aa17f0710050806p79241f3al27655e6240c94c9b@mail.gmail.com>
Message-ID: <4706AA30.2080505@noaa.gov>

Matthieu Brucher wrote:

>                 And if there is a way to add a
>                 formatting option ('1.1f' for instance), it would be
>                 even better.

For full control of the formatting, you can use python's string 
formatting, and a nested list comprehension:

[ ["%.3f"%i for i in r] for r in x]

I don't know how to generalize this to n-d though -- maybe numpy.vectorize?

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From sdb at cloud9.net  Fri Oct  5 17:32:03 2007
From: sdb at cloud9.net (Stuart Brorson)
Date: Fri, 5 Oct 2007 17:32:03 -0400 (EDT)
Subject: [Numpy-discussion] histogram using decending range -- what do the
	results mean?
In-Reply-To: <4706AA30.2080505@noaa.gov>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>
	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>
	<80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>
	<e76aa17f0710050806p79241f3al27655e6240c94c9b@mail.gmail.com>
	<4706AA30.2080505@noaa.gov>
Message-ID: <Pine.BSF.4.64.0710051724450.50575@earl-grey.cloud9.net>

Guys --

I'm a little puzzled by a NumPy behavior.  Perhaps the gurus on this
list can enlighten me, please!

I am working with numpy.histogram.  I have a decent understanding of
how it works when given an ascending range to bin into.  However, when
I give it a *decending* range, I can't figure out what the results
mean.  Here's an example:

------------------------  <session log>  --------------------

>>> A = numpy.array([1, 2, 3, 4, 5, 6, 5, 4, 5, 4, 3, 2, 1])
>>> (x, y) = numpy.histogram(A, range=(0, 7))
>>> x
array([0, 2, 2, 0, 2, 3, 0, 3, 1, 0])
>>> 
>>> (x, y) = numpy.histogram(A, range=(7, 0))
>>> x
array([ 0, -1, -3,  0, -3, -2,  0, -2, -2, 13])
>>> 
--------------------  </session log>  ------------------------

Please set aside the natural response "the user shouldn't bin into 
a decending range!" since I am trying to figure out what computation
NumPy actually does in this case and I don't want a work-around.  And
yes, I have looked at the source.  It's nicely vectorized, so I find
the source rather opaque.

Therefore, I would appreciate it if if some kind soul could answer a
couple of questions:

*  What does the return mean for range=(7, 0)?

*  Why should the return histogram have negative elements?

*  If it truely isn't meaningful, why not catch the case and reject
input?  Maybe this is a bug....  ???

Thanks!

Stuart


From peridot.faceted at gmail.com  Fri Oct  5 17:39:28 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 5 Oct 2007 17:39:28 -0400
Subject: [Numpy-discussion] bug in vectorize? (was: Re: Casting a float
	array into a string array)
Message-ID: <ce557a360710051439i2c230db8pa86d6e0012f70f40@mail.gmail.com>

On 05/10/2007, Christopher Barker <Chris.Barker at noaa.gov> wrote:

> I don't know how to generalize this to n-d though -- maybe numpy.vectorize?

Oops! Looks like there's a big somewhere:

In [1]: from numpy import *

In [2]: vectorize(lambda x: "%5.3g" % x)(ones((2,2,2)))
Out[2]:
array([[[' ', '\xc1'],
        [' ', '\xc1']],

       [[' ', '\xc1'],
        [' ', '\xc1']]],
      dtype='|S1')

In [3]: vectorize?
Segmentation fault (core dumped)

In particular, vectorize creates a string array (rather than an object
array), and some unsafe operation is done on that array.

Anne


From robert.kern at gmail.com  Fri Oct  5 17:41:04 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 05 Oct 2007 16:41:04 -0500
Subject: [Numpy-discussion] histogram using decending range -- what do
 the	results mean?
In-Reply-To: <Pine.BSF.4.64.0710051724450.50575@earl-grey.cloud9.net>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>	<80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>	<e76aa17f0710050806p79241f3al27655e6240c94c9b@mail.gmail.com>	<4706AA30.2080505@noaa.gov>
	<Pine.BSF.4.64.0710051724450.50575@earl-grey.cloud9.net>
Message-ID: <4706AF70.2060208@gmail.com>

Stuart Brorson wrote:
> Guys --
> 
> I'm a little puzzled by a NumPy behavior.  Perhaps the gurus on this
> list can enlighten me, please!
> 
> I am working with numpy.histogram.  I have a decent understanding of
> how it works when given an ascending range to bin into.  However, when
> I give it a *decending* range, I can't figure out what the results
> mean.  Here's an example:
> 
> ------------------------  <session log>  --------------------
> 
>>>> A = numpy.array([1, 2, 3, 4, 5, 6, 5, 4, 5, 4, 3, 2, 1])
>>>> (x, y) = numpy.histogram(A, range=(0, 7))
>>>> x
> array([0, 2, 2, 0, 2, 3, 0, 3, 1, 0])
>>>> (x, y) = numpy.histogram(A, range=(7, 0))
>>>> x
> array([ 0, -1, -3,  0, -3, -2,  0, -2, -2, 13])
> --------------------  </session log>  ------------------------
> 
> Please set aside the natural response "the user shouldn't bin into 
> a decending range!" since I am trying to figure out what computation
> NumPy actually does in this case and I don't want a work-around.  And
> yes, I have looked at the source.  It's nicely vectorized, so I find
> the source rather opaque.
> 
> Therefore, I would appreciate it if if some kind soul could answer a
> couple of questions:
> 
> *  What does the return mean for range=(7, 0)?

Nothing.

> *  Why should the return histogram have negative elements?

Because there are subtractions involved that depend on the bins being increasing
which they are not if the given range is incorrect.

> *  If it truely isn't meaningful, why not catch the case and reject
> input?  Maybe this is a bug....  ???

Patches are welcome.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From mpmusu at cc.usu.edu  Fri Oct  5 17:54:50 2007
From: mpmusu at cc.usu.edu (Mark.Miller)
Date: Fri, 05 Oct 2007 15:54:50 -0600
Subject: [Numpy-discussion] histogram using decending range -- what do
 the results mean?
In-Reply-To: <Pine.BSF.4.64.0710051724450.50575@earl-grey.cloud9.net>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>
	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>
	<80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>
	<e76aa17f0710050806p79241f3al27655e6240c94c9b@mail.gmail.com>
	<4706AA30.2080505@noaa.gov>
	<Pine.BSF.4.64.0710051724450.50575@earl-grey.cloud9.net>
Message-ID: <4706B2AA.5090805@cc.usu.edu>

Check how you're implementing the histogram function with respect to 
that range statement.  It seems to make a difference, desirable or not.

 >>> import numpy
 >>> numpy.__version__
'1.0.4.dev3982'
 >>> A = numpy.array([1, 2, 3, 4, 5, 6, 5, 4, 5, 4, 3, 2, 1])
 >>> (x, y) = numpy.histogram(A, range(0, 7))
 >>> x
array([0, 2, 2, 2, 3, 3, 1])
 >>> y
[0, 1, 2, 3, 4, 5, 6]
 >>>
 >>> (x, y) = numpy.histogram(A, range=(0, 7))
 >>> x
array([0, 2, 2, 0, 2, 3, 0, 3, 1, 0])
 >>> y
array([ 0. ,  0.7,  1.4,  2.1,  2.8,  3.5,  4.2,  4.9,  5.6,  6.3])
 >>>
 >>>
 >>> (x, y) = numpy.histogram(A, range(7,0))
 >>> x
array([], dtype=int32)
 >>> y
[]
 >>>

Note that in the last case, the histogram function isn't returning 
anything for a descending range.

Also notice that you're overwriting a python function with the way 
you're assigning things....

 >>> range(0,7)
[0, 1, 2, 3, 4, 5, 6]
 >>> range=(0,7)
 >>> range
(0, 7)

I'll leave it to others to decide if this is problematic.

-Mark

Stuart Brorson wrote:
> Guys --
> 
> I'm a little puzzled by a NumPy behavior.  Perhaps the gurus on this
> list can enlighten me, please!
> 
> I am working with numpy.histogram.  I have a decent understanding of
> how it works when given an ascending range to bin into.  However, when
> I give it a *decending* range, I can't figure out what the results
> mean.  Here's an example:
> 
> ------------------------  <session log>  --------------------
> 
>>>> A = numpy.array([1, 2, 3, 4, 5, 6, 5, 4, 5, 4, 3, 2, 1])
>>>> (x, y) = numpy.histogram(A, range=(0, 7))
>>>> x
> array([0, 2, 2, 0, 2, 3, 0, 3, 1, 0])
>>>> (x, y) = numpy.histogram(A, range=(7, 0))
>>>> x
> array([ 0, -1, -3,  0, -3, -2,  0, -2, -2, 13])
> --------------------  </session log>  ------------------------
> 
> Please set aside the natural response "the user shouldn't bin into 
> a decending range!" since I am trying to figure out what computation
> NumPy actually does in this case and I don't want a work-around.  And
> yes, I have looked at the source.  It's nicely vectorized, so I find
> the source rather opaque.
> 
> Therefore, I would appreciate it if if some kind soul could answer a
> couple of questions:
> 
> *  What does the return mean for range=(7, 0)?
> 
> *  Why should the return histogram have negative elements?
> 
> *  If it truely isn't meaningful, why not catch the case and reject
> input?  Maybe this is a bug....  ???
> 
> Thanks!
> 
> Stuart
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Fri Oct  5 18:04:47 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 05 Oct 2007 17:04:47 -0500
Subject: [Numpy-discussion] histogram using decending range -- what do
 the results mean?
In-Reply-To: <4706B2AA.5090805@cc.usu.edu>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>	<80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>	<e76aa17f0710050806p79241f3al27655e6240c94c9b@mail.gmail.com>	<4706AA30.2080505@noaa.gov>	<Pine.BSF.4.64.0710051724450.50575@earl-grey.cloud9.net>
	<4706B2AA.5090805@cc.usu.edu>
Message-ID: <4706B4FF.4040908@gmail.com>

Mark.Miller wrote:
> Check how you're implementing the histogram function with respect to 
> that range statement.  It seems to make a difference, desirable or not.
> 
>  >>> import numpy
>  >>> numpy.__version__
> '1.0.4.dev3982'
>  >>> A = numpy.array([1, 2, 3, 4, 5, 6, 5, 4, 5, 4, 3, 2, 1])
>  >>> (x, y) = numpy.histogram(A, range(0, 7))
>  >>> x
> array([0, 2, 2, 2, 3, 3, 1])
>  >>> y
> [0, 1, 2, 3, 4, 5, 6]
>  >>>
>  >>> (x, y) = numpy.histogram(A, range=(0, 7))
>  >>> x
> array([0, 2, 2, 0, 2, 3, 0, 3, 1, 0])
>  >>> y
> array([ 0. ,  0.7,  1.4,  2.1,  2.8,  3.5,  4.2,  4.9,  5.6,  6.3])

Please check the signature of numpy.histogram(). The two aren't intended to be
the same. The range argument has nothing to do with the builtin range() function.

>  >>>
>  >>>
>  >>> (x, y) = numpy.histogram(A, range(7,0))
>  >>> x
> array([], dtype=int32)
>  >>> y
> []
>  >>>
> 
> Note that in the last case, the histogram function isn't returning 
> anything for a descending range.
> 
> Also notice that you're overwriting a python function with the way 
> you're assigning things....

No, he's not. "range" is a keyword argument to histogram(). He's using it correctly.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From mpmusu at cc.usu.edu  Fri Oct  5 18:14:47 2007
From: mpmusu at cc.usu.edu (Mark.Miller)
Date: Fri, 05 Oct 2007 16:14:47 -0600
Subject: [Numpy-discussion] histogram using decending range -- what do
 the results mean?
In-Reply-To: <4706B4FF.4040908@gmail.com>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>
	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>
	<80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>
	<e76aa17f0710050806p79241f3al27655e6240c94c9b@mail.gmail.com>
	<4706AA30.2080505@noaa.gov>
	<Pine.BSF.4.64.0710051724450.50575@earl-grey.cloud9.net>
	<4706B2AA.5090805@cc.usu.edu> <4706B4FF.4040908@gmail.com>
Message-ID: <4706B757.90700@cc.usu.edu>

My bad...I also note that I forgot to decrement the descending list in 
my example.  Ignore....

Robert Kern wrote:
> Mark.Miller wrote:
>> Check how you're implementing the histogram function with respect to 
>> that range statement.  It seems to make a difference, desirable or not.
>>
>>  >>> import numpy
>>  >>> numpy.__version__
>> '1.0.4.dev3982'
>>  >>> A = numpy.array([1, 2, 3, 4, 5, 6, 5, 4, 5, 4, 3, 2, 1])
>>  >>> (x, y) = numpy.histogram(A, range(0, 7))
>>  >>> x
>> array([0, 2, 2, 2, 3, 3, 1])
>>  >>> y
>> [0, 1, 2, 3, 4, 5, 6]
>>  >>>
>>  >>> (x, y) = numpy.histogram(A, range=(0, 7))
>>  >>> x
>> array([0, 2, 2, 0, 2, 3, 0, 3, 1, 0])
>>  >>> y
>> array([ 0. ,  0.7,  1.4,  2.1,  2.8,  3.5,  4.2,  4.9,  5.6,  6.3])
> 
> Please check the signature of numpy.histogram(). The two aren't intended to be
> the same. The range argument has nothing to do with the builtin range() function.
> 
>>  >>>
>>  >>>
>>  >>> (x, y) = numpy.histogram(A, range(7,0))
>>  >>> x
>> array([], dtype=int32)
>>  >>> y
>> []
>>  >>>
>>
>> Note that in the last case, the histogram function isn't returning 
>> anything for a descending range.
>>
>> Also notice that you're overwriting a python function with the way 
>> you're assigning things....
> 
> No, he's not. "range" is a keyword argument to histogram(). He's using it correctly.
> 


From Chris.Barker at noaa.gov  Fri Oct  5 19:01:12 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Fri, 05 Oct 2007 16:01:12 -0700
Subject: [Numpy-discussion] Python Magazine
Message-ID: <4706C238.60906@noaa.gov>

Hey folks,

There is a new Python Magazine out there:

http://www.pythonmagazine.com/

Let's get a numpy article in there!


NOTE: I have nothing to do with the production of the magazine, I'd just 
like to have it be a success so I can read it.

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From sdb at cloud9.net  Fri Oct  5 20:19:40 2007
From: sdb at cloud9.net (Stuart Brorson)
Date: Fri, 5 Oct 2007 20:19:40 -0400 (EDT)
Subject: [Numpy-discussion] histogram using decending range -- what do
 the results mean?
In-Reply-To: <4706AF70.2060208@gmail.com>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>
	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>
	<80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>
	<e76aa17f0710050806p79241f3al27655e6240c94c9b@mail.gmail.com>
	<4706AA30.2080505@noaa.gov>
	<Pine.BSF.4.64.0710051724450.50575@earl-grey.cloud9.net>
	<4706AF70.2060208@gmail.com>
Message-ID: <Pine.BSF.4.64.0710052016040.95074@earl-grey.cloud9.net>

Robert,

Thanks for your answers about histogram's meaning for range=(7, 0)!

>> *  If it truely isn't meaningful, why not catch the case and reject
>> input?  Maybe this is a bug....  ???
>
> Patches are welcome.

OK.  I don't know if you have a patch tracking system, so I'll just
post it here.  If you have a patch tracker, point me to it and I'll
enter the patch there.

----------------------  <patch>  -------------------------
Index: function_base.py
===================================================================
--- function_base.py    (revision 4155)
+++ function_base.py    (working copy)
@@ -108,6 +108,12 @@

      """
      a = asarray(a).ravel()
+
+    if (range is not None):
+        mn, mx = range
+        if (mn > mx):
+            raise AttributeError, 'max must be larger than min in range parameter.'
+
      if not iterable(bins):
          if range is None:
              range = (a.min(), a.max())
@@ -116,6 +122,9 @@
              mn -= 0.5
              mx += 0.5
          bins = linspace(mn, mx, bins, endpoint=False)
+    else:
+        if(any(bins[1:]-bins[:-1] < 0)):
+            raise AttributeError, 'bins must increase monotonically.'

      # best block size probably depends on processor cache size
      block = 65536

-----------------------  </patch>  -------------------------

And here's what it does:

-----------------------  <session log>  ----------------------
/home/sdb> python
Python 2.5 (r25:51908, Feb 19 2007, 04:41:03)
[GCC 3.3.5 20050117 (prerelease) (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> A = numpy.array([1, 2, 3, 4, 5, 6, 5, 4, 5, 4, 3, 2, 1])
>>>
>>> (x, y) = numpy.histogram(A, range=(0, 7))
>>> x
array([0, 2, 2, 0, 2, 3, 0, 3, 1, 0])
>>>
>>> (x, y) = numpy.histogram(A, range=(7, 0))
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.5/site-packages/numpy/lib/function_base.py", line 115, in histogram
     raise AttributeError, 'max must be larger than min in range parameter.'
AttributeError: max must be larger than min in range parameter.
>>>
>>>
>>> bins = numpy.arange(0, 7)
>>> (x, y) = numpy.histogram(A, bins=bins)
>>>
>>> bins = bins[::-1]
>>> bins
array([6, 5, 4, 3, 2, 1, 0])
>>> (x, y) = numpy.histogram(A, bins=bins)
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.5/site-packages/numpy/lib/function_base.py", line 127, in histogram
     raise AttributeError, 'bins must increase monotonically.'
AttributeError: bins must increase monotonically.
>>>
>>>
-------------------------  </session log>  --------------------------

Cheers,

Stuart



From robert.kern at gmail.com  Fri Oct  5 20:22:34 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 05 Oct 2007 19:22:34 -0500
Subject: [Numpy-discussion] histogram using decending range -- what do
 the results mean?
In-Reply-To: <Pine.BSF.4.64.0710052016040.95074@earl-grey.cloud9.net>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>	<80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>	<e76aa17f0710050806p79241f3al27655e6240c94c9b@mail.gmail.com>	<4706AA30.2080505@noaa.gov>	<Pine.BSF.4.64.0710051724450.50575@earl-grey.cloud9.net>	<4706AF70.2060208@gmail.com>
	<Pine.BSF.4.64.0710052016040.95074@earl-grey.cloud9.net>
Message-ID: <4706D54A.50201@gmail.com>

Stuart Brorson wrote:
> Robert,
> 
> Thanks for your answers about histogram's meaning for range=(7, 0)!
> 
>>> *  If it truely isn't meaningful, why not catch the case and reject
>>> input?  Maybe this is a bug....  ???
>> Patches are welcome.
> 
> OK.  I don't know if you have a patch tracking system, so I'll just
> post it here.  If you have a patch tracker, point me to it and I'll
> enter the patch there.

http://projects.scipy.org/scipy/numpy

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From sdb at cloud9.net  Fri Oct  5 20:35:06 2007
From: sdb at cloud9.net (Stuart Brorson)
Date: Fri, 5 Oct 2007 20:35:06 -0400 (EDT)
Subject: [Numpy-discussion] histogram using decending range -- what do
 the results mean?
In-Reply-To: <4706D54A.50201@gmail.com>
References: <e76aa17f0710050151s60d933d2gc9fafb658dfe0905@mail.gmail.com>
	<80c99e790710050634i154a7eeaj6574e439024dfa60@mail.gmail.com>
	<e76aa17f0710050719t4dd90cb6h37e11b2fa4264a8a@mail.gmail.com>
	<80c99e790710050753u451022c3m7f78b1eff16b5194@mail.gmail.com>
	<e76aa17f0710050806p79241f3al27655e6240c94c9b@mail.gmail.com>
	<4706AA30.2080505@noaa.gov>
	<Pine.BSF.4.64.0710051724450.50575@earl-grey.cloud9.net>
	<4706AF70.2060208@gmail.com>
	<Pine.BSF.4.64.0710052016040.95074@earl-grey.cloud9.net>
	<4706D54A.50201@gmail.com>
Message-ID: <Pine.BSF.4.64.0710052027240.96702@earl-grey.cloud9.net>

>> OK.  I don't know if you have a patch tracking system, so I'll just
>> post it here.  If you have a patch tracker, point me to it and I'll
>> enter the patch there.
>
> http://projects.scipy.org/scipy/numpy

OK, entered as ticket #586.

Cheers,

Stuart


From ramercer at gmail.com  Sat Oct  6 23:35:06 2007
From: ramercer at gmail.com (Adam Mercer)
Date: Sat, 6 Oct 2007 23:35:06 -0400
Subject: [Numpy-discussion] Convert array type
Message-ID: <799406d60710062035o544a0530u83e8a3586e71185a@mail.gmail.com>

Hi

I am fairly new to using numpy and am running into a problem regarding
the type of an array.  The array in question is created using the
following code:

values = array(wavearray.split())

where wavearray is a string containing a series of floats separated by
white space, it appears that the individual elements of the values
array are strings, where I need them to be floats.  How can I ensure
that the individual elements of the values array are floats, not
strings?

I have tried using

values = array(wavearray.split(), dtype=float)

but that results in the error:

ValueError: setting an array element with a sequence.

Cheers

Adam


From gruben at bigpond.net.au  Sun Oct  7 00:37:42 2007
From: gruben at bigpond.net.au (Gary Ruben)
Date: Sun, 07 Oct 2007 14:37:42 +1000
Subject: [Numpy-discussion] Convert array type
In-Reply-To: <799406d60710062035o544a0530u83e8a3586e71185a@mail.gmail.com>
References: <799406d60710062035o544a0530u83e8a3586e71185a@mail.gmail.com>
Message-ID: <47086296.4060303@bigpond.net.au>

Try using astype. This works:

values = array(wavearray.split()).astype(float)

Gary R.

Adam Mercer wrote:
> values = array(wavearray.split())
> 
> where wavearray is a string containing a series of floats separated by
> white space, it appears that the individual elements of the values
> array are strings, where I need them to be floats.  How can I ensure
> that the individual elements of the values array are floats, not
> strings?
> 
> I have tried using
> 
> values = array(wavearray.split(), dtype=float)


From ramercer at gmail.com  Sun Oct  7 01:02:43 2007
From: ramercer at gmail.com (Adam Mercer)
Date: Sun, 7 Oct 2007 01:02:43 -0400
Subject: [Numpy-discussion] Convert array type
In-Reply-To: <47086296.4060303@bigpond.net.au>
References: <799406d60710062035o544a0530u83e8a3586e71185a@mail.gmail.com>
	<47086296.4060303@bigpond.net.au>
Message-ID: <799406d60710062202g684708e8ra04ecf510d798cf1@mail.gmail.com>

On 07/10/2007, Gary Ruben <gruben at bigpond.net.au> wrote:

> Try using astype. This works:
>
> values = array(wavearray.split()).astype(float)

Thanks Gary, that does the trick.

Cheers

Adam


From ndbecker2 at gmail.com  Sun Oct  7 06:52:11 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sun, 07 Oct 2007 06:52:11 -0400
Subject: [Numpy-discussion] Iterate over all 1-dim views
Message-ID: <feadpn$mkq$1@sea.gmane.org>

Suppose I have a function F(), which is defined for 1-dim arguments.  If the
user passes an n>1 dim array, I want to apply F to each 1-dim view.

For example, for a 2-d array, apply F to each row and return a 2-d result.

For a 3-d array, select each 2-d subarray and see above.  Return 3-d result.

Any suggestions on how to code something like this in numpy?



From ptvirtan at elisanet.fi  Sun Oct  7 08:40:46 2007
From: ptvirtan at elisanet.fi (Pauli Virtanen)
Date: Sun, 7 Oct 2007 15:40:46 +0300 (EEST)
Subject: [Numpy-discussion] Iterate over all 1-dim views
Message-ID: <30605585.421011191760846823.JavaMail.ptvirtan@elisanet.fi>

Neal Becker <ndbecker2 at gmail.com> kirjoitti: 
> Suppose I have a function F(), which is defined for 1-dim arguments.  If the
> user passes an n>1 dim array, I want to apply F to each 1-dim view.
> 
> For example, for a 2-d array, apply F to each row and return a 2-d result.
> 
> For a 3-d array, select each 2-d subarray and see above.  Return 3-d result.
> 
> Any suggestions on how to code something like this in numpy?

You may be looking for numpy.apply_along_axis:

>>> from numpy import *
>>> x=arange(2*3*4); x.shape=(2,3,4)
>>> x
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11]],

       [[12, 13, 14, 15],
        [16, 17, 18, 19],
        [20, 21, 22, 23]]])
>>> apply_along_axis(cumsum, 0, x)
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11]],

       [[12, 14, 16, 18],
        [20, 22, 24, 26],
        [28, 30, 32, 34]]])
>>> apply_along_axis(cumsum, 1, x)
array([[[ 0,  1,  2,  3],
        [ 4,  6,  8, 10],
        [12, 15, 18, 21]],

       [[12, 13, 14, 15],
        [28, 30, 32, 34],
        [48, 51, 54, 57]]])
>>> apply_along_axis(cumsum, 2, x)
array([[[ 0,  1,  3,  6],
        [ 4,  9, 15, 22],
        [ 8, 17, 27, 38]],

       [[12, 25, 39, 54],
        [16, 33, 51, 70],
        [20, 41, 63, 86]]])


-- 




From gael.varoquaux at normalesup.org  Sun Oct  7 11:55:25 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Sun, 7 Oct 2007 17:55:25 +0200
Subject: [Numpy-discussion] Iterate over all 1-dim views
In-Reply-To: <feadpn$mkq$1@sea.gmane.org>
References: <feadpn$mkq$1@sea.gmane.org>
Message-ID: <20071007155525.GC24237@clipper.ens.fr>

On Sun, Oct 07, 2007 at 06:52:11AM -0400, Neal Becker wrote:
> Suppose I have a function F(), which is defined for 1-dim arguments.  If the
> user passes an n>1 dim array, I want to apply F to each 1-dim view.

> For example, for a 2-d array, apply F to each row and return a 2-d result.

> For a 3-d array, select each 2-d subarray and see above.  Return 3-d result.

> Any suggestions on how to code something like this in numpy?

Code your function so that it works well for 2D arrays (using axis=-1 and
co), then use a decorator on it so that if you pass it an N-d array, it
transforms it in a 2D array, passes it to the decorator, then transforms
the output back to the right shape.

The idea is quite theoretical, and I have never gotten to implement it,
because when I was facing similar problems, it didn't come to my mind,
but I think it can work in a very general way.

Ga?l


From ndbecker2 at gmail.com  Sun Oct  7 15:51:15 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sun, 07 Oct 2007 15:51:15 -0400
Subject: [Numpy-discussion] Interested in adding new data type
Message-ID: <febdcn$gt0$1@sea.gmane.org>

I'm interested in experimenting with adding complex<int> data type.  I
have "Guide to Numpy".  I'm looking at section 15.3. It looks like the
first thing is a PyArray_Desc.  There doesn't seem to be much info on what
needs to go in this.  Does anyone have any examples I could look at?




From rosset at lal.in2p3.fr  Sun Oct  7 19:49:44 2007
From: rosset at lal.in2p3.fr (Cyrille Rosset)
Date: Mon, 08 Oct 2007 01:49:44 +0200
Subject: [Numpy-discussion] Memory leak in vectorize ?
Message-ID: <47097098.3090101@lal.in2p3.fr>

Hello,
the following code seems to create a memory leak in Python.
(around 230 MB).
Any ideas what's wrong ?
I'm using python 2.5 and numpy 1.0.3
-------------------
def toto(x):
     return x**2

tutu=vectorize(toto)

Nbins=10000
for i in xrange(1000):
     c=tutu(arange(Nbins))
-------------------

Thanks,
Cyrille.


From aisaac at american.edu  Sun Oct  7 22:38:59 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Sun, 7 Oct 2007 22:38:59 -0400
Subject: [Numpy-discussion] Python Magazine
In-Reply-To: <4706C238.60906@noaa.gov>
References: <4706C238.60906@noaa.gov>
Message-ID: <Mahogany-0.67.0-632-20071007-223859.00@american.edu>

On Fri, 05 Oct 2007, Christopher Barker apparently wrote:
> There is a new Python Magazine out there: 
> http://www.pythonmagazine.com/ 

Looks useful.
If you think so too, make sure you library subscribes.

Cheers,
Alan Isaac




From stefan at sun.ac.za  Mon Oct  8 06:40:20 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Mon, 8 Oct 2007 12:40:20 +0200
Subject: [Numpy-discussion] Memory leak in vectorize ?
In-Reply-To: <47097098.3090101@lal.in2p3.fr>
References: <47097098.3090101@lal.in2p3.fr>
Message-ID: <20071008104020.GN9063@mentat.za.net>

Hi Cryille

On Mon, Oct 08, 2007 at 01:49:44AM +0200, Cyrille Rosset wrote:
> Hello,
> the following code seems to create a memory leak in Python.
> (around 230 MB).
> Any ideas what's wrong ?

The relevant ticket is here:

http://projects.scipy.org/scipy/numpy/ticket/570

Regards
St?fan


From stefan at sun.ac.za  Mon Oct  8 06:52:19 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Mon, 8 Oct 2007 12:52:19 +0200
Subject: [Numpy-discussion] Iterate over all 1-dim views
In-Reply-To: <feadpn$mkq$1@sea.gmane.org>
References: <feadpn$mkq$1@sea.gmane.org>
Message-ID: <20071008105219.GO9063@mentat.za.net>

On Sun, Oct 07, 2007 at 06:52:11AM -0400, Neal Becker wrote:
> Suppose I have a function F(), which is defined for 1-dim arguments.  If the
> user passes an n>1 dim array, I want to apply F to each 1-dim view.
> 
> For example, for a 2-d array, apply F to each row and return a 2-d result.
> 
> For a 3-d array, select each 2-d subarray and see above.  Return 3-d result.
> 
> Any suggestions on how to code something like this in numpy?

Not the most efficient way, but easy to read and understand:

import numpy as N

def func(a):
    return a.shape

z = N.zeros((2,2,2,2))
print N.array([func(sub) for sub in z])

Regards
St?fan


From robince at gmail.com  Mon Oct  8 12:32:43 2007
From: robince at gmail.com (Robin)
Date: Mon, 8 Oct 2007 17:32:43 +0100
Subject: [Numpy-discussion] dot product not behaving as expected
Message-ID: <b14b627d0710080932t74ec1607l39e8a7b72c1b2768@mail.gmail.com>

Hi,

I have a problem using numpy.dot, see below:

In [151]: m=5

In [152]: n=5

In [153]: x=(m*ones((1,5)))**arange(0,n)

In [154]: y=test.order_length[::-1]

In [155]: x
Out[155]: array([[   1.,    5.,   25.,  125.,  625.]])

In [156]: y
Out[156]:
array([[ 1024.],
       [ 1280.],
       [  640.],
       [  160.],
       [   20.]])

In [157]: numpy.dot(x,y)
Out[157]: array([[ 640000.]])

In [158]: sum(x*y.T)
Out[158]: 55924.0

Am I missing something?

Thanks,

Robin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071008/124f470c/attachment.html>

From robert.kern at gmail.com  Mon Oct  8 12:40:27 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 08 Oct 2007 11:40:27 -0500
Subject: [Numpy-discussion] dot product not behaving as expected
In-Reply-To: <b14b627d0710080932t74ec1607l39e8a7b72c1b2768@mail.gmail.com>
References: <b14b627d0710080932t74ec1607l39e8a7b72c1b2768@mail.gmail.com>
Message-ID: <470A5D7B.5050905@gmail.com>

Robin wrote:
> Hi,
> 
> I have a problem using numpy.dot, see below:
> 
> In [151]: m=5
> 
> In [152]: n=5
> 
> In [153]: x=(m*ones((1,5)))**arange(0,n)
> 
> In [154]: y=test.order_length[::-1]
> 
> In [155]: x
> Out[155]: array([[   1.,    5.,   25.,  125.,  625.]])
> 
> In [156]: y
> Out[156]:
> array([[ 1024.],
>        [ 1280.],
>        [  640.],
>        [  160.],
>        [   20.]])
> 
> In [157]: numpy.dot(x,y)
> Out[157]: array([[ 640000.]])
> 
> In [158]: sum(x* y.T)
> Out[158]: 55924.0
> 
> Am I missing something?

In [1]: from numpy import *

In [2]: x = array([[   1.,    5.,   25.,  125.,  625.]])

In [3]: y = array([[ 1024.],
   ...:        [ 1280.],
   ...:        [  640.],
   ...:        [  160.],
   ...:        [   20.]])

In [4]: dot(x, y)
Out[4]: array([[ 55924.]])

In [5]: sum(x * y.T)
Out[5]: 55924.0


It works for me with a recent SVN numpy on OS X. What version of numpy are you
using? What platform are you on? Did you build with ATLAS or other optimized
linear algebra library?

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robince at gmail.com  Mon Oct  8 12:57:37 2007
From: robince at gmail.com (Robin)
Date: Mon, 8 Oct 2007 17:57:37 +0100
Subject: [Numpy-discussion] dot product not behaving as expected
In-Reply-To: <470A5D7B.5050905@gmail.com>
References: <b14b627d0710080932t74ec1607l39e8a7b72c1b2768@mail.gmail.com>
	<470A5D7B.5050905@gmail.com>
Message-ID: <b14b627d0710080957s4f0688c5je073ed617ec9777e@mail.gmail.com>

>
> It works for me with a recent SVN numpy on OS X. What version of numpy are
> you
> using? What platform are you on? Did you build with ATLAS or other
> optimized
> linear algebra library?


I am on Ubuntu 7.04, gcc 4.1.2

In [181]: numpy.__version__
Out[181]: '1.0.4.dev4155'

Built with ATLAS 3.7.37, Lapack 3.1.1

It also occurs for me on Windows XP SP2 with 1.0.4dev4154 built with cygwin
gcc 3.4.4 and atlas 3.7.37 and lapack 3.1.1.

Actually I found that for me it also doesn't occur if I type them in like
that. The problem comes when y is reverse with the [::-1] index. Is this the
correct way to reverse a vector?

Anyway try this:

In [36]: x=array([[1.,5.,25.,125.,625.]])

In [37]: y=array([[20.],[160.],[640.],[1280.],[1024.]])

In [38]: y
Out[38]:
array([[   20.],
       [  160.],
       [  640.],
       [ 1280.],
       [ 1024.]])

In [39]: y2=y[::-1]

In [40]: y2
Out[40]:
array([[ 1024.],
       [ 1280.],
       [  640.],
       [  160.],
       [   20.]])

In [41]: dot(x,y)
Out[41]: array([[ 816820.]])

In [42]: dot(x,y2)
Out[42]: array([[ 640000.]])

Thanks,

Robin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071008/c6397763/attachment.html>

From robert.kern at gmail.com  Mon Oct  8 13:03:01 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 08 Oct 2007 12:03:01 -0500
Subject: [Numpy-discussion] dot product not behaving as expected
In-Reply-To: <b14b627d0710080957s4f0688c5je073ed617ec9777e@mail.gmail.com>
References: <b14b627d0710080932t74ec1607l39e8a7b72c1b2768@mail.gmail.com>	<470A5D7B.5050905@gmail.com>
	<b14b627d0710080957s4f0688c5je073ed617ec9777e@mail.gmail.com>
Message-ID: <470A62C5.6010507@gmail.com>

Robin wrote:
>     It works for me with a recent SVN numpy on OS X. What version of
>     numpy are you
>     using? What platform are you on? Did you build with ATLAS or other
>     optimized
>     linear algebra library?
> 
> 
> I am on Ubuntu 7.04, gcc 4.1.2
> 
> In [181]: numpy.__version__
> Out[181]: '1.0.4.dev4155'
> 
> Built with ATLAS 3.7.37, Lapack 3.1.1
> 
> It also occurs for me on Windows XP SP2 with 1.0.4dev4154 built with
> cygwin gcc 3.4.4 and atlas 3.7.37 and lapack 3.1.1.
> 
> Actually I found that for me it also doesn't occur if I type them in
> like that. The problem comes when y is reverse with the [::-1] index. Is
> this the correct way to reverse a vector?

Aha. Yes, that is the correct way to do it. However, it appears that there is a
bug in how dot() is interpreting the non-contiguous array. In the meantime, you
can make a contiguous array with array(y2).

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From robince at gmail.com  Mon Oct  8 13:24:32 2007
From: robince at gmail.com (Robin)
Date: Mon, 8 Oct 2007 18:24:32 +0100
Subject: [Numpy-discussion] dot product not behaving as expected
In-Reply-To: <470A62C5.6010507@gmail.com>
References: <b14b627d0710080932t74ec1607l39e8a7b72c1b2768@mail.gmail.com>
	<470A5D7B.5050905@gmail.com>
	<b14b627d0710080957s4f0688c5je073ed617ec9777e@mail.gmail.com>
	<470A62C5.6010507@gmail.com>
Message-ID: <b14b627d0710081024w175159afj6a02d92b0163ed8b@mail.gmail.com>

On 10/8/07, Robert Kern <robert.kern at gmail.com> wrote:
>
> Aha. Yes, that is the correct way to do it. However, it appears that there
> is a
> bug in how dot() is interpreting the non-contiguous array. In the
> meantime, you
> can make a contiguous array with array(y2).


Thanks very much for the prompt help!

Should I open a ticket for this issue?

Robin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071008/e98750a4/attachment.html>

From robert.kern at gmail.com  Mon Oct  8 13:26:10 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 08 Oct 2007 12:26:10 -0500
Subject: [Numpy-discussion] dot product not behaving as expected
In-Reply-To: <b14b627d0710081024w175159afj6a02d92b0163ed8b@mail.gmail.com>
References: <b14b627d0710080932t74ec1607l39e8a7b72c1b2768@mail.gmail.com>	<470A5D7B.5050905@gmail.com>	<b14b627d0710080957s4f0688c5je073ed617ec9777e@mail.gmail.com>	<470A62C5.6010507@gmail.com>
	<b14b627d0710081024w175159afj6a02d92b0163ed8b@mail.gmail.com>
Message-ID: <470A6832.8000507@gmail.com>

Robin wrote:
> 
> On 10/8/07, *Robert Kern* <robert.kern at gmail.com
> <mailto:robert.kern at gmail.com>> wrote:
> 
>     Aha. Yes, that is the correct way to do it. However, it appears that
>     there is a
>     bug in how dot() is interpreting the non-contiguous array. In the
>     meantime, you
>     can make a contiguous array with array(y2).
> 
> 
> Thanks very much for the prompt help!
> 
> Should I open a ticket for this issue?

Please!

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From rmay at ou.edu  Mon Oct  8 13:43:00 2007
From: rmay at ou.edu (Ryan May)
Date: Mon, 08 Oct 2007 12:43:00 -0500
Subject: [Numpy-discussion] Convert array type
In-Reply-To: <47086296.4060303@bigpond.net.au>
References: <799406d60710062035o544a0530u83e8a3586e71185a@mail.gmail.com>
	<47086296.4060303@bigpond.net.au>
Message-ID: <470A6C24.3010209@ou.edu>

Gary Ruben wrote:
> Try using astype. This works:
> 
> values = array(wavearray.split()).astype(float)
> 

Why not use numpy.fromstring?

    fromstring(string, dtype=float, count=-1, sep='')

    Return a new 1d array initialized from the raw binary data in string.

    If count is positive, the new array will have count elements,
otherwise its
    size is determined by the size of string.  If sep is not empty then the
    string is interpreted in ASCII mode and converted to the desired
number type
    using sep as the separator between elements (extra whitespace is
ignored).

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma


From subscriber100 at rjs.org  Mon Oct  8 15:15:52 2007
From: subscriber100 at rjs.org (Ray S)
Date: Mon, 08 Oct 2007 12:15:52 -0700
Subject: [Numpy-discussion] numpy/Windows shared arrays between processes?
Message-ID: <6.2.3.4.2.20071008105940.09e5fed8@blue-cove.com>

Is anyone sharing arrays between processes on Windows?
I tried compiling the posh sources (once, so far) with the new MS 
toolkit and failed...
What other solutions are in use?

Have a second process create an array view from an address would 
suffice for this particular purpose. I could pass the address as a 
parameter of the second process's argv.

I've also tried things like
pb=pythonapi.PyBuffer_FromReadWriteMemory(9508824, 9*sizeof(c_int))
N.frombuffer(pb, N.int32)
which fails since pb is and int. What are my options?

Ray Schumacher



From ramercer at gmail.com  Mon Oct  8 15:36:39 2007
From: ramercer at gmail.com (Adam Mercer)
Date: Mon, 8 Oct 2007 15:36:39 -0400
Subject: [Numpy-discussion] Convert array type
In-Reply-To: <470A6C24.3010209@ou.edu>
References: <799406d60710062035o544a0530u83e8a3586e71185a@mail.gmail.com>
	<47086296.4060303@bigpond.net.au> <470A6C24.3010209@ou.edu>
Message-ID: <799406d60710081236x7f7a23cata33cb14c4fdbeaf3@mail.gmail.com>

On 08/10/2007, Ryan May <rmay at ou.edu> wrote:

> Why not use numpy.fromstring?

because that results in the array being filled with gibberish

values = numpy.fromstring(wavearray, dtype=float, count=-1, sep='')
print values

gives:

[  1.39804329e-076   1.30354290e-076   1.18295070e-076 ...,
   5.45168074e-067   2.11101912e-052   6.58519056e-260]

where using

values = array(wavearray.split()).astype(float)
print values

results in the correct

[  0.00000000e+00   0.00000000e+00   0.00000000e+00 ...,
   4.22233200e-23   3.86799900e-23   3.48452000e-23]

Cheers

Adam


From robert.kern at gmail.com  Mon Oct  8 15:51:48 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 08 Oct 2007 14:51:48 -0500
Subject: [Numpy-discussion] Convert array type
In-Reply-To: <799406d60710081236x7f7a23cata33cb14c4fdbeaf3@mail.gmail.com>
References: <799406d60710062035o544a0530u83e8a3586e71185a@mail.gmail.com>	<47086296.4060303@bigpond.net.au>
	<470A6C24.3010209@ou.edu>
	<799406d60710081236x7f7a23cata33cb14c4fdbeaf3@mail.gmail.com>
Message-ID: <470A8A54.8080804@gmail.com>

Adam Mercer wrote:
> On 08/10/2007, Ryan May <rmay at ou.edu> wrote:
> 
>> Why not use numpy.fromstring?
> 
> because that results in the array being filled with gibberish
> 
> values = numpy.fromstring(wavearray, dtype=float, count=-1, sep='')

Use sep=' '. As the docstring says, if sep is empty, then the string is
interpreted as binary data. If it is not empty, then the string is interpreted
as ASCII.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Mon Oct  8 15:52:46 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Mon, 8 Oct 2007 13:52:46 -0600
Subject: [Numpy-discussion] Convert array type
In-Reply-To: <799406d60710081236x7f7a23cata33cb14c4fdbeaf3@mail.gmail.com>
References: <799406d60710062035o544a0530u83e8a3586e71185a@mail.gmail.com>
	<47086296.4060303@bigpond.net.au> <470A6C24.3010209@ou.edu>
	<799406d60710081236x7f7a23cata33cb14c4fdbeaf3@mail.gmail.com>
Message-ID: <e06186140710081252m3c893571jb87aaa5e856bfd@mail.gmail.com>

On 10/8/07, Adam Mercer <ramercer at gmail.com> wrote:
>
> On 08/10/2007, Ryan May <rmay at ou.edu> wrote:
>
> > Why not use numpy.fromstring?
>
> because that results in the array being filled with gibberish
>
> values = numpy.fromstring(wavearray, dtype=float, count=-1, sep='')
> print values


You need to use

values = numpy.fromstring(wavearray, dtype=float, count=-1, sep=' ')

Note that sep is a blank (whitespace). This will convert ascii strings.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071008/8d40f21c/attachment.html>

From robince at gmail.com  Mon Oct  8 18:00:39 2007
From: robince at gmail.com (Robin)
Date: Mon, 8 Oct 2007 23:00:39 +0100
Subject: [Numpy-discussion] beginner question: rank-1 arrays
Message-ID: <b14b627d0710081500q7fa640ds969d2c7d4bbe8232@mail.gmail.com>

Hi,

I am trying to implement a project in scipy. I think I am getting somewhere
finally.

However in my code (I am converting from MATLAB) it is important to maintain
2d arrays, and keep the difference between row and column vectors. After
working through some initial problems I think I am getting more of a picture
of how things work in numpy.

However I am finding my code littered with things like:
np.array(np.r_[0:nterms],ndmin=2)    (for a row vector)
or
np.array(np.r_[0:nterms],ndmin=2).T (for a column vector)

Coming from matlab and being use to 0:10 for row or (0:10)' for column this
seems a bit messy. Is there a better way of constructing row/column 2d
arrays from a slice type range?

Thanks,

Robin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071008/af097fa5/attachment.html>

From gael.varoquaux at normalesup.org  Mon Oct  8 18:07:09 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 9 Oct 2007 00:07:09 +0200
Subject: [Numpy-discussion] beginner question: rank-1 arrays
In-Reply-To: <b14b627d0710081500q7fa640ds969d2c7d4bbe8232@mail.gmail.com>
References: <b14b627d0710081500q7fa640ds969d2c7d4bbe8232@mail.gmail.com>
Message-ID: <20071008220709.GC7767@clipper.ens.fr>

On Mon, Oct 08, 2007 at 11:00:39PM +0100, Robin wrote:
>    Coming from matlab and being use to 0:10 for row or (0:10)' for column
>    this seems a bit messy. Is there a better way of constructing row/column
>    2d arrays from a slice type range?

r_[0:10] and c_[0:10].

Does that suit you ? The first one is indeed only 1D, but I don't see the
problem with that. If you really want 2D you can use c_[0:10].T .

Ga?l


From robince at gmail.com  Mon Oct  8 18:12:07 2007
From: robince at gmail.com (Robin)
Date: Mon, 8 Oct 2007 23:12:07 +0100
Subject: [Numpy-discussion] beginner question: rank-1 arrays
In-Reply-To: <20071008220709.GC7767@clipper.ens.fr>
References: <b14b627d0710081500q7fa640ds969d2c7d4bbe8232@mail.gmail.com>
	<20071008220709.GC7767@clipper.ens.fr>
Message-ID: <b14b627d0710081512q46ae7d15n68cfc1eb9e0c8eeb@mail.gmail.com>

On 10/8/07, Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:
>
> r_[0:10] and c_[0:10].
>
> Does that suit you ? The first one is indeed only 1D, but I don't see the
> problem with that. If you really want 2D you can use c_[0:10].T .


Thanks, but not really :)

Firstly - for me I don' see any difference between the two, they both give a
numpy rank-1 array which doesn't have a row/column characteristic.
x.shape for both of the above is (10,) I need (10,1) or (1,10) for my code.

Robin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071008/6476f412/attachment.html>

From stefan at sun.ac.za  Mon Oct  8 18:15:46 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Tue, 9 Oct 2007 00:15:46 +0200
Subject: [Numpy-discussion] beginner question: rank-1 arrays
In-Reply-To: <b14b627d0710081500q7fa640ds969d2c7d4bbe8232@mail.gmail.com>
References: <b14b627d0710081500q7fa640ds969d2c7d4bbe8232@mail.gmail.com>
Message-ID: <20071008221546.GA21653@mentat.za.net>

On Mon, Oct 08, 2007 at 11:00:39PM +0100, Robin wrote:
> Hi,
> 
> I am trying to implement a project in scipy. I think I am getting somewhere
> finally.
> 
> However in my code (I am converting from MATLAB) it is important to maintain 2d
> arrays, and keep the difference between row and column vectors. After working
> through some initial problems I think I am getting more of a picture of how
> things work in numpy.
> 
> However I am finding my code littered with things like:
> np.array(np.r_[0:nterms],ndmin=2)    (for a row vector)
> or
> np.array(np.r_[0:nterms],ndmin=2).T (for a column vector)

You can use

N.arange(10).reshape(-1,1)

or

N.c_[:10,]

But if you use this sort of thing often, just write your own factory
method:

def col(n):
    return N.arange(n).reshape(-1,1)

Cheers
St?fan


From gael.varoquaux at normalesup.org  Mon Oct  8 18:18:48 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Tue, 9 Oct 2007 00:18:48 +0200
Subject: [Numpy-discussion] beginner question: rank-1 arrays
In-Reply-To: <b14b627d0710081512q46ae7d15n68cfc1eb9e0c8eeb@mail.gmail.com>
References: <b14b627d0710081500q7fa640ds969d2c7d4bbe8232@mail.gmail.com>
	<20071008220709.GC7767@clipper.ens.fr>
	<b14b627d0710081512q46ae7d15n68cfc1eb9e0c8eeb@mail.gmail.com>
Message-ID: <20071008221848.GD7767@clipper.ens.fr>

On Mon, Oct 08, 2007 at 11:12:07PM +0100, Robin wrote:
>    On 10/8/07, Gael Varoquaux <[1]gael.varoquaux at normalesup.org> wrote:

>      r_[0:10] and c_[0:10].

>      Does that suit you ? The first one is indeed only 1D, but I don't see
>      the
>      problem with that. If you really want 2D you can use c_[0:10].T .

>    Thanks, but not really :)

>    Firstly - for me I don' see any difference between the two, they both give
>    a numpy rank-1 array which doesn't have a row/column characteristic.
>    x.shape for both of the above is (10,) I need (10,1) or (1,10) for my
>    code.

Damn it. Shame on me. I meant c_[0:10,]. If you really need a shape of
(1,10) (I have never had such a need) you can use c_[0:10,].T.

HTH,

Ga?l


From aisaac at american.edu  Mon Oct  8 18:28:15 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Mon, 8 Oct 2007 18:28:15 -0400
Subject: [Numpy-discussion] beginner question: rank-1 arrays
In-Reply-To: <b14b627d0710081500q7fa640ds969d2c7d4bbe8232@mail.gmail.com>
References: <b14b627d0710081500q7fa640ds969d2c7d4bbe8232@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-1716-20071008-182815.00@american.edu>

On Mon, 8 Oct 2007, Robin apparently wrote:
> However in my code (I am converting from MATLAB) it is 
> important to maintain 2d arrays, and keep the difference 
> between row and column vectors.

How about using matrices?
help(numpy.mat)

hth,
Alan Isaac





From robince at gmail.com  Mon Oct  8 18:24:10 2007
From: robince at gmail.com (Robin)
Date: Mon, 8 Oct 2007 23:24:10 +0100
Subject: [Numpy-discussion] beginner question: rank-1 arrays
In-Reply-To: <20071008221848.GD7767@clipper.ens.fr>
References: <b14b627d0710081500q7fa640ds969d2c7d4bbe8232@mail.gmail.com>
	<20071008220709.GC7767@clipper.ens.fr>
	<b14b627d0710081512q46ae7d15n68cfc1eb9e0c8eeb@mail.gmail.com>
	<20071008221848.GD7767@clipper.ens.fr>
Message-ID: <b14b627d0710081524k592d3aey2d0a2c966921f1c2@mail.gmail.com>

On 10/8/07, Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:

> Damn it. Shame on me. I meant c_[0:10,]. If you really need a shape of
> (1,10) (I have never had such a need) you can use c_[0:10,].T.


Thanks! - the trick with the , is just the sort of thing I was looking for -
I knew there must be an easy way...

Cheers

Robin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071008/e7c9095a/attachment.html>

From ramercer at gmail.com  Mon Oct  8 18:51:24 2007
From: ramercer at gmail.com (Adam Mercer)
Date: Mon, 8 Oct 2007 18:51:24 -0400
Subject: [Numpy-discussion] Convert array type
In-Reply-To: <470A8A54.8080804@gmail.com>
References: <799406d60710062035o544a0530u83e8a3586e71185a@mail.gmail.com>
	<47086296.4060303@bigpond.net.au> <470A6C24.3010209@ou.edu>
	<799406d60710081236x7f7a23cata33cb14c4fdbeaf3@mail.gmail.com>
	<470A8A54.8080804@gmail.com>
Message-ID: <799406d60710081551v1e6429cft850c05343305d4ac@mail.gmail.com>

On 08/10/2007, Robert Kern <robert.kern at gmail.com> wrote:

> Use sep=' '. As the docstring says, if sep is empty, then the string is
> interpreted as binary data. If it is not empty, then the string is interpreted
> as ASCII.

Thanks, got it the wrong way round. That works now.

Cheers

Adam


From david at ar.media.kyoto-u.ac.jp  Tue Oct  9 00:00:15 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 09 Oct 2007 13:00:15 +0900
Subject: [Numpy-discussion] numpy/Windows shared arrays between
	processes?
In-Reply-To: <6.2.3.4.2.20071008105940.09e5fed8@blue-cove.com>
References: <6.2.3.4.2.20071008105940.09e5fed8@blue-cove.com>
Message-ID: <470AFCCF.4000300@ar.media.kyoto-u.ac.jp>

Ray S wrote:
> Is anyone sharing arrays between processes on Windows?
> I tried compiling the posh sources (once, so far) with the new MS 
> toolkit and failed...
> What other solutions are in use?
>
> Have a second process create an array view from an address would 
> suffice for this particular purpose. I could pass the address as a 
> parameter of the second process's argv.
>
> I've also tried things like
> pb=pythonapi.PyBuffer_FromReadWriteMemory(9508824, 9*sizeof(c_int))
> N.frombuffer(pb, N.int32)
> which fails since pb is and int. What are my options?
>
(disclaimer: I know nothing about windows idiosyncraties)

Could not this be because you compiled the posh sources with a 
compiler/runtime which is different than the other extensions and python 
interpreter ? I don't know the details, but since most of the posix 
functions related to files and processes are broken beyond despair in 
windows, and in particular, many posix handles cannot cross dll 
boundaries compiled by different compilers, I would not be surprised if 
this cause some trouble.

The fact that POSH is said to be posix-only on python.org 
(http://wiki.python.org/moin/ParallelProcessing) would imply that people 
do not care much about windows, too (but again, this is just from 
reading what posh is about; I have never used it personnally).

cheers,

David



From haase at msg.ucsf.edu  Tue Oct  9 03:19:29 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 9 Oct 2007 09:19:29 +0200
Subject: [Numpy-discussion] numpy/Windows shared arrays between
	processes?
In-Reply-To: <470AFCCF.4000300@ar.media.kyoto-u.ac.jp>
References: <6.2.3.4.2.20071008105940.09e5fed8@blue-cove.com>
	<470AFCCF.4000300@ar.media.kyoto-u.ac.jp>
Message-ID: <bc657ead0710090019t4a0876d3u26d2e8080182d2c9@mail.gmail.com>

Hi!
I was in fact experimenting with this. The solution seemed to lie in
"simple" memmap as it is implemented in Windows:

import numpy as N
def arrSharedMemory(shape, dtype, tag="PriithonSharedMemory"):
    """
    Windows only !
    share memory between different processes if same `tag` is used.
    """
    itemsize = N.dtype(dtype).itemsize
    count = N.product(shape)
    size =  count * itemsize

    import mmap
    sharedmem = mmap.mmap(0, size, tag)
    a=N.frombuffer(sharedmem, dtype, count)
    a.shape = shape
    return a

For explaintion look up the microsoft site for the mmap documentation.
And/or the Python-doc for mmap.
(( I have to mention, that I could crash a process while testing this ... ))

If anyone here would know an equivalent way of doing this on
Linux/OS-X  we were back to a cross-platfrom function.



Hope this helps,
Sebastian Haase

On 10/9/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> Ray S wrote:
> > Is anyone sharing arrays between processes on Windows?
> > I tried compiling the posh sources (once, so far) with the new MS
> > toolkit and failed...
> > What other solutions are in use?
> >
> > Have a second process create an array view from an address would
> > suffice for this particular purpose. I could pass the address as a
> > parameter of the second process's argv.
> >
> > I've also tried things like
> > pb=pythonapi.PyBuffer_FromReadWriteMemory(9508824, 9*sizeof(c_int))
> > N.frombuffer(pb, N.int32)
> > which fails since pb is and int. What are my options?
> >
> (disclaimer: I know nothing about windows idiosyncraties)
>
> Could not this be because you compiled the posh sources with a
> compiler/runtime which is different than the other extensions and python
> interpreter ? I don't know the details, but since most of the posix
> functions related to files and processes are broken beyond despair in
> windows, and in particular, many posix handles cannot cross dll
> boundaries compiled by different compilers, I would not be surprised if
> this cause some trouble.
>
> The fact that POSH is said to be posix-only on python.org
> (http://wiki.python.org/moin/ParallelProcessing) would imply that people
> do not care much about windows, too (but again, this is just from
> reading what posh is about; I have never used it personnally).
>
> cheers,
>
> David
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From svetosch at gmx.net  Tue Oct  9 02:36:27 2007
From: svetosch at gmx.net (Sven Schreiber)
Date: Tue, 09 Oct 2007 07:36:27 +0100
Subject: [Numpy-discussion] beginner question: rank-1 arrays
In-Reply-To: <Mahogany-0.67.0-1716-20071008-182815.00@american.edu>
References: <b14b627d0710081500q7fa640ds969d2c7d4bbe8232@mail.gmail.com>
	<Mahogany-0.67.0-1716-20071008-182815.00@american.edu>
Message-ID: <470B216B.5040808@gmx.net>

Alan G Isaac schrieb:
> On Mon, 8 Oct 2007, Robin apparently wrote:
>> However in my code (I am converting from MATLAB) it is 
>> important to maintain 2d arrays, and keep the difference 
>> between row and column vectors.
> 
> How about using matrices?
> help(numpy.mat)
> 
> hth,
> Alan Isaac
> 

Robin, Alan is right, you want numpy matrices which are always 2d. Check
out numpy.matlib; if you replace
from numpy import [whatever]
by
from numpy.matlib import [whatever]
you get everything there is in numpy, and things like ones() zeros()
empty() etc. will always be 2d matrices.

-sven



From david at ar.media.kyoto-u.ac.jp  Tue Oct  9 03:32:16 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 09 Oct 2007 16:32:16 +0900
Subject: [Numpy-discussion] numpy/Windows shared arrays
	between	processes?
In-Reply-To: <bc657ead0710090019t4a0876d3u26d2e8080182d2c9@mail.gmail.com>
References: <6.2.3.4.2.20071008105940.09e5fed8@blue-cove.com>	<470AFCCF.4000300@ar.media.kyoto-u.ac.jp>
	<bc657ead0710090019t4a0876d3u26d2e8080182d2c9@mail.gmail.com>
Message-ID: <470B2E80.7000006@ar.media.kyoto-u.ac.jp>

Sebastian Haase wrote:
> Hi!
> I was in fact experimenting with this. The solution seemed to lie in
> "simple" memmap as it is implemented in Windows:
>
> import numpy as N
> def arrSharedMemory(shape, dtype, tag="PriithonSharedMemory"):
>     """
>     Windows only !
>     share memory between different processes if same `tag` is used.
>     """
>     itemsize = N.dtype(dtype).itemsize
>     count = N.product(shape)
>     size =  count * itemsize
>
>     import mmap
>     sharedmem = mmap.mmap(0, size, tag)
>     a=N.frombuffer(sharedmem, dtype, count)
>     a.shape = shape
>     return a
>
> For explaintion look up the microsoft site for the mmap documentation.
> And/or the Python-doc for mmap.
> (( I have to mention, that I could crash a process while testing this ... ))
>
> If anyone here would know an equivalent way of doing this on
> Linux/OS-X  we were back to a cross-platfrom function.
>   
AFAIK, the tag thing is pretty much windows specific, so why not just 
ignoring it on non windows platforms ? (or interpreting the tag argument 
as the flag argument for mmap, which would be consistent with python 
mmap API ?)

cheers,

David


From haase at msg.ucsf.edu  Tue Oct  9 08:22:36 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 9 Oct 2007 14:22:36 +0200
Subject: [Numpy-discussion] numpy/Windows shared arrays between
	processes?
In-Reply-To: <470B2E80.7000006@ar.media.kyoto-u.ac.jp>
References: <6.2.3.4.2.20071008105940.09e5fed8@blue-cove.com>
	<470AFCCF.4000300@ar.media.kyoto-u.ac.jp>
	<bc657ead0710090019t4a0876d3u26d2e8080182d2c9@mail.gmail.com>
	<470B2E80.7000006@ar.media.kyoto-u.ac.jp>
Message-ID: <bc657ead0710090522v370ff18cx29b59e2f58027720@mail.gmail.com>

On 10/9/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> Sebastian Haase wrote:
> > Hi!
> > I was in fact experimenting with this. The solution seemed to lie in
> > "simple" memmap as it is implemented in Windows:
> >
> > import numpy as N
> > def arrSharedMemory(shape, dtype, tag="PriithonSharedMemory"):
> >     """
> >     Windows only !
> >     share memory between different processes if same `tag` is used.
> >     """
> >     itemsize = N.dtype(dtype).itemsize
> >     count = N.product(shape)
> >     size =  count * itemsize
> >
> >     import mmap
> >     sharedmem = mmap.mmap(0, size, tag)
> >     a=N.frombuffer(sharedmem, dtype, count)
> >     a.shape = shape
> >     return a
> >
> > For explaintion look up the microsoft site for the mmap documentation.
> > And/or the Python-doc for mmap.
> > (( I have to mention, that I could crash a process while testing this ... ))
> >
> > If anyone here would know an equivalent way of doing this on
> > Linux/OS-X  we were back to a cross-platfrom function.
> >
> AFAIK, the tag thing is pretty much windows specific, so why not just
> ignoring it on non windows platforms ? (or interpreting the tag argument
> as the flag argument for mmap, which would be consistent with python
> mmap API ?)
>
As I recollect, the tag thing was the key for  turning the mmap into a
"not really memmaped file", that is, a memmap without a corresponding
file on the disk.
In other words, isn't a mmap ( without(!) tag ) always bound to a
"real" file in the file system ?

-Sebastian


From cournape at gmail.com  Tue Oct  9 09:05:22 2007
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 9 Oct 2007 22:05:22 +0900
Subject: [Numpy-discussion] numpy/Windows shared arrays between
	processes?
In-Reply-To: <bc657ead0710090522v370ff18cx29b59e2f58027720@mail.gmail.com>
References: <6.2.3.4.2.20071008105940.09e5fed8@blue-cove.com>
	<470AFCCF.4000300@ar.media.kyoto-u.ac.jp>
	<bc657ead0710090019t4a0876d3u26d2e8080182d2c9@mail.gmail.com>
	<470B2E80.7000006@ar.media.kyoto-u.ac.jp>
	<bc657ead0710090522v370ff18cx29b59e2f58027720@mail.gmail.com>
Message-ID: <5b8d13220710090605l661d476bvf4af415a4516db73@mail.gmail.com>

On 10/9/07, Sebastian Haase <haase at msg.ucsf.edu> wrote:
> >
> As I recollect, the tag thing was the key for  turning the mmap into a
> "not really memmaped file", that is, a memmap without a corresponding
> file on the disk.

All this is windows specific, I have never seen this tag thing on other OS.

> In other words, isn't a mmap ( without(!) tag ) always bound to a
> "real" file in the file system ?
No: mmap makes the link between a memory area and a file descriptor.
Nothing force the file descriptor to be bound to a "real" file. POSIX
defines shm_open to create shared memory file descriptor. AFAIK, this
is the standard way to share memory between processes on modern
unices: first create a memory area using shm_open, and then mmap it.

David


From subscriber100 at rjs.org  Tue Oct  9 11:07:36 2007
From: subscriber100 at rjs.org (Ray Schumacher)
Date: Tue, 09 Oct 2007 08:07:36 -0700
Subject: [Numpy-discussion] numpy/Windows shared arrays between
	processes?
In-Reply-To: <mailman.101802.1191932560.30507.numpy-discussion@scipy.org
 >
References: <mailman.101802.1191932560.30507.numpy-discussion@scipy.org>
Message-ID: <6.2.3.4.2.20071009074318.02e9ac00@rjs.org>

At 05:22 AM 10/9/2007, David Cournapeau wrote:
>Could not this be because you compiled the posh sources with a
>compiler/runtime which is different than the other extensions and python
>interpreter ?

It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and 
anyone else who didn't download it in time) are now seemingly SOL 
since it is no longer available. I saw much discussion of this as 
well, but even 2.5 is now "fixed" on 7.1 and reports of compiling 
distutil modules with the new MS SDK and having them work at all with 
2.4 were very mixed. I also tried GCC and had a litany of other 
errors with the posh.

>Sebastian Haase added:
>I was in fact experimenting with this. The solution seemed to lie in
>"simple" memmap as it is implemented in Windows:

<snip>
I had just found and started to write some tests with that MS 
function. If I can truly write to the array in one process and 
instantly read it in the other I'll be happy. Did you find that locks 
or semaphores were needed?

>(( I have to mention, that I could crash a process while testing this ... ))

That was one of my first results! I also found that using ctypes to 
create arrays from the other process's address and laying a numpy 
array on top was prone to that in experimentation. But I had the same 
issue as Mark Heslep
http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422
of creating a numpy array from a raw address (not a c_array).

Thanks,
Ray Schumacher




From haase at msg.ucsf.edu  Tue Oct  9 11:44:28 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 9 Oct 2007 17:44:28 +0200
Subject: [Numpy-discussion] numpy/Windows shared arrays between
	processes?
In-Reply-To: <6.2.3.4.2.20071009074318.02e9ac00@rjs.org>
References: <mailman.101802.1191932560.30507.numpy-discussion@scipy.org>
	<6.2.3.4.2.20071009074318.02e9ac00@rjs.org>
Message-ID: <bc657ead0710090844i265399d0t3eb77774850a1bcc@mail.gmail.com>

On 10/9/07, Ray Schumacher <subscriber100 at rjs.org> wrote:
> At 05:22 AM 10/9/2007, David Cournapeau wrote:
> >Could not this be because you compiled the posh sources with a
> >compiler/runtime which is different than the other extensions and python
> >interpreter ?
>
> It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and
> anyone else who didn't download it in time) are now seemingly SOL
> since it is no longer available. I saw much discussion of this as
> well, but even 2.5 is now "fixed" on 7.1 and reports of compiling
> distutil modules with the new MS SDK and having them work at all with
> 2.4 were very mixed. I also tried GCC and had a litany of other
> errors with the posh.
>
> >Sebastian Haase added:
> >I was in fact experimenting with this. The solution seemed to lie in
> >"simple" memmap as it is implemented in Windows:
>
> <snip>
> I had just found and started to write some tests with that MS
> function. If I can truly write to the array in one process and
> instantly read it in the other I'll be happy. Did you find that locks
> or semaphores were needed?

Maybe that's why it crashed ;-) !?  But for simple use it seems fine.

>
> >(( I have to mention, that I could crash a process while testing this ... ))
>
> That was one of my first results! I also found that using ctypes to
> create arrays from the other process's address and laying a numpy
> array on top was prone to that in experimentation. But I had the same
> issue as Mark Heslep
> http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422
> of creating a numpy array from a raw address (not a c_array).
>
I assume this is a different issue, but haven't looked into it yet.

-Sebastian


From robince at gmail.com  Tue Oct  9 13:55:12 2007
From: robince at gmail.com (Robin)
Date: Tue, 9 Oct 2007 18:55:12 +0100
Subject: [Numpy-discussion] diff of find of diff trick
Message-ID: <b14b627d0710091055x5d9f844dw4f8e065f72c1b14e@mail.gmail.com>

Hello,

As a converting MATLAB user I am bringing over some of my code. A core
function I need is to sample the probability distribution of a given set of
data. Sometimes these sets are large so I would like to make it as efficient
as possible. (the data are integers representing members of a discrete
space)

In MATLAB the best way I found was the "diff of find of diff" trick which
resulted in the completely vectorised solution (below). Does it make sense
to translate this into numpy? I don't have a feel yet for what is fast/slow
- are the functions below built in and so quick (I thought diff might not
be).

Is there a better/more pythonic way to do it?

--------
function Pr=prob(data, nR)

Pr    = zeros(nR,1);
% diff of find of diff trick for counting number of elements
temp     = sort(data(data>0));    % want vector excluding P(0)
dtemp    = diff([temp;max(temp)+1]);
count     = diff(find([1;dtemp]));
indx     = temp(dtemp>0);
Pr(indx)= count ./ numel(data);    % probability
--------

Thanks

Robin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071009/10daddfb/attachment.html>

From tgrav at mac.com  Tue Oct  9 16:37:04 2007
From: tgrav at mac.com (Tommy Grav)
Date: Tue, 9 Oct 2007 16:37:04 -0400
Subject: [Numpy-discussion] Warnings
Message-ID: <ECF6C406-1C5C-411B-8993-EF0CA8E4721B@mac.com>

How can I get the line number of where a numpy warning message is  
envoked in my code?

Cheers
   Tommy


From subscriber100 at rjs.org  Tue Oct  9 17:32:45 2007
From: subscriber100 at rjs.org (Ray S)
Date: Tue, 09 Oct 2007 14:32:45 -0700
Subject: [Numpy-discussion] numpy/Windows shared arrays between
	processes?
Message-ID: <6.2.3.4.2.20071009135108.05950a88@blue-cove.com>

On 10/9/07, Sebastian Haase replied:
 > > Did you find that locks
 > > or semaphores were needed?
 > Maybe that's why it crashed ;-) !?  But for simple use it seems 
fine.

I just did some code (below) that does read/write to the array AFAP, 
and there is no crash, or any other issue (Win2000, py2.4, numpy 
1.0b1).
Without the print statements, it does max both processors; with 
printing I/O only 58%.
Both processes can modify the array without issue either.
I'll experiment with

I had seen the Win mmap in this thread:
http://objectmix.com/python/127666-shared-memory-pointer.html
and here:
http://www.codeproject.com/cpp/embedpython_2.asp

Note also that the Python mmap docs read "In either case you must 
provide a file descriptor for a file opened for update." and no 
mention of the integer zero descriptor option.
Access options behave as presented.

Because *NIX has MAP_SHARED as an option you'd think that there might 
be cross-platform share behavior with some platform checking if 
statements. Without a tag though, how does another process reference 
the same memory on NIX, a filename? (It seems)

 > > But I had the same
 > > issue as Mark Heslep
 > > 
http://aspn.activestate.com/ASPN/Mail/Message/ctypes-users/3192422
 > > of creating a numpy array from a raw address (not a c_array).
 >I assume this is a different issue, but haven't looked into it yet.

Yes, a different methodology attempt. It would be interesting to know 
anyway how to create a numpy array from an address; it's probably 
buried in the undocumented C-API that I don't grok, and likely 
frowned upon.

Thanks,
Ray



From robert.kern at gmail.com  Tue Oct  9 17:50:05 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 09 Oct 2007 16:50:05 -0500
Subject: [Numpy-discussion] numpy/Windows shared arrays
	between	processes?
In-Reply-To: <6.2.3.4.2.20071009135108.05950a88@blue-cove.com>
References: <6.2.3.4.2.20071009135108.05950a88@blue-cove.com>
Message-ID: <470BF78D.7010207@gmail.com>

Ray S wrote:

> Yes, a different methodology attempt. It would be interesting to know 
> anyway how to create a numpy array from an address; it's probably 
> buried in the undocumented C-API that I don't grok, and likely 
> frowned upon.

Make a dummy object that exposes the __array_interface__ attribute filled with
the appropriate information:

   http://numpy.scipy.org/array_interface.shtml

Something like the following should suffice (untested, though I've done similar
things with ctypes before):


import numpy

def fromaddress(address, dtype, shape, strides=None):
    """ Create a numpy array from an integer address, a dtype, a shape
    tuple, and possibly strides.
    """
    # Make sure our dtype is a dtype, not just "f" or whatever.
    dtype = numpy.dtype(dtype)

    class Dummy(object):
        pass
    d = Dummy()
    d.__array_interface__ = dict(
        data = (address, False),
        typestr = dtype.str,
        descr = dtype.descr,
        shape = shape,
        strides = strides,
        version = 3,
    )
    return numpy.asarray(d)

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From subscriber100 at rjs.org  Tue Oct  9 17:52:36 2007
From: subscriber100 at rjs.org (Ray S)
Date: Tue, 09 Oct 2007 14:52:36 -0700
Subject: [Numpy-discussion] numpy/Windows shared arrays between
	processes? CODE...
Message-ID: <6.2.3.4.2.20071009145124.07132d88@rjs.org>

# numpyShared.py

import numpy as N
def arrSharedMemory(shape, dtype, tag="PySharedMemory", access=None):
     """
     Windows only !
     share memory between different processes if same `tag` is used.
     """
     itemsize = N.dtype(dtype).itemsize
     count = N.product(shape)
     size =  count * itemsize

     import mmap
     sharedmem = mmap.mmap(0, size, tag, access)
     a=N.frombuffer(sharedmem, dtype, count)
     a.shape = shape
     return a

if __name__ == '__main__':
     ## test it
     ## produce a mish-mash of output on one shell screen
     import random
     import time
     import subprocess, sys
     import mmap

     if len(sys.argv)<2:
         pid = subprocess.Popen([r'C:\python24\python.exe', 
['numpyShared.py rjs100']]).pid
         ## the parent process
         a1 = arrSharedMemory((10000,), N.int32, tag='rjs100', 
access=mmap.ACCESS_WRITE)
         time.sleep(.7)
         now=time.time();
         while time.time()<now+3:
             tmp = a1[0]**.5
             a1[0]=random.random()*10
             #print '\np%d' % a1[0]
     else:
         tag = sys.argv[0][1]
         a2 = arrSharedMemory((10000,), N.int32, tag='rjs100', 
access=mmap.ACCESS_READ)
         now=time.time()
         while time.time()<now+3:
             tmp = a2[0]**.5
             #print '\nc%d' % a2[0]
     



From david at ar.media.kyoto-u.ac.jp  Tue Oct  9 20:55:07 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 10 Oct 2007 09:55:07 +0900
Subject: [Numpy-discussion] numpy/Windows shared arrays
	between	processes?
In-Reply-To: <6.2.3.4.2.20071009074318.02e9ac00@rjs.org>
References: <mailman.101802.1191932560.30507.numpy-discussion@scipy.org>
	<6.2.3.4.2.20071009074318.02e9ac00@rjs.org>
Message-ID: <470C22EB.4020406@ar.media.kyoto-u.ac.jp>

Ray Schumacher wrote:
> At 05:22 AM 10/9/2007, David Cournapeau wrote:
>   
>> Could not this be because you compiled the posh sources with a
>> compiler/runtime which is different than the other extensions and python
>> interpreter ?
>>     
>
> It definitely was - since my 2.4 wanted the free 7.1 compiler, I (and 
> anyone else who didn't download it in time) are now seemingly SOL 
> since it is no longer available. I saw much discussion of this as 
> well, but even 2.5 is now "fixed" on 7.1 and reports of compiling 
> distutil modules with the new MS SDK and having them work at all with 
> 2.4 were very mixed. I also tried GCC and had a litany of other 
> errors with the posh.
>   
Basically, you cannot expect file descriptors (or even file handles: the 
standard ones from C library fopen) to cross dll boundaries if the dll 
do not have exactly the same runtime. The only solution is to convert 
your file descriptor to the non standard HANDLE things from the Win32 
API, and pass this at the API level. Now, I have no idea if this works 
also for mmap files; maybe this is broken too.

cheers,

David


From subscriber100 at rjs.org  Wed Oct 10 14:57:10 2007
From: subscriber100 at rjs.org (Ray S)
Date: Wed, 10 Oct 2007 11:57:10 -0700
Subject: [Numpy-discussion] numpy/Windows shared arrays between
	processes?
In-Reply-To: <mailman.13.1192035604.17671.numpy-discussion@scipy.org>
References: <mailman.13.1192035604.17671.numpy-discussion@scipy.org>
Message-ID: <6.2.3.4.2.20071010113132.059f04e0@rjs.org>

Thanks all:

At 10:00 AM 10/10/2007, Robert Kern wrote:
 >Something like the following should suffice (untested, though 
I've >done similar things with ctypes before):

I tested, successfully:
""" nFromAddress.py
"""

def fromaddress(address, dtype, shape, strides=None):
     """ Create a numpy array from an integer address, a dtype
     or dtype string, a shape tuple, and possibly strides.
     """
     import numpy
     # Make sure our dtype is a dtype, not just "f" or whatever.
     dtype = numpy.dtype(dtype)

     class Dummy(object):
         pass
     d = Dummy()
     d.__array_interface__ = dict(
         data = (address, False),
         typestr = dtype.str,
         descr = dtype.descr,
         shape = shape,
         strides = strides,
         version = 3,
     )
     return numpy.asarray(d)

## Numeric example, with address kludge
import Numeric, numpy, ctypes, string
a0 = Numeric.zeros((10000), Numeric.Int16)
nAddress = int(string.split(repr(a0.__copy__))[-1][:-1], 16)
tmp=(ctypes.c_long*1)(0)
ctypes.memmove(tmp, nAddress+8, 4)
nAddress = tmp[0]
a1 = fromaddress(nAddress, numpy.int16, (10000,)) ## explicit type
a0[0] = 5
print a1[0]

## numpy example
a2 = numpy.zeros(10000, numpy.int16)
nAddress = a2.__array_interface__['data'][0]
nType = a2.__array_interface__['typestr']
nShape = a2.__array_interface__['shape']
a3 = fromaddress(nAddress, nType, nShape)
a2[0] = 5
print a3[0]

So, now with little effort the relevant info can be passed over 
pipes, shared memory, etc. and shares/views created in other 
processes, since they are not objects but ints and strings.

 >David Cournapeau Wrote:
 >Basically, you cannot expect file descriptors (or even file 
handles: >the
 >standard ones from C library fopen) to cross dll boundaries if 
the >dll do not have exactly the same runtime.

It sounds like there is a general dilemma: no one with Python 2.4 or 
2.5 can reliably expect to compile extensions/modules if they did not 
install the 7.1 compiler in time.
The 2.6 seems to use VC 2005 Express, I don't know about py3000(?), 
with associated upgrade issues.
It would be nice if the build bots could also compile suggested 
modules/extentions.

Thanks again,
Ray



From guillemborrell at gmail.com  Wed Oct 10 14:58:48 2007
From: guillemborrell at gmail.com (Guillem Borrell i Nogueras)
Date: Wed, 10 Oct 2007 20:58:48 +0200
Subject: [Numpy-discussion] How does Python treat commons in fortran code?
Message-ID: <200710102058.49101.guillemborrell@gmail.com>

Hello all

This afternoon I introduced Python to my boss and the rest of my colleagues in 
the CFD laboratory (universidad politecnica de madrid, school of 
aeronautics).  My boss wanted to know if it would be helpful to manage a 
quite complex parallel algorithm while keeping its performance.  He has a 
long background in high performance computing, mainly in fortran.  He asked 
me something i could not answer, in fact I just have no clue about it.

We have some very tuned parallel subroutines to perform FFTs that are written 
in fortran 77; they have lots of _common_ structures that are used to manage 
large chunks of memory.  Those subroutines have shown to be scalable to 2000 
processors so it is very important that they perform as expected. Of course 
our intention is to use them and I had the idea of building a wrapper using 
f2py.

His question was.  ?How does ctypes or f2py handle the _common_ structures 
present in fortran 77?  ?Are all they allocated at load? ?Do they work 
exactly as they were called from a fortran executable and the main program is 
aware of all the _commons_ and allocates the storage when it is asked for?

I told him that I see no difference between the python interpreter and any 
other executable but my knowledge of python is not that deep. I could not 
explain him successfully exactly what is stored in the interpreter stack and 
what is not neither.

He has been programming for more than 40 years and I could not get him much 
into object oriented programing, however he found python+numpy+scipy a very 
promising tool.

guillem


From humufr at yahoo.fr  Wed Oct 10 15:04:52 2007
From: humufr at yahoo.fr (humufr at yahoo.fr)
Date: Wed, 10 Oct 2007 15:04:52 -0400
Subject: [Numpy-discussion] fortran 90 compiler problem
Message-ID: <200710101504.53064.humufr@yahoo.fr>

		Hello,

I'm using the last svn version of numpy (.

I tried the solution found on the mail list:

python setup.py config_fc --fcompiler=gnu95 build

the two interesting part (at least for me) in the building process are:

customize GnuFCompiler Found executable /usr/bin/g77 
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found 
customize GnuFCompiler
gnu: no Fortran 90 compiler found 
gnu: no Fortran 90 compiler found 
customize GnuFCompiler using config

and later: 
customize G95FCompiler Found executable /usr/local/bin/g95

but I still have the same answer when I'm trying to do: f2py -c -m hello 
hello.f90

customize GnuFCompiler Found executable /usr/bin/g77 
gnu: no Fortran 90 compiler found gnu: no Fortran 90 
compiler found customize GnuFCompiler 
gnu: no Fortran 90 compiler found 
gnu: no Fortran 90 compiler found

and so:

error: f90 not supported by GnuFCompiler needed for hello.f90

I can't compile my fortran module because of this. I'll attach the file obtain 
with

python setup.py -v config_fc --fcompiler=gnu95 build

And I have the same result for g95.

thank you for numpy and f2py. 

N.
-------------- next part --------------
non-existing path in 'numpy/distutils': 'site.cfg'
F2PY Version 2_4026
blas_opt_info:
blas_mkl_info:
( library_dirs = /usr/local/lib:/usr/lib )
( include_dirs = /usr/local/include:/usr/include )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries mkl,vml,guide not found in /usr/local/lib
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries mkl,vml,guide not found in /usr/lib
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
( library_dirs = /usr/local/lib:/usr/lib )
(paths: )
(paths: /usr/lib/atlas,/usr/lib/sse2)
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/atlas
(paths: )
(paths: )
(paths: )
(paths: )
(paths: /usr/lib/sse2/libatlas.so)
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
(paths: )
(paths: )
(paths: )
(paths: )
(paths: /usr/lib/libatlas.so)
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
  NOT AVAILABLE

atlas_blas_info:
( library_dirs = /usr/local/lib:/usr/lib )
(paths: )
(paths: /usr/lib/atlas,/usr/lib/sse2)
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries f77blas,cblas,atlas not found in /usr/local/lib
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries f77blas,cblas,atlas not found in /usr/lib/atlas
(paths: /usr/lib/sse2/libf77blas.so)
(paths: /usr/lib/sse2/libcblas.so)
(paths: /usr/lib/sse2/libatlas.so)
( include_dirs = /usr/local/include:/usr/include )
(paths: )
(paths: /usr/include/atlas_type.h,/usr/include/atlas_misc.h,/usr/include/atlas_enum.h,/usr/include/atlas_aux.h)
(paths: /usr/include/cblas.h)
( library_dirs = /usr/local/lib:/usr/lib )
(paths: )
(paths: /usr/lib/atlas,/usr/lib/sse2)
  FOUND:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['/usr/lib/sse2']
    language = c
    include_dirs = ['/usr/include']

new_compiler returns distutils.unixccompiler.UnixCCompiler
customize GnuFCompiler
find_executable('g77')
Found executable /usr/bin/g77
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
exec_command(['/usr/bin/g77', '--version'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/bin/g77 --version ; echo $? > /tmp/tmphW9yyT/O1TTQZ ) > /tmp/tmphW9yyT/TsbhKl 2>&1')
_update_environment(...)
exec_command(['/usr/bin/g77', '-g', '-Wall', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer', '-print-libgcc-file-name'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/bin/g77 -g -Wall -fno-second-underscore -fPIC -O3 -funroll-loops -march=pentium-m -mmmx -msse2 -msse -fomit-frame-pointer -print-libgcc-file-name ; echo $? > /tmp/tmphW9yyT/kZfve1 ) > /tmp/tmphW9yyT/oZpGzj 2>&1')
_update_environment(...)
customize GnuFCompiler
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
exec_command(['/usr/bin/g77', '--version'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/bin/g77 --version ; echo $? > /tmp/tmphW9yyT/yuFkkf ) > /tmp/tmphW9yyT/J1E0P7 2>&1')
_update_environment(...)
exec_command(['/usr/bin/g77', '-g', '-Wall', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer', '-print-libgcc-file-name'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/bin/g77 -g -Wall -fno-second-underscore -fPIC -O3 -funroll-loops -march=pentium-m -mmmx -msse2 -msse -fomit-frame-pointer -print-libgcc-file-name ; echo $? > /tmp/tmphW9yyT/Y5cb-T ) > /tmp/tmphW9yyT/y1_TOM 2>&1')
_update_environment(...)
customize GnuFCompiler using config
********************************************************************************
numpy.distutils.fcompiler.gnu.GnuFCompiler
compiler_f77    = ['/usr/bin/g77', '-g', '-Wall', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer']
compiler_fix    = None
ranlib          = ['/usr/bin/g77']
linker_exe      = ['/usr/bin/g77', '-g', '-Wall', '-g', '-Wall']
archiver        = ['/usr/bin/g77', '-cr']
version_cmd     = ['/usr/bin/g77', '--version']
linker_so       = ['/usr/bin/g77', '-g', '-Wall', '-g', '-Wall', '-shared']
compiler_f90    = None
version         = LooseVersion ('3.4.6')
libraries       = ['g2c']
library_dirs    = []
object_switch   = '-o '
compile_switch  = '-c'
include_dirs    = []
********************************************************************************
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -Wall -Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
exec_command(['gcc', '-pthread', '-fno-strict-aliasing', '-DNDEBUG', '-g', '-O2', '-Wall', '-Wstrict-prototypes', '-fPIC', '-c', '_configtest.c', '-o', '_configtest.o'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -Wall -Wstrict-prototypes -fPIC -c _configtest.c -o _configtest.o ; echo $? > /tmp/tmphW9yyT/Nfe9Bl ) 2>&1 | tee /tmp/tmphW9yyT/WIMuQD ')
_update_environment(...)
gcc -pthread _configtest.o -L/usr/lib/sse2 -lf77blas -lcblas -latlas -o _configtest
exec_command(['gcc', '-pthread', '_configtest.o', '-L/usr/lib/sse2', '-lf77blas', '-lcblas', '-latlas', '-o', '_configtest'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( gcc -pthread _configtest.o -L/usr/lib/sse2 -lf77blas -lcblas -latlas -o _configtest ; echo $? > /tmp/tmphW9yyT/PW1tm6 ) 2>&1 | tee /tmp/tmphW9yyT/DtQWRF ')
_update_environment(...)
exec_command('./_configtest',)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( ./_configtest ; echo $? > /tmp/tmphW9yyT/O7bQ57 ) 2>&1 | tee /tmp/tmphW9yyT/SHuXWy ')
ATLAS version 3.6.0 built by root on Fri Jan  9 15:57:20 UTC 2004:
   UNAME    : Linux intech67 2.4.20 #1 SMP Fri Jan 10 18:29:51 EST 2003 i686 GNU/Linux
   INSTFLG  : 
   MMDEF    : /fix/g/camm/atlas3-3.6.0/CONFIG/ARCHS/P4SSE2/gcc/gemm
   ARCHDEF  : /fix/g/camm/atlas3-3.6.0/CONFIG/ARCHS/P4SSE2/gcc/misc
   F2CDEFS  : -DAdd__ -DStringSunStyle
   CACHEEDGE: 1048576
   F77      : /usr/bin/g77, version GNU Fortran (GCC) 3.3.3 20031229 (prerelease) (Debian)
   F77FLAGS : -fomit-frame-pointer -O
   CC       : /usr/bin/gcc, version gcc (GCC) 3.3.3 20031229 (prerelease) (Debian)
   CC FLAGS : -fomit-frame-pointer -O3 -funroll-all-loops
   MCC      : /usr/bin/gcc, version gcc (GCC) 3.3.3 20031229 (prerelease) (Debian)
   MCCFLAGS : -fomit-frame-pointer -O
_update_environment(...)
success!
removing: _configtest.c _configtest.o _configtest
( library_dirs = /usr/local/lib:/usr/lib )
  FOUND:
    libraries = ['f77blas', 'cblas', 'atlas']
    library_dirs = ['/usr/lib/sse2']
    language = c
    define_macros = [('ATLAS_INFO', '"\\"3.6.0\\""')]
    include_dirs = ['/usr/include']

lapack_opt_info:
lapack_mkl_info:
mkl_info:
( library_dirs = /usr/local/lib:/usr/lib )
( include_dirs = /usr/local/include:/usr/include )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries mkl,vml,guide not found in /usr/local/lib
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries mkl,vml,guide not found in /usr/lib
  NOT AVAILABLE

  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
( library_dirs = /usr/local/lib:/usr/lib )
(paths: )
(paths: /usr/lib/atlas,/usr/lib/sse2)
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
(paths: )
(paths: )
  libraries lapack_atlas not found in /usr/local/lib
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/atlas
(paths: )
(paths: )
  libraries lapack_atlas not found in /usr/lib/atlas
(paths: )
(paths: )
(paths: )
(paths: )
(paths: /usr/lib/sse2/libatlas.so)
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
(paths: /usr/lib/sse2/liblapack_atlas.so)
(paths: )
(paths: )
(paths: )
(paths: )
(paths: /usr/lib/libatlas.so)
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
(paths: /usr/lib/liblapack_atlas.so)
numpy.distutils.system_info.atlas_threads_info
  NOT AVAILABLE

atlas_info:
( library_dirs = /usr/local/lib:/usr/lib )
(paths: )
(paths: /usr/lib/atlas,/usr/lib/sse2)
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries f77blas,cblas,atlas not found in /usr/local/lib
(paths: )
(paths: )
  libraries lapack_atlas not found in /usr/local/lib
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
(paths: )
  libraries f77blas,cblas,atlas not found in /usr/lib/atlas
(paths: )
(paths: )
  libraries lapack_atlas not found in /usr/lib/atlas
(paths: /usr/lib/sse2/libf77blas.so)
(paths: /usr/lib/sse2/libcblas.so)
(paths: /usr/lib/sse2/libatlas.so)
(paths: /usr/lib/sse2/liblapack_atlas.so)
(paths: )
(paths: )
(paths: )
  libraries lapack not found in /usr/lib/sse2
(paths: /usr/lib/libf77blas.so)
(paths: /usr/lib/libcblas.so)
(paths: /usr/lib/libatlas.so)
(paths: /usr/lib/liblapack_atlas.so)
(paths: /usr/lib/atlas)
(paths: /usr/lib/liblapack.so)
numpy.distutils.system_info.atlas_info
( include_dirs = /usr/local/include:/usr/include )
(paths: )
(paths: /usr/include/atlas_type.h,/usr/include/atlas_misc.h,/usr/include/atlas_enum.h,/usr/include/atlas_aux.h)
(paths: /usr/include/cblas.h)
( library_dirs = /usr/local/lib:/usr/lib )
(paths: )
(paths: /usr/lib/atlas,/usr/lib/sse2)
  FOUND:
    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = f77
    include_dirs = ['/usr/include']

new_compiler returns distutils.unixccompiler.UnixCCompiler
customize GnuFCompiler
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
exec_command(['/usr/bin/g77', '--version'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/bin/g77 --version ; echo $? > /tmp/tmphW9yyT/lA2al1 ) > /tmp/tmphW9yyT/uS8AqW 2>&1')
_update_environment(...)
exec_command(['/usr/bin/g77', '-g', '-Wall', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer', '-print-libgcc-file-name'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/bin/g77 -g -Wall -fno-second-underscore -fPIC -O3 -funroll-loops -march=pentium-m -mmmx -msse2 -msse -fomit-frame-pointer -print-libgcc-file-name ; echo $? > /tmp/tmphW9yyT/cEJBJL ) > /tmp/tmphW9yyT/D_flLH 2>&1')
_update_environment(...)
customize GnuFCompiler
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
exec_command(['/usr/bin/g77', '--version'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/bin/g77 --version ; echo $? > /tmp/tmphW9yyT/KP4n80 ) > /tmp/tmphW9yyT/ttMXMf 2>&1')
_update_environment(...)
exec_command(['/usr/bin/g77', '-g', '-Wall', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer', '-print-libgcc-file-name'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/bin/g77 -g -Wall -fno-second-underscore -fPIC -O3 -funroll-loops -march=pentium-m -mmmx -msse2 -msse -fomit-frame-pointer -print-libgcc-file-name ; echo $? > /tmp/tmphW9yyT/1AHFMy ) > /tmp/tmphW9yyT/nIy-o0 2>&1')
_update_environment(...)
customize GnuFCompiler using config
********************************************************************************
numpy.distutils.fcompiler.gnu.GnuFCompiler
compiler_f77    = ['/usr/bin/g77', '-g', '-Wall', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer']
compiler_fix    = None
ranlib          = ['/usr/bin/g77']
linker_exe      = ['/usr/bin/g77', '-g', '-Wall', '-g', '-Wall']
archiver        = ['/usr/bin/g77', '-cr']
version_cmd     = ['/usr/bin/g77', '--version']
linker_so       = ['/usr/bin/g77', '-g', '-Wall', '-g', '-Wall', '-shared']
compiler_f90    = None
version         = LooseVersion ('3.4.6')
libraries       = ['g2c']
library_dirs    = []
object_switch   = '-o '
compile_switch  = '-c'
include_dirs    = []
********************************************************************************
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -Wall -Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
exec_command(['gcc', '-pthread', '-fno-strict-aliasing', '-DNDEBUG', '-g', '-O2', '-Wall', '-Wstrict-prototypes', '-fPIC', '-c', '_configtest.c', '-o', '_configtest.o'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -Wall -Wstrict-prototypes -fPIC -c _configtest.c -o _configtest.o ; echo $? > /tmp/tmphW9yyT/B5bzKU ) 2>&1 | tee /tmp/tmphW9yyT/WoAbnx ')
_update_environment(...)
gcc -pthread _configtest.o -llapack -lf77blas -lcblas -latlas -o _configtest
exec_command(['gcc', '-pthread', '_configtest.o', '-llapack', '-lf77blas', '-lcblas', '-latlas', '-o', '_configtest'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( gcc -pthread _configtest.o -llapack -lf77blas -lcblas -latlas -o _configtest ; echo $? > /tmp/tmphW9yyT/PYE2rx ) 2>&1 | tee /tmp/tmphW9yyT/c_H36m ')
_update_environment(...)
exec_command('./_configtest',)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( ./_configtest ; echo $? > /tmp/tmphW9yyT/Kv64cP ) 2>&1 | tee /tmp/tmphW9yyT/jRMK1p ')
ATLAS version 3.6.0 built by root on Fri Jan  9 15:57:20 UTC 2004:
   UNAME    : Linux intech67 2.4.20 #1 SMP Fri Jan 10 18:29:51 EST 2003 i686 GNU/Linux
   INSTFLG  : 
   MMDEF    : /fix/g/camm/atlas3-3.6.0/CONFIG/ARCHS/P4SSE2/gcc/gemm
   ARCHDEF  : /fix/g/camm/atlas3-3.6.0/CONFIG/ARCHS/P4SSE2/gcc/misc
   F2CDEFS  : -DAdd__ -DStringSunStyle
   CACHEEDGE: 1048576
   F77      : /usr/bin/g77, version GNU Fortran (GCC) 3.3.3 20031229 (prerelease) (Debian)
   F77FLAGS : -fomit-frame-pointer -O
   CC       : /usr/bin/gcc, version gcc (GCC) 3.3.3 20031229 (prerelease) (Debian)
   CC FLAGS : -fomit-frame-pointer -O3 -funroll-all-loops
   MCC      : /usr/bin/gcc, version gcc (GCC) 3.3.3 20031229 (prerelease) (Debian)
   MCCFLAGS : -fomit-frame-pointer -O
_update_environment(...)
success!
removing: _configtest.c _configtest.o _configtest
( library_dirs = /usr/local/lib:/usr/lib )
  FOUND:
    libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('ATLAS_INFO', '"\\"3.6.0\\""')]
    include_dirs = ['/usr/include']

running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running build_src
building py_modules sources
building extension "numpy.core.multiarray" sources
  adding 'build/src.linux-i686-2.5/numpy/core/config.h' to sources.
executing numpy/core/code_generators/generate_array_api.py
  adding 'build/src.linux-i686-2.5/numpy/core/__multiarray_api.h' to sources.
  adding 'build/src.linux-i686-2.5/numpy/core/src' to include_dirs.
numpy.core - nothing done with h_files = ['build/src.linux-i686-2.5/numpy/core/src/scalartypes.inc', 'build/src.linux-i686-2.5/numpy/core/src/arraytypes.inc', 'build/src.linux-i686-2.5/numpy/core/config.h', 'build/src.linux-i686-2.5/numpy/core/__multiarray_api.h']
building extension "numpy.core.umath" sources
  adding 'build/src.linux-i686-2.5/numpy/core/config.h' to sources.
executing numpy/core/code_generators/generate_ufunc_api.py
  adding 'build/src.linux-i686-2.5/numpy/core/__ufunc_api.h' to sources.
  adding 'build/src.linux-i686-2.5/numpy/core/src' to include_dirs.
numpy.core - nothing done with h_files = ['build/src.linux-i686-2.5/numpy/core/src/scalartypes.inc', 'build/src.linux-i686-2.5/numpy/core/src/arraytypes.inc', 'build/src.linux-i686-2.5/numpy/core/config.h', 'build/src.linux-i686-2.5/numpy/core/__ufunc_api.h']
building extension "numpy.core._sort" sources
  adding 'build/src.linux-i686-2.5/numpy/core/config.h' to sources.
executing numpy/core/code_generators/generate_array_api.py
  adding 'build/src.linux-i686-2.5/numpy/core/__multiarray_api.h' to sources.
numpy.core - nothing done with h_files = ['build/src.linux-i686-2.5/numpy/core/config.h', 'build/src.linux-i686-2.5/numpy/core/__multiarray_api.h']
building extension "numpy.core.scalarmath" sources
  adding 'build/src.linux-i686-2.5/numpy/core/config.h' to sources.
executing numpy/core/code_generators/generate_array_api.py
  adding 'build/src.linux-i686-2.5/numpy/core/__multiarray_api.h' to sources.
executing numpy/core/code_generators/generate_ufunc_api.py
  adding 'build/src.linux-i686-2.5/numpy/core/__ufunc_api.h' to sources.
numpy.core - nothing done with h_files = ['build/src.linux-i686-2.5/numpy/core/config.h', 'build/src.linux-i686-2.5/numpy/core/__multiarray_api.h', 'build/src.linux-i686-2.5/numpy/core/__ufunc_api.h']
building extension "numpy.core._dotblas" sources
  adding 'numpy/core/blasdot/_dotblas.c' to sources.
building extension "numpy.lib._compiled_base" sources
building extension "numpy.numarray._capi" sources
building extension "numpy.fft.fftpack_lite" sources
building extension "numpy.linalg.lapack_lite" sources
  adding 'numpy/linalg/lapack_litemodule.c' to sources.
building extension "numpy.random.mtrand" sources
new_compiler returns distutils.unixccompiler.UnixCCompiler
customize Gnu95FCompiler
find_executable('gfortran')
Found executable /usr/local/bin/gfortran
exec_command(['/usr/local/bin/gfortran', '--version'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/local/bin/gfortran --version ; echo $? > /tmp/tmphW9yyT/GYP3Wz ) > /tmp/tmphW9yyT/KZs0pi 2>&1')
_update_environment(...)
exec_command(['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer', '-print-libgcc-file-name'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/local/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops -march=pentium-m -mmmx -msse2 -msse -fomit-frame-pointer -print-libgcc-file-name ; echo $? > /tmp/tmphW9yyT/YZ1YO2 ) > /tmp/tmphW9yyT/CWhSug 2>&1')
_update_environment(...)
customize Gnu95FCompiler using config
********************************************************************************
numpy.distutils.fcompiler.gnu.Gnu95FCompiler
compiler_f77    = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer']
compiler_fix    = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '-fno-second-underscore', '-Wall', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer']
ranlib          = ['/usr/local/bin/gfortran']
linker_exe      = ['/usr/local/bin/gfortran', '-Wall', '-Wall']
archiver        = ['/usr/local/bin/gfortran', '-cr']
version_cmd     = ['/usr/local/bin/gfortran', '--version']
linker_so       = ['/usr/local/bin/gfortran', '-Wall', '-Wall', '-shared']
compiler_f90    = ['/usr/local/bin/gfortran', '-Wall', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer']
version         = LooseVersion ('4.3.0')
libraries       = ['gfortran']
library_dirs    = []
object_switch   = '-o '
compile_switch  = '-c'
include_dirs    = ['numpy/core/src', 'numpy/core/include', '/usr/include/python2.5']
********************************************************************************
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -Wall -Wstrict-prototypes -fPIC

compile options: '-Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c'
gcc: _configtest.c
exec_command(['gcc', '-pthread', '-fno-strict-aliasing', '-DNDEBUG', '-g', '-O2', '-Wall', '-Wstrict-prototypes', '-fPIC', '-Inumpy/core/src', '-Inumpy/core/include', '-I/usr/include/python2.5', '-c', '_configtest.c', '-o', '_configtest.o'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/src -Inumpy/core/include -I/usr/include/python2.5 -c _configtest.c -o _configtest.o ; echo $? > /tmp/tmphW9yyT/nrRzw6 ) 2>&1 | tee /tmp/tmphW9yyT/DE8zX0 ')
_update_environment(...)
gcc -pthread _configtest.o -o _configtest
exec_command(['gcc', '-pthread', '_configtest.o', '-o', '_configtest'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( gcc -pthread _configtest.o -o _configtest ; echo $? > /tmp/tmphW9yyT/0c8YGJ ) 2>&1 | tee /tmp/tmphW9yyT/DdaP1N ')
_update_environment(...)
_configtest
failure.
removing: _configtest.c _configtest.o _configtest
building data_files sources
running build_py
not copying numpy/ctypeslib.py (output up-to-date)
not copying numpy/dual.py (output up-to-date)
not copying numpy/matlib.py (output up-to-date)
not copying numpy/setup.py (output up-to-date)
not copying numpy/add_newdocs.py (output up-to-date)
not copying numpy/_import_tools.py (output up-to-date)
not copying numpy/__init__.py (output up-to-date)
not copying numpy/version.py (output up-to-date)
copying build/src.linux-i686-2.5/numpy/__config__.py -> build/lib.linux-i686-2.5/numpy
not copying numpy/distutils/lib2def.py (output up-to-date)
not copying numpy/distutils/environment.py (output up-to-date)
not copying numpy/distutils/exec_command.py (output up-to-date)
not copying numpy/distutils/info.py (output up-to-date)
not copying numpy/distutils/mingw32ccompiler.py (output up-to-date)
not copying numpy/distutils/cpuinfo.py (output up-to-date)
not copying numpy/distutils/line_endings.py (output up-to-date)
not copying numpy/distutils/interactive.py (output up-to-date)
not copying numpy/distutils/from_template.py (output up-to-date)
not copying numpy/distutils/conv_template.py (output up-to-date)
not copying numpy/distutils/setup.py (output up-to-date)
not copying numpy/distutils/__version__.py (output up-to-date)
not copying numpy/distutils/system_info.py (output up-to-date)
not copying numpy/distutils/intelccompiler.py (output up-to-date)
not copying numpy/distutils/log.py (output up-to-date)
not copying numpy/distutils/misc_util.py (output up-to-date)
not copying numpy/distutils/__init__.py (output up-to-date)
not copying numpy/distutils/ccompiler.py (output up-to-date)
not copying numpy/distutils/extension.py (output up-to-date)
not copying numpy/distutils/core.py (output up-to-date)
not copying numpy/distutils/unixccompiler.py (output up-to-date)
copying build/src.linux-i686-2.5/numpy/distutils/__config__.py -> build/lib.linux-i686-2.5/numpy/distutils
not copying numpy/distutils/command/install.py (output up-to-date)
not copying numpy/distutils/command/egg_info.py (output up-to-date)
not copying numpy/distutils/command/config_compiler.py (output up-to-date)
not copying numpy/distutils/command/bdist_rpm.py (output up-to-date)
not copying numpy/distutils/command/build_src.py (output up-to-date)
not copying numpy/distutils/command/sdist.py (output up-to-date)
not copying numpy/distutils/command/build_clib.py (output up-to-date)
not copying numpy/distutils/command/build_ext.py (output up-to-date)
not copying numpy/distutils/command/config.py (output up-to-date)
not copying numpy/distutils/command/__init__.py (output up-to-date)
not copying numpy/distutils/command/build_scripts.py (output up-to-date)
not copying numpy/distutils/command/build_py.py (output up-to-date)
not copying numpy/distutils/command/install_headers.py (output up-to-date)
not copying numpy/distutils/command/build.py (output up-to-date)
not copying numpy/distutils/command/install_data.py (output up-to-date)
not copying numpy/distutils/fcompiler/hpux.py (output up-to-date)
not copying numpy/distutils/fcompiler/nag.py (output up-to-date)
not copying numpy/distutils/fcompiler/mips.py (output up-to-date)
not copying numpy/distutils/fcompiler/g95.py (output up-to-date)
not copying numpy/distutils/fcompiler/gnu.py (output up-to-date)
not copying numpy/distutils/fcompiler/lahey.py (output up-to-date)
not copying numpy/distutils/fcompiler/intel.py (output up-to-date)
not copying numpy/distutils/fcompiler/ibm.py (output up-to-date)
not copying numpy/distutils/fcompiler/none.py (output up-to-date)
not copying numpy/distutils/fcompiler/vast.py (output up-to-date)
not copying numpy/distutils/fcompiler/compaq.py (output up-to-date)
not copying numpy/distutils/fcompiler/__init__.py (output up-to-date)
not copying numpy/distutils/fcompiler/sun.py (output up-to-date)
not copying numpy/distutils/fcompiler/absoft.py (output up-to-date)
not copying numpy/distutils/fcompiler/pg.py (output up-to-date)
not copying numpy/testing/numpytest.py (output up-to-date)
not copying numpy/testing/info.py (output up-to-date)
not copying numpy/testing/setup.py (output up-to-date)
not copying numpy/testing/parametric.py (output up-to-date)
not copying numpy/testing/__init__.py (output up-to-date)
not copying numpy/testing/utils.py (output up-to-date)
not copying numpy/f2py/info.py (output up-to-date)
not copying numpy/f2py/f2py_testing.py (output up-to-date)
not copying numpy/f2py/func2subr.py (output up-to-date)
not copying numpy/f2py/cfuncs.py (output up-to-date)
not copying numpy/f2py/rules.py (output up-to-date)
not copying numpy/f2py/f90mod_rules.py (output up-to-date)
not copying numpy/f2py/f2py2e.py (output up-to-date)
not copying numpy/f2py/setup.py (output up-to-date)
not copying numpy/f2py/capi_maps.py (output up-to-date)
not copying numpy/f2py/__version__.py (output up-to-date)
copying numpy/f2py/__svn_version__.py -> build/lib.linux-i686-2.5/numpy/f2py
not copying numpy/f2py/cb_rules.py (output up-to-date)
not copying numpy/f2py/common_rules.py (output up-to-date)
not copying numpy/f2py/use_rules.py (output up-to-date)
not copying numpy/f2py/__init__.py (output up-to-date)
not copying numpy/f2py/diagnose.py (output up-to-date)
not copying numpy/f2py/auxfuncs.py (output up-to-date)
not copying numpy/f2py/crackfortran.py (output up-to-date)
not copying numpy/f2py/lib/api.py (output up-to-date)
not copying numpy/f2py/lib/py_wrap_type.py (output up-to-date)
not copying numpy/f2py/lib/setup.py (output up-to-date)
not copying numpy/f2py/lib/nary.py (output up-to-date)
not copying numpy/f2py/lib/py_wrap_subprogram.py (output up-to-date)
not copying numpy/f2py/lib/__init__.py (output up-to-date)
not copying numpy/f2py/lib/wrapper_base.py (output up-to-date)
not copying numpy/f2py/lib/main.py (output up-to-date)
not copying numpy/f2py/lib/py_wrap.py (output up-to-date)
not copying numpy/f2py/lib/parser/api.py (output up-to-date)
not copying numpy/f2py/lib/parser/base_classes.py (output up-to-date)
not copying numpy/f2py/lib/parser/parsefortran.py (output up-to-date)
not copying numpy/f2py/lib/parser/sourceinfo.py (output up-to-date)
not copying numpy/f2py/lib/parser/statements.py (output up-to-date)
not copying numpy/f2py/lib/parser/splitline.py (output up-to-date)
not copying numpy/f2py/lib/parser/test_Fortran2003.py (output up-to-date)
not copying numpy/f2py/lib/parser/readfortran.py (output up-to-date)
not copying numpy/f2py/lib/parser/__init__.py (output up-to-date)
not copying numpy/f2py/lib/parser/pattern_tools.py (output up-to-date)
not copying numpy/f2py/lib/parser/utils.py (output up-to-date)
not copying numpy/f2py/lib/parser/test_parser.py (output up-to-date)
not copying numpy/f2py/lib/parser/typedecl_statements.py (output up-to-date)
not copying numpy/f2py/lib/parser/block_statements.py (output up-to-date)
not copying numpy/f2py/lib/parser/Fortran2003.py (output up-to-date)
not copying numpy/f2py/lib/extgen/base.py (output up-to-date)
not copying numpy/f2py/lib/extgen/py_support.py (output up-to-date)
not copying numpy/f2py/lib/extgen/c_support.py (output up-to-date)
not copying numpy/f2py/lib/extgen/__init__.py (output up-to-date)
not copying numpy/f2py/lib/extgen/utils.py (output up-to-date)
not copying numpy/f2py/lib/extgen/setup_py.py (output up-to-date)
not copying numpy/core/info.py (output up-to-date)
not copying numpy/core/numeric.py (output up-to-date)
not copying numpy/core/records.py (output up-to-date)
not copying numpy/core/defmatrix.py (output up-to-date)
not copying numpy/core/setup.py (output up-to-date)
not copying numpy/core/memmap.py (output up-to-date)
not copying numpy/core/arrayprint.py (output up-to-date)
copying numpy/core/__svn_version__.py -> build/lib.linux-i686-2.5/numpy/core
not copying numpy/core/defchararray.py (output up-to-date)
not copying numpy/core/__init__.py (output up-to-date)
not copying numpy/core/ma.py (output up-to-date)
not copying numpy/core/_internal.py (output up-to-date)
not copying numpy/core/fromnumeric.py (output up-to-date)
not copying numpy/core/numerictypes.py (output up-to-date)
not copying numpy/core/code_generators/generate_array_api.py (output up-to-date)
not copying numpy/core/info.py (output up-to-date)
not copying numpy/core/numeric.py (output up-to-date)
not copying numpy/core/records.py (output up-to-date)
not copying numpy/core/defmatrix.py (output up-to-date)
not copying numpy/core/setup.py (output up-to-date)
not copying numpy/core/memmap.py (output up-to-date)
not copying numpy/core/arrayprint.py (output up-to-date)
not copying numpy/core/defchararray.py (output up-to-date)
not copying numpy/core/__init__.py (output up-to-date)
not copying numpy/core/ma.py (output up-to-date)
not copying numpy/core/_internal.py (output up-to-date)
not copying numpy/core/fromnumeric.py (output up-to-date)
not copying numpy/core/numerictypes.py (output up-to-date)
not copying numpy/lib/function_base.py (output up-to-date)
not copying numpy/lib/info.py (output up-to-date)
not copying numpy/lib/shape_base.py (output up-to-date)
not copying numpy/lib/user_array.py (output up-to-date)
not copying numpy/lib/arraysetops.py (output up-to-date)
not copying numpy/lib/index_tricks.py (output up-to-date)
not copying numpy/lib/setup.py (output up-to-date)
not copying numpy/lib/polynomial.py (output up-to-date)
not copying numpy/lib/twodim_base.py (output up-to-date)
not copying numpy/lib/getlimits.py (output up-to-date)
not copying numpy/lib/scimath.py (output up-to-date)
not copying numpy/lib/convdtype.py (output up-to-date)
not copying numpy/lib/__init__.py (output up-to-date)
not copying numpy/lib/machar.py (output up-to-date)
not copying numpy/lib/utils.py (output up-to-date)
not copying numpy/lib/ufunclike.py (output up-to-date)
not copying numpy/lib/type_check.py (output up-to-date)
not copying numpy/oldnumeric/rng_stats.py (output up-to-date)
not copying numpy/oldnumeric/alter_code1.py (output up-to-date)
not copying numpy/oldnumeric/user_array.py (output up-to-date)
not copying numpy/oldnumeric/array_printer.py (output up-to-date)
not copying numpy/oldnumeric/functions.py (output up-to-date)
not copying numpy/oldnumeric/typeconv.py (output up-to-date)
not copying numpy/oldnumeric/ufuncs.py (output up-to-date)
not copying numpy/oldnumeric/fft.py (output up-to-date)
not copying numpy/oldnumeric/arrayfns.py (output up-to-date)
not copying numpy/oldnumeric/fix_default_axis.py (output up-to-date)
not copying numpy/oldnumeric/matrix.py (output up-to-date)
not copying numpy/oldnumeric/setup.py (output up-to-date)
not copying numpy/oldnumeric/misc.py (output up-to-date)
not copying numpy/oldnumeric/random_array.py (output up-to-date)
not copying numpy/oldnumeric/precision.py (output up-to-date)
not copying numpy/oldnumeric/rng.py (output up-to-date)
not copying numpy/oldnumeric/compat.py (output up-to-date)
not copying numpy/oldnumeric/__init__.py (output up-to-date)
not copying numpy/oldnumeric/ma.py (output up-to-date)
not copying numpy/oldnumeric/alter_code2.py (output up-to-date)
not copying numpy/oldnumeric/linear_algebra.py (output up-to-date)
not copying numpy/oldnumeric/mlab.py (output up-to-date)
not copying numpy/numarray/alter_code1.py (output up-to-date)
not copying numpy/numarray/functions.py (output up-to-date)
not copying numpy/numarray/ufuncs.py (output up-to-date)
not copying numpy/numarray/fft.py (output up-to-date)
not copying numpy/numarray/matrix.py (output up-to-date)
not copying numpy/numarray/util.py (output up-to-date)
not copying numpy/numarray/setup.py (output up-to-date)
not copying numpy/numarray/random_array.py (output up-to-date)
not copying numpy/numarray/compat.py (output up-to-date)
not copying numpy/numarray/__init__.py (output up-to-date)
not copying numpy/numarray/ma.py (output up-to-date)
not copying numpy/numarray/alter_code2.py (output up-to-date)
not copying numpy/numarray/nd_image.py (output up-to-date)
not copying numpy/numarray/linear_algebra.py (output up-to-date)
not copying numpy/numarray/session.py (output up-to-date)
not copying numpy/numarray/image.py (output up-to-date)
not copying numpy/numarray/convolve.py (output up-to-date)
not copying numpy/numarray/numerictypes.py (output up-to-date)
not copying numpy/numarray/mlab.py (output up-to-date)
not copying numpy/fft/info.py (output up-to-date)
not copying numpy/fft/setup.py (output up-to-date)
not copying numpy/fft/helper.py (output up-to-date)
not copying numpy/fft/__init__.py (output up-to-date)
not copying numpy/fft/fftpack.py (output up-to-date)
not copying numpy/linalg/info.py (output up-to-date)
not copying numpy/linalg/setup.py (output up-to-date)
not copying numpy/linalg/__init__.py (output up-to-date)
not copying numpy/linalg/linalg.py (output up-to-date)
not copying numpy/random/info.py (output up-to-date)
not copying numpy/random/setup.py (output up-to-date)
not copying numpy/random/__init__.py (output up-to-date)
running build_ext
new_compiler returns distutils.unixccompiler.UnixCCompiler
customize UnixCCompiler
customize UnixCCompiler using build_ext
********************************************************************************
distutils.unixccompiler.UnixCCompiler
compiler_cxx  = ['g++', '-pthread']
ranlib        = None
linker_exe    = ['gcc', '-pthread']
compiler_so   = ['gcc', '-pthread', '-fno-strict-aliasing', '-DNDEBUG', '-g', '-O2', '-Wall', '-Wstrict-prototypes', '-fPIC']
archiver      = ['ar', '-cr']
preprocessor  = ['gcc', '-pthread', '-E']
linker_so     = ['gcc', '-pthread', '-shared', '-Wl,-O1']
compiler      = ['gcc', '-pthread', '-fno-strict-aliasing', '-DNDEBUG', '-g', '-O2', '-Wall', '-Wstrict-prototypes']
libraries     = []
library_dirs  = []
include_dirs  = ['numpy/core/src', 'numpy/core/include', '/usr/include/python2.5']
********************************************************************************
customize Gnu95FCompiler
exec_command(['/usr/local/bin/gfortran', '--version'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/local/bin/gfortran --version ; echo $? > /tmp/tmphW9yyT/wQSPWL ) > /tmp/tmphW9yyT/HsfR2S 2>&1')
_update_environment(...)
exec_command(['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer', '-print-libgcc-file-name'],)
Retaining cwd: /home/gruel/tmp/Python/SCIPY/numpy
_preserve_environment([])
_update_environment(...)
_exec_command_posix(...)
Running os.system('( /usr/local/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops -march=pentium-m -mmmx -msse2 -msse -fomit-frame-pointer -print-libgcc-file-name ; echo $? > /tmp/tmphW9yyT/JIkx4b ) > /tmp/tmphW9yyT/bncwfA 2>&1')
_update_environment(...)
customize Gnu95FCompiler using build_ext
********************************************************************************
numpy.distutils.fcompiler.gnu.Gnu95FCompiler
compiler_f77    = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer']
compiler_fix    = ['/usr/local/bin/gfortran', '-Wall', '-ffixed-form', '-fno-second-underscore', '-Wall', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer']
ranlib          = ['/usr/local/bin/gfortran']
linker_exe      = ['/usr/local/bin/gfortran', '-Wall', '-Wall']
archiver        = ['/usr/local/bin/gfortran', '-cr']
version_cmd     = ['/usr/local/bin/gfortran', '--version']
linker_so       = ['/usr/local/bin/gfortran', '-Wall', '-Wall', '-shared']
compiler_f90    = ['/usr/local/bin/gfortran', '-Wall', '-fno-second-underscore', '-fPIC', '-O3', '-funroll-loops', '-march=pentium-m', '-mmmx', '-msse2', '-msse', '-fomit-frame-pointer']
version         = LooseVersion ('4.3.0')
libraries       = ['gfortran']
library_dirs    = []
object_switch   = '-o '
compile_switch  = '-c'
include_dirs    = ['numpy/core/src', 'numpy/core/include', '/usr/include/python2.5']
********************************************************************************
skipping 'numpy.core.multiarray' extension (up-to-date)
skipping 'numpy.core.umath' extension (up-to-date)
skipping 'numpy.core._sort' extension (up-to-date)
skipping 'numpy.core.scalarmath' extension (up-to-date)
skipping 'numpy.core._dotblas' extension (up-to-date)
skipping 'numpy.lib._compiled_base' extension (up-to-date)
skipping 'numpy.numarray._capi' extension (up-to-date)
skipping 'numpy.fft.fftpack_lite' extension (up-to-date)
skipping 'numpy.linalg.lapack_lite' extension (up-to-date)
skipping 'numpy.random.mtrand' extension (up-to-date)
running build_scripts
  adding 'build/scripts.linux-i686-2.5/f2py' to scripts
not copying build/scripts.linux-i686-2.5/f2py (up-to-date)
removing /tmp/tmphW9yyT

From lou_boog2000 at yahoo.com  Wed Oct 10 15:31:54 2007
From: lou_boog2000 at yahoo.com (Lou Pecora)
Date: Wed, 10 Oct 2007 12:31:54 -0700 (PDT)
Subject: [Numpy-discussion] fortran 90 compiler problem
In-Reply-To: <200710101504.53064.humufr@yahoo.fr>
Message-ID: <579657.49649.qm@web34410.mail.mud.yahoo.com>


--- humufr at yahoo.fr wrote:

> 		Hello,
> 
> I'm using the last svn version of numpy (.
> 
> I tried the solution found on the mail list:
> 
> python setup.py config_fc --fcompiler=gnu95 build
> 
> the two interesting part (at least for me) in the
> building process are:
> 
> customize GnuFCompiler Found executable /usr/bin/g77
> 
> gnu: no Fortran 90 compiler found
> gnu: no Fortran 90 compiler found 

[cut]

I had a similar problem (not sure if it's yours).  I
was told to install the gfortran compiler for scipy
and numpy.  I did and that enabled the install.

Here are my notes.  I hope they are understandable and
they help.  I think Robert Kern straightened me out on
this, so he gets the credit.  If I'm wrong, of course,
that's not Robert's fault.  

* Installing gfortran (needed for SciPy & NumPy):

 Get gfortran from  http://r.research.att.com/tools/
(.dmg file)

 Run the install package on the .dmg. (note it gives
you an uninstall, too)

 Add the paths /usr/local and /usr/local/bin to the
$PATH variable in the .profile startup file so other
apps can access the compiler and related stuff


Let me know if it works.  I like to keep on top of
this since these things seem to change at times and
little things like this can trip you up.




-- Lou Pecora,   my views are my own.


       
____________________________________________________________________________________
Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out.
http://answers.yahoo.com/dir/?link=list&sid=396545469


From humufr at yahoo.fr  Wed Oct 10 16:08:12 2007
From: humufr at yahoo.fr (humufr at yahoo.fr)
Date: Wed, 10 Oct 2007 16:08:12 -0400
Subject: [Numpy-discussion] fortran 90 compiler problem
Message-ID: <200710101608.12529.humufr@yahoo.fr>

I have already gfortran and g95 installed and working fine for all the system 
but numpy and scipy. 

By the way I'm using linux not MacOSX. My problem is that when I'm installing 
numpy gfortran (or g95) is detected but not at all the step and so f2py seem 
to be unable to use it. 

look at:

line 92 in my attach file:

find_executable('g77')
Found executable /usr/bin/g77
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found

but line 469:
 
customize Gnu95FCompiler
find_executable('gfortran')
Found executable /usr/local/bin/gfortran

I don't understand the problem.

Thanks

N.


From robert.kern at gmail.com  Wed Oct 10 16:15:32 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 10 Oct 2007 15:15:32 -0500
Subject: [Numpy-discussion] fortran 90 compiler problem
In-Reply-To: <200710101504.53064.humufr@yahoo.fr>
References: <200710101504.53064.humufr@yahoo.fr>
Message-ID: <470D32E4.5050908@gmail.com>

humufr at yahoo.fr wrote:
> 		Hello,
> 
> I'm using the last svn version of numpy (.
> 
> I tried the solution found on the mail list:
> 
> python setup.py config_fc --fcompiler=gnu95 build
> 
> the two interesting part (at least for me) in the building process are:
> 
> customize GnuFCompiler Found executable /usr/bin/g77 
> gnu: no Fortran 90 compiler found
> gnu: no Fortran 90 compiler found 
> customize GnuFCompiler
> gnu: no Fortran 90 compiler found 
> gnu: no Fortran 90 compiler found 
> customize GnuFCompiler using config
> 
> and later: 
> customize G95FCompiler Found executable /usr/local/bin/g95
> 
> but I still have the same answer when I'm trying to do: f2py -c -m hello 
> hello.f90
> 
> customize GnuFCompiler Found executable /usr/bin/g77 
> gnu: no Fortran 90 compiler found gnu: no Fortran 90 
> compiler found customize GnuFCompiler 
> gnu: no Fortran 90 compiler found 
> gnu: no Fortran 90 compiler found

This is a correct message and does not imply that no Fortran 90 compiler exists.
The GnuFCompiler ("gnu") only looks at g77, which is not a Fortran 90 compiler.
That message only applies to the "gnu" compiler, not any other.

> and so:
> 
> error: f90 not supported by GnuFCompiler needed for hello.f90
> 
> I can't compile my fortran module because of this. I'll attach the file obtain 
> with
> 
> python setup.py -v config_fc --fcompiler=gnu95 build
> 
> And I have the same result for g95.
> 
> thank you for numpy and f2py. 

Please give us the build log that actually failed and the command that you used
to obtain it. The one you show, for numpy, succeeded.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From humufr at yahoo.fr  Wed Oct 10 16:51:53 2007
From: humufr at yahoo.fr (humufr at yahoo.fr)
Date: Wed, 10 Oct 2007 16:51:53 -0400
Subject: [Numpy-discussion] fortran 90 compiler problem
In-Reply-To: <470D32E4.5050908@gmail.com>
References: <200710101504.53064.humufr@yahoo.fr> <470D32E4.5050908@gmail.com>
Message-ID: <200710101651.54591.humufr@yahoo.fr>

This is the log I have when I'm trying to use f2py with a fortran 90 file. 
The last line seems to tell me that f2py cannot see gfortran and g95 even if 
it detected them when I installed f2py.

thanks,

N.

gfortran and g95 are installed and I can use them:

> gfortran
gfortran: no input files

>gfortran -c hello.f90
> ls hello*
hello.f90 hello.o

> g95 
g95: no input files

>g95 -c hello.f90
> ls hello*
hello.f90 hello.o

and now

>f2py -c -m hello hello.f90
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler 
opti
ons
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler 
opt
ions
running build_src
building extension "hello" sources
f2py options: []
f2py:> /tmp/tmpg5exdv/src.linux-i686-2.5/hellomodule.c
creating /tmp/tmpg5exdv
creating /tmp/tmpg5exdv/src.linux-i686-2.5
Reading fortran codes...
        Reading file 'hello.f90' (format:free)
Post-processing...
        Block: hello
                        Block: foo
Post-processing (stage 2)...
Building modules...
        Building module "hello"...
                Constructing wrapper function "foo"...
                  foo(a)
        Wrote C/API module "hello" to 
file "/tmp/tmpg5exdv/src.linux-i686-2.5/he
llomodule.c"
  adding '/tmp/tmpg5exdv/src.linux-i686-2.5/fortranobject.c' to sources.
  adding '/tmp/tmpg5exdv/src.linux-i686-2.5' to include_dirs.
copying /home/gruel/usr/lib/python2.5/site-packages/numpy/f2py/src/fortranobject
.c -> /tmp/tmpg5exdv/src.linux-i686-2.5
copying /home/gruel/usr/lib/python2.5/site-packages/numpy/f2py/src/fortranobject
.h -> /tmp/tmpg5exdv/src.linux-i686-2.5
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize GnuFCompiler
Found executable /usr/bin/g77
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
customize GnuFCompiler
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
customize GnuFCompiler using build_ext
building 'hello' extension
compiling C sources
C compiler: 
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -Wall -Wstrict-pro                               
totypes -fPIC

creating /tmp/tmpg5exdv/tmp
creating /tmp/tmpg5exdv/tmp/tmpg5exdv
creating /tmp/tmpg5exdv/tmp/tmpg5exdv/src.linux-i686-2.5
compile 
options: '-I/tmp/tmpg5exdv/src.linux-i686-2.5 -I/home/gruel/usr/lib/pyth                               
on2.5/site-packages/numpy/core/include -I/usr/include/python2.5 -c'
gcc: /tmp/tmpg5exdv/src.linux-i686-2.5/hellomodule.c
gcc: /tmp/tmpg5exdv/src.linux-i686-2.5/fortranobject.c
compiling Fortran sources
Fortran f77 
compiler: /usr/bin/g77 -g -Wall -fno-second-underscore -fPIC -O3 -fu                               
nroll-loops -march=pentium-m -mmmx -msse2 -msse -fomit-frame-pointer
compile 
options: '-I/tmp/tmpg5exdv/src.linux-i686-2.5 -I/home/gruel/usr/lib/pyth                               
on2.5/site-packages/numpy/core/include -I/usr/include/python2.5 -c'
error: f90 not supported by GnuFCompiler needed for hello.f90



On Wednesday 10 October 2007 16:15:32 Robert Kern wrote:
> humufr at yahoo.fr wrote:
> > 		Hello,
> >
> > I'm using the last svn version of numpy (.
> >
> > I tried the solution found on the mail list:
> >
> > python setup.py config_fc --fcompiler=gnu95 build
> >
> > the two interesting part (at least for me) in the building process are:
> >
> > customize GnuFCompiler Found executable /usr/bin/g77
> > gnu: no Fortran 90 compiler found
> > gnu: no Fortran 90 compiler found
> > customize GnuFCompiler
> > gnu: no Fortran 90 compiler found
> > gnu: no Fortran 90 compiler found
> > customize GnuFCompiler using config
> >
> > and later:
> > customize G95FCompiler Found executable /usr/local/bin/g95
> >
> > but I still have the same answer when I'm trying to do: f2py -c -m hello
> > hello.f90
> >
> > customize GnuFCompiler Found executable /usr/bin/g77
> > gnu: no Fortran 90 compiler found gnu: no Fortran 90
> > compiler found customize GnuFCompiler
> > gnu: no Fortran 90 compiler found
> > gnu: no Fortran 90 compiler found
>
> This is a correct message and does not imply that no Fortran 90 compiler
> exists. The GnuFCompiler ("gnu") only looks at g77, which is not a Fortran
> 90 compiler. That message only applies to the "gnu" compiler, not any
> other.
>
> > and so:
> >
> > error: f90 not supported by GnuFCompiler needed for hello.f90
> >
> > I can't compile my fortran module because of this. I'll attach the file
> > obtain with
> >
> > python setup.py -v config_fc --fcompiler=gnu95 build
> >
> > And I have the same result for g95.
> >
> > thank you for numpy and f2py.
>
> Please give us the build log that actually failed and the command that you
> used to obtain it. The one you show, for numpy, succeeded.


From robince at gmail.com  Wed Oct 10 17:11:54 2007
From: robince at gmail.com (Robin)
Date: Wed, 10 Oct 2007 22:11:54 +0100
Subject: [Numpy-discussion] fortran 90 compiler problem
In-Reply-To: <200710101651.54591.humufr@yahoo.fr>
References: <200710101504.53064.humufr@yahoo.fr> <470D32E4.5050908@gmail.com>
	<200710101651.54591.humufr@yahoo.fr>
Message-ID: <b14b627d0710101411x5a944fc1ocb773fa865e5e4e6@mail.gmail.com>

I had the same problem building on Linux. I found this on a wiki somewhere
(can only find it in google cache now):

"""(Note: Make sure that if you build with *gfortran* that *g77* is not
installed on your system (or at least is not in your *PATH* when *numpy* is
being built) as you need to link with the same compiler that you built
lapack with when *numpy* builds. It will try and find *g77* first which will
lead to linking errors if you have built lapack with *gfortran*)."""

The trick for me was that step of removing g77 (or taking it out of the
path) - then distutils seems to pick up gfortran fine, but otherwise it
doesn't seem to work (even with --fcompiler=gnu95 etc)

Cheers

Robin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071010/2cf0f2f7/attachment.html>

From Chris.Barker at noaa.gov  Wed Oct 10 17:25:57 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Wed, 10 Oct 2007 14:25:57 -0700
Subject: [Numpy-discussion] How does Python treat commons in fortran
	code?
In-Reply-To: <200710102058.49101.guillemborrell@gmail.com>
References: <200710102058.49101.guillemborrell@gmail.com>
Message-ID: <470D4365.2080707@noaa.gov>

Guillem Borrell i Nogueras wrote:
?How does ctypes or f2py handle the _common_ structures
> present in fortran 77?

I think that's covered (for f2py) by this:

http://cens.ioc.ee/projects/f2py2e/usersguide/#common-blocks


   ?Are all they allocated at load? ?Do they work
> exactly as they were called from a fortran executable and the main program is 
> aware of all the _commons_ and allocates the storage when it is asked for?

I'm pretty sure that's the case.

> He has been programming for more than 40 years and I could not get him much 
> into object oriented programing, however he found python+numpy+scipy a very 
> promising tool.

Good news -- you could save a lot of time by using FORTRAN only for what 
it's good for: the number crunching.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov



From robert.kern at gmail.com  Wed Oct 10 17:33:15 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 10 Oct 2007 16:33:15 -0500
Subject: [Numpy-discussion] fortran 90 compiler problem
In-Reply-To: <200710101651.54591.humufr@yahoo.fr>
References: <200710101504.53064.humufr@yahoo.fr> <470D32E4.5050908@gmail.com>
	<200710101651.54591.humufr@yahoo.fr>
Message-ID: <470D451B.1050506@gmail.com>

humufr at yahoo.fr wrote:
> This is the log I have when I'm trying to use f2py with a fortran 90 file. 
> The last line seems to tell me that f2py cannot see gfortran and g95 even if 
> it detected them when I installed f2py.
> 
> thanks,
> 
> N.
> 
> gfortran and g95 are installed and I can use them:
> 
>> gfortran
> gfortran: no input files
> 
>> gfortran -c hello.f90
>> ls hello*
> hello.f90 hello.o
> 
>> g95 
> g95: no input files
> 
>> g95 -c hello.f90
>> ls hello*
> hello.f90 hello.o
> 
> and now
> 
>> f2py -c -m hello hello.f90

Tell f2py that you want it to use gfortran instead of the default.

  f2py --fcompiler=gnu95 -c -m hello hello.f90

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Wed Oct 10 23:20:34 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 10 Oct 2007 21:20:34 -0600
Subject: [Numpy-discussion] fortran 90 compiler problem
In-Reply-To: <470D32E4.5050908@gmail.com>
References: <200710101504.53064.humufr@yahoo.fr> <470D32E4.5050908@gmail.com>
Message-ID: <e06186140710102020g3a0185aapcb348670e872c5c4@mail.gmail.com>

On 10/10/07, Robert Kern <robert.kern at gmail.com> wrote:
>
> humufr at yahoo.fr wrote:
> >               Hello,
> >
> > I'm using the last svn version of numpy (.
> >
> > I tried the solution found on the mail list:
> >
> > python setup.py config_fc --fcompiler=gnu95 build
> >
> > the two interesting part (at least for me) in the building process are:
> >
> > customize GnuFCompiler Found executable /usr/bin/g77
> > gnu: no Fortran 90 compiler found
> > gnu: no Fortran 90 compiler found
> > customize GnuFCompiler
> > gnu: no Fortran 90 compiler found
> > gnu: no Fortran 90 compiler found
> > customize GnuFCompiler using config
> >
> > and later:
> > customize G95FCompiler Found executable /usr/local/bin/g95
> >
> > but I still have the same answer when I'm trying to do: f2py -c -m hello
> > hello.f90
> >
> > customize GnuFCompiler Found executable /usr/bin/g77
> > gnu: no Fortran 90 compiler found gnu: no Fortran 90
> > compiler found customize GnuFCompiler
> > gnu: no Fortran 90 compiler found
> > gnu: no Fortran 90 compiler found
>
> This is a correct message and does not imply that no Fortran 90 compiler
> exists.
> The GnuFCompiler ("gnu") only looks at g77, which is not a Fortran 90
> compiler.
> That message only applies to the "gnu" compiler, not any other.


I have always found these messages confusing. Perhaps they could be better
worded to reflect the process, or, even better, silenced. All that really
matters is success or failure and the reason, not every little dead end
explored on the way. If they are needed for debugging, write the whole mess
to a file in the build directory instead of the console.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071010/2df6dd12/attachment.html>

From robert.kern at gmail.com  Wed Oct 10 23:28:34 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 10 Oct 2007 22:28:34 -0500
Subject: [Numpy-discussion] fortran 90 compiler problem
In-Reply-To: <e06186140710102020g3a0185aapcb348670e872c5c4@mail.gmail.com>
References: <200710101504.53064.humufr@yahoo.fr> <470D32E4.5050908@gmail.com>
	<e06186140710102020g3a0185aapcb348670e872c5c4@mail.gmail.com>
Message-ID: <470D9862.9070908@gmail.com>

Charles R Harris wrote:
> On 10/10/07, *Robert Kern* <robert.kern at gmail.com
> <mailto:robert.kern at gmail.com>> wrote:
> 
>     humufr at yahoo.fr <mailto:humufr at yahoo.fr> wrote:
>     >               Hello,
>     >
>     > I'm using the last svn version of numpy (.
>     >
>     > I tried the solution found on the mail list:
>     >
>     > python setup.py config_fc --fcompiler=gnu95 build
>     >
>     > the two interesting part (at least for me) in the building process
>     are:
>     >
>     > customize GnuFCompiler Found executable /usr/bin/g77
>     > gnu: no Fortran 90 compiler found
>     > gnu: no Fortran 90 compiler found
>     > customize GnuFCompiler
>     > gnu: no Fortran 90 compiler found
>     > gnu: no Fortran 90 compiler found
>     > customize GnuFCompiler using config
>     >
>     > and later:
>     > customize G95FCompiler Found executable /usr/local/bin/g95
>     >
>     > but I still have the same answer when I'm trying to do: f2py -c -m
>     hello
>     > hello.f90
>     >
>     > customize GnuFCompiler Found executable /usr/bin/g77
>     > gnu: no Fortran 90 compiler found gnu: no Fortran 90
>     > compiler found customize GnuFCompiler
>     > gnu: no Fortran 90 compiler found
>     > gnu: no Fortran 90 compiler found
> 
>     This is a correct message and does not imply that no Fortran 90
>     compiler exists.
>     The GnuFCompiler ("gnu") only looks at g77, which is not a Fortran
>     90 compiler.
>     That message only applies to the "gnu" compiler, not any other.
> 
> 
> I have always found these messages confusing. Perhaps they could be
> better worded to reflect the process, or, even better, silenced. All
> that really matters is success or failure and the reason, not every
> little dead end explored on the way. If they are needed for debugging,
> write the whole mess to a file in the build directory instead of the
> console.

Go for it.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From david at ar.media.kyoto-u.ac.jp  Wed Oct 10 23:55:46 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 11 Oct 2007 12:55:46 +0900
Subject: [Numpy-discussion] For review: first milestone of scons support in
	numpy
Message-ID: <470D9EC2.5000209@ar.media.kyoto-u.ac.jp>

Hi,

    (if you are not interested in numpy developement, you can stop now :) ).
    Following the discussion a few days ago on using scons to build 
extensions in numpy, I have reached a somewhat usable milestone, in the 
numpy.scons branch of numpy, and would like to hear some comments, 
remarks, critics, etc...:

Where to get/see:
-----------------

svn repository : http://svn.scipy.org/svn/numpy/branches/numpy.scons
looking at the code: 
http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons

Examples:
---------

To see how it feels from the package developer point of view, I have put 
three really simple examples:
    - Building a python extension: 
http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/scons_fake/pyext
    - Building a ctypes-based package: 
http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/scons_fake/ctypesext
    - An example on how to check for libraries and symbols in them: 
http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/scons_fake/pyext

For the numpy user, this should be totally transparent (no difference 
when building/installing).

What:
-----

This first milestone implements the following:
    - adding a scons command to numpy.distutils
    - adding an add_sconscript function to numpy.distutils setup, for 
packages willing to use scons
    - two builders: one for ctypes extension, and one for python extension
    - a basic implementation to check for libraries (the paths can be 
overwritten exactly like with distutils, using site.cfg; I have not yet 
implemented overwriting with environment variables).

I have been testing this on the following platforms:
    - linux with gcc
    - linux with icc
    - linux with suncc
    - windows with MS toolikit 2003
    - solaris studio express with suncc
    - mac os X (tiger, x86)

And now ?
---------

As discussed previously, I think numpy would benefit from exclusively 
using scons to build compiled extensions. I have started working on 
fortran support for scons (separate project, since this may be useful to 
all scons users, not just numpy):

https://launchpad.net/numpy.scons.support

and I can already do some non trivial things, not possible with 
numpy.distutils (automatically figuring out fortran mangling, flags for 
linking with C, blas/lapack flags). As expected, this is much more 
robust than distutils approach of hardcoding everything: although I used 
g77 for development, it worked without any change with ifort, gfortran 
and sun fortran compiler (on linux). There are still some issues for 
sure, but I don't see big problems. I don't want to do the work for 
nothing, though, so I would like to know the feeling of numpy developers 
first on this direction, in particular which platforms should work 
before merging consideration, etc...

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Thu Oct 11 00:24:28 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 11 Oct 2007 13:24:28 +0900
Subject: [Numpy-discussion] numpy/Windows shared arrays
	between	processes?
In-Reply-To: <6.2.3.4.2.20071010113132.059f04e0@rjs.org>
References: <mailman.13.1192035604.17671.numpy-discussion@scipy.org>
	<6.2.3.4.2.20071010113132.059f04e0@rjs.org>
Message-ID: <470DA57C.7090001@ar.media.kyoto-u.ac.jp>

Ray S wrote:
> Thanks all:
>
> At 10:00 AM 10/10/2007, Robert Kern wrote:
>  >Something like the following should suffice (untested, though 
> I've >done similar things with ctypes before):
>
> I tested, successfully:
> """ nFromAddress.py
> """
>
> def fromaddress(address, dtype, shape, strides=None):
>      """ Create a numpy array from an integer address, a dtype
>      or dtype string, a shape tuple, and possibly strides.
>      """
>      import numpy
>      # Make sure our dtype is a dtype, not just "f" or whatever.
>      dtype = numpy.dtype(dtype)
>
>      class Dummy(object):
>          pass
>      d = Dummy()
>      d.__array_interface__ = dict(
>          data = (address, False),
>          typestr = dtype.str,
>          descr = dtype.descr,
>          shape = shape,
>          strides = strides,
>          version = 3,
>      )
>      return numpy.asarray(d)
>
> ## Numeric example, with address kludge
> import Numeric, numpy, ctypes, string
> a0 = Numeric.zeros((10000), Numeric.Int16)
> nAddress = int(string.split(repr(a0.__copy__))[-1][:-1], 16)
> tmp=(ctypes.c_long*1)(0)
> ctypes.memmove(tmp, nAddress+8, 4)
> nAddress = tmp[0]
> a1 = fromaddress(nAddress, numpy.int16, (10000,)) ## explicit type
> a0[0] = 5
> print a1[0]
>
> ## numpy example
> a2 = numpy.zeros(10000, numpy.int16)
> nAddress = a2.__array_interface__['data'][0]
> nType = a2.__array_interface__['typestr']
> nShape = a2.__array_interface__['shape']
> a3 = fromaddress(nAddress, nType, nShape)
> a2[0] = 5
> print a3[0]
>
> So, now with little effort the relevant info can be passed over 
> pipes, shared memory, etc. and shares/views created in other 
> processes, since they are not objects but ints and strings.
>
>  >David Cournapeau Wrote:
>  >Basically, you cannot expect file descriptors (or even file 
> handles: >the
>  >standard ones from C library fopen) to cross dll boundaries if 
> the >dll do not have exactly the same runtime.
>
> It sounds like there is a general dilemma: no one with Python 2.4 or 
> 2.5 can reliably expect to compile extensions/modules if they did not 
> install the 7.1 compiler in time.
Well, in theory you could: 'just' recompile python. The problem is not 
the compiler as such, but the C runtime. I don't see any solution to 
this situation, unfortunately; if MS decides to ship a broken libc, it 
is hard to get around that in a portable way.

For files (I don't know any other problems, but this certainly do not 
mean they do not exist), the only way I know is to use the win32 files 
handles. At least, it works in C (I had similar problems when dealing 
with tmp files on win32). To do it directly in python, you may need 
pywin32 specific functions (I cannot remember them on the top of my head).

> The 2.6 seems to use VC 2005 Express, I don't know about py3000(?), 
> with associated upgrade issues.
But what if the next MS compiler has again broken libc implementation ? 
(Incidently, VS2005 was not used for python2.5 for even more broken libc 
than in 2003): 
http://mail.python.org/pipermail/python-dev/2006-April/064126.html

cheers,

David


From ramercer at gmail.com  Thu Oct 11 00:36:44 2007
From: ramercer at gmail.com (Adam Mercer)
Date: Thu, 11 Oct 2007 00:36:44 -0400
Subject: [Numpy-discussion] appending extra items to arrays
Message-ID: <799406d60710102136n3bf3906pd2b6be00472cb1ca@mail.gmail.com>

Hi

In some code I have, I need to append some extra data to a given
array.  At the moment I construct the data in a list, append the extra
information I need and then convert the final list to an array.  Is
there a way that I can append extra information to an existing array
thereby negating the need for the list, as the array object doesn't
seem to have an append() method?

Cheers

Adam


From david at ar.media.kyoto-u.ac.jp  Thu Oct 11 00:35:14 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 11 Oct 2007 13:35:14 +0900
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <799406d60710102136n3bf3906pd2b6be00472cb1ca@mail.gmail.com>
References: <799406d60710102136n3bf3906pd2b6be00472cb1ca@mail.gmail.com>
Message-ID: <470DA802.4010609@ar.media.kyoto-u.ac.jp>

Adam Mercer wrote:
> Hi
>
> In some code I have, I need to append some extra data to a given
> array.  At the moment I construct the data in a list, append the extra
> information I need and then convert the final list to an array.  Is
> there a way that I can append extra information to an existing array
> thereby negating the need for the list, as the array object doesn't
> seem to have an append() method?
>   
Fundamentally, numpy works because its underlying data is in a block of 
memory. This is incompatible with appending data at will, which is the 
exact thing lists are good at.

Now, depending on the problem you are trying to solve, it may be better 
(read faster, more memory efficient, etc...) to use numpy arrays, and 
create new bigger ones when you need to append data. It all depends on 
your problem.

cheers,

David


From oliphant at enthought.com  Thu Oct 11 00:50:37 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 10 Oct 2007 23:50:37 -0500
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <799406d60710102136n3bf3906pd2b6be00472cb1ca@mail.gmail.com>
References: <799406d60710102136n3bf3906pd2b6be00472cb1ca@mail.gmail.com>
Message-ID: <470DAB9D.6010808@enthought.com>

Adam Mercer wrote:
> Hi
>
> In some code I have, I need to append some extra data to a given
> array.  At the moment I construct the data in a list, append the extra
> information I need and then convert the final list to an array.  Is
> there a way that I can append extra information to an existing array
> thereby negating the need for the list, as the array object doesn't
> seem to have an append() method?
>
>   
You can resize arrays (using the resize method), or you can use 
concatenate.  But, depending on the situation, using a list to grow the 
object and then converting at the end may be a better solution than 
resizing a large chunk of memory over and over again.

-Travis



From robert.kern at gmail.com  Thu Oct 11 00:52:33 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 10 Oct 2007 23:52:33 -0500
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <799406d60710102136n3bf3906pd2b6be00472cb1ca@mail.gmail.com>
References: <799406d60710102136n3bf3906pd2b6be00472cb1ca@mail.gmail.com>
Message-ID: <470DAC11.9090205@gmail.com>

Adam Mercer wrote:
> Hi
> 
> In some code I have, I need to append some extra data to a given
> array.  At the moment I construct the data in a list, append the extra
> information I need and then convert the final list to an array.  Is
> there a way that I can append extra information to an existing array
> thereby negating the need for the list, as the array object doesn't
> seem to have an append() method?

Appending to a list then converting the list to an array is the most
straightforward way to do it. If the performance of this isn't a problem, I
recommend leaving it alone.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From peridot.faceted at gmail.com  Thu Oct 11 00:58:25 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Thu, 11 Oct 2007 00:58:25 -0400
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <470DAC11.9090205@gmail.com>
References: <799406d60710102136n3bf3906pd2b6be00472cb1ca@mail.gmail.com>
	<470DAC11.9090205@gmail.com>
Message-ID: <ce557a360710102158p61c5e64cldbda7fffc27b8e8@mail.gmail.com>

On 11/10/2007, Robert Kern <robert.kern at gmail.com> wrote:

> Appending to a list then converting the list to an array is the most
> straightforward way to do it. If the performance of this isn't a problem, I
> recommend leaving it alone.

Just a speculation:

Python strings have a similar problem - they're immutable, and so are
even more resistant to growth than numpy arrays. For those situations
where you really really want to grow a srting, python provides
StringIO, where you keep efficiently adding to the string, then
finalize it to get the real string out. Would something analogous be
interesting for arrays?

Anne

P.S. A (somewhat) slow python implementation would be fairly easy to write. -A


From robert.kern at gmail.com  Thu Oct 11 01:10:49 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 11 Oct 2007 00:10:49 -0500
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <ce557a360710102158p61c5e64cldbda7fffc27b8e8@mail.gmail.com>
References: <799406d60710102136n3bf3906pd2b6be00472cb1ca@mail.gmail.com>	<470DAC11.9090205@gmail.com>
	<ce557a360710102158p61c5e64cldbda7fffc27b8e8@mail.gmail.com>
Message-ID: <470DB059.7020501@gmail.com>

Anne Archibald wrote:
> On 11/10/2007, Robert Kern <robert.kern at gmail.com> wrote:
> 
>> Appending to a list then converting the list to an array is the most
>> straightforward way to do it. If the performance of this isn't a problem, I
>> recommend leaving it alone.
> 
> Just a speculation:
> 
> Python strings have a similar problem - they're immutable, and so are
> even more resistant to growth than numpy arrays. For those situations
> where you really really want to grow a srting, python provides
> StringIO, where you keep efficiently adding to the string, then
> finalize it to get the real string out. Would something analogous be
> interesting for arrays?

The Python version of StringIO, at least, just keeps a list, too. The primary
benefit of StringIO is that it exposes the file interface. For building up a
string/array, using the list is just as convenient (and for arrays, probably
more convenient given the variety of shapes one might want to build) and at
least as readable.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From humufr at yahoo.fr  Thu Oct 11 01:14:55 2007
From: humufr at yahoo.fr (humufr at yahoo.fr)
Date: Thu, 11 Oct 2007 01:14:55 -0400
Subject: [Numpy-discussion] fortran 90 compiler problem
In-Reply-To: <470D451B.1050506@gmail.com>
References: <200710101504.53064.humufr@yahoo.fr>
	<200710101651.54591.humufr@yahoo.fr> <470D451B.1050506@gmail.com>
Message-ID: <200710110114.55905.humufr@yahoo.fr>

It's working fine thank you. I'm a little bit surprised because I was thinking 
that the --fcompiler when building numpy was the way to change the default 
compiler but anyway, that's work now.

N.
>
> Tell f2py that you want it to use gfortran instead of the default.
>
>   f2py --fcompiler=gnu95 -c -m hello hello.f90


From matthieu.brucher at gmail.com  Thu Oct 11 03:27:39 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 11 Oct 2007 09:27:39 +0200
Subject: [Numpy-discussion] numpy/Windows shared arrays between
	processes?
In-Reply-To: <470DA57C.7090001@ar.media.kyoto-u.ac.jp>
References: <mailman.13.1192035604.17671.numpy-discussion@scipy.org>
	<6.2.3.4.2.20071010113132.059f04e0@rjs.org>
	<470DA57C.7090001@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0710110027ib729996g6a161993f47d592a@mail.gmail.com>

>
> > The 2.6 seems to use VC 2005 Express, I don't know about py3000(?),
> > with associated upgrade issues.
> But what if the next MS compiler has again broken libc implementation ?
> (Incidently, VS2005 was not used for python2.5 for even more broken libc
> than in 2003):
> http://mail.python.org/pipermail/python-dev/2006-April/064126.html
>

I don't what he meant by a broken libc, if it is the fact that there is a
lot of deprecated standard functions, I don't call it broken (besides, this
deprecation follows a technical paper that describe the new safe functions,
although it does not deprecate these functions).

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071011/97f06bf6/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Thu Oct 11 03:38:34 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 11 Oct 2007 16:38:34 +0900
Subject: [Numpy-discussion] numpy/Windows shared arrays
	between	processes?
In-Reply-To: <e76aa17f0710110027ib729996g6a161993f47d592a@mail.gmail.com>
References: <mailman.13.1192035604.17671.numpy-discussion@scipy.org>	<6.2.3.4.2.20071010113132.059f04e0@rjs.org>	<470DA57C.7090001@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710110027ib729996g6a161993f47d592a@mail.gmail.com>
Message-ID: <470DD2FA.3070507@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
>     > The 2.6 seems to use VC 2005 Express, I don't know about py3000(?),
>     > with associated upgrade issues.
>     But what if the next MS compiler has again broken libc
>     implementation ?
>     (Incidently, VS2005 was not used for python2.5 for even more
>     broken libc
>     than in 2003):
>     http://mail.python.org/pipermail/python-dev/2006-April/064126.html
>
>
> I don't what he meant by a broken libc, if it is the fact that there 
> is a lot of deprecated standard functions, I don't call it broken 
> (besides, this deprecation follows a technical paper that describe the 
> new safe functions, although it does not deprecate these functions).
If unilaterally deprecating standard functions which exist for years is 
not broken, I really wonder what is :)

cheers,

David


From pearu at cens.ioc.ee  Thu Oct 11 04:00:09 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Thu, 11 Oct 2007 10:00:09 +0200
Subject: [Numpy-discussion] For review: first milestone of scons support
 in	numpy
In-Reply-To: <470D9EC2.5000209@ar.media.kyoto-u.ac.jp>
References: <470D9EC2.5000209@ar.media.kyoto-u.ac.jp>
Message-ID: <470DD809.5050308@cens.ioc.ee>

Hi,

Examples look good. It seems that you have lots of work ahead;) to add
numpy.distutils features that are required to build numpy/scipy.

Few comments:
1) Why SConstruct does not have extension? It looks like a python file
and .py extension could be used.

2) It seems that scons does not interfare with numpy.distutils much.
If this is true and numpy/scipy builds will not break when scons is
not installed then I think you could continue the scons support
development in trunk.

3) In future, if we are going to replace using distutils with scons
then all numpy/scipy need SConstruct scripts. I think one can implement
these scripts already now and use, say, setupscons.py, containing only

def configuration(parent_package='',top_path=None):
     from numpy.distutils.misc_util import Configuration
     config = Configuration('packagename',parent_package,top_path)
     config.add_sconscript('SConstruct')
     return config
if __name__ == '__main__':
     from numpy.distutils.core import setup
     setup(configuration=configuration)

to build the packages. Or can scons already be used with only
SConstruct script and setupscons.py are not needed? Implementing these 
scripts now would give a good idea what features are required in using
scons to build numpy/scipy packages.
Also, it will prove that scons can replace numpy.distutils in future,

4) though, we cannot remove numpy.distutils for backward compatibility
with software using numpy.distutils. However, it should be possible
to enhance Configuration class to generate the corresponding SConstruct
scripts. It will save some work when converting configuration() 
functions to SConstruct scripts.

Looking forward for not using distutils,
Pearu


From david at ar.media.kyoto-u.ac.jp  Thu Oct 11 04:54:13 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 11 Oct 2007 17:54:13 +0900
Subject: [Numpy-discussion] For review: first milestone of scons support
 in	numpy
In-Reply-To: <470DD809.5050308@cens.ioc.ee>
References: <470D9EC2.5000209@ar.media.kyoto-u.ac.jp>
	<470DD809.5050308@cens.ioc.ee>
Message-ID: <470DE4B5.3080007@ar.media.kyoto-u.ac.jp>

Pearu Peterson wrote:
> Hi,
>
> Examples look good. It seems that you have lots of work ahead;) to add
> numpy.distutils features that are required to build numpy/scipy.
>
Hi Pearu,

    Thanks for reviewing this, especially since you are arguably the 
most knowledgeable about this part of numpy :)
> Few comments:
> 1) Why SConstruct does not have extension? It looks like a python file
> and .py extension could be used.
>
It looks like but isn't. This is actually the part of scons I like the 
least: scons uses python, but sconscripts files are not proper python  
modules. Sconscripts files are used declaratively:

http://www.scons.org/doc/HTML/scons-user/x348.html

I think this is one of the biggest design mistake of scons. Waf, which 
was briefly mentioned by David M. Cooke, and which is based on scons, 
does not do this, though (I did not use waf because it is new, is 
maintained only by one developer, and is more geared toward unix, that 
is MS tools are not supported).


> 2) It seems that scons does not interfare with numpy.distutils much.
> If this is true and numpy/scipy builds will not break when scons is
> not installed then I think you could continue the scons support
> development in trunk.
It won't break if scons is not installed because scons sources are 
copied into the branch. Scons developers explicitely support this:

http://www.scons.org/faq.php#SS_3_3

(AFAIK, it does not pose any problem license-wise, since scons is new 
BSD license; it adds ~350 kb of compressed source code to numpy).
>
> 3) In future, if we are going to replace using distutils with scons
> then all numpy/scipy need SConstruct scripts. I think one can implement
> these scripts already now and use, say, setupscons.py, containing only
>
> def configuration(parent_package='',top_path=None):
>      from numpy.distutils.misc_util import Configuration
>      config = Configuration('packagename',parent_package,top_path)
>      config.add_sconscript('SConstruct')
>      return config
> if __name__ == '__main__':
>      from numpy.distutils.core import setup
>      setup(configuration=configuration)
>
> to build the packages. Or can scons already be used with only
> SConstruct script and setupscons.py are not needed? Implementing these 
> scripts now would give a good idea what features are required in using
> scons to build numpy/scipy packages.
> Also, it will prove that scons can replace numpy.distutils in future,
That's what I had in mind (setupscons.py). I don't see any big reasons 
for not merging the current branch (since it is optional and should not 
break anything).

Now, concerning migrating all compiled extensions to scons, I would 
prefer avoiding doing it in the trunk; but I heard horror stories about 
subversion and merging, so maybe staying outside the trunk is too risky 
? Also, when I said that I don't see big problems to replace distutils 
for compiled extensions, I lied by over-simplification. If scons is used 
for compiled extension, with the current design, distutils will call 
scons for each package. Calling scons is expensive (importing many 
modules, etc...: this easily takes around one second for each non 
trivial Sconscript), and also, because each of them is independent, we 
may check several times for the same thing (fortran compiler, etc...), 
which would really add up to the build time.

I see two approaches here:
    - do not care about it because numpy is unlikely to become really bigger
    - considering that numpy is really one big package (contrary to 
scipy which, at least in my understanding, is gearing toward less 
inter-package dependencies ?), we should only have one big sconscript 
for configuration (checking blas/lapack, compilers, etc...), and use 
scons recursively. In this case, it should not be much slower than the 
current system.
>
> 4) though, we cannot remove numpy.distutils for backward compatibility
> with software using numpy.distutils. However, it should be possible
> to enhance Configuration class to generate the corresponding SConstruct
> scripts. It will save some work when converting configuration() 
> functions to SConstruct scripts.
Could you elaborate on this point ? I am not sure what you mean by 
generating Configuration class ?

cheers,

David


From pearu at cens.ioc.ee  Thu Oct 11 05:45:18 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Thu, 11 Oct 2007 11:45:18 +0200
Subject: [Numpy-discussion] For review: first milestone of scons support
 in	numpy
In-Reply-To: <470DE4B5.3080007@ar.media.kyoto-u.ac.jp>
References: <470D9EC2.5000209@ar.media.kyoto-u.ac.jp>	<470DD809.5050308@cens.ioc.ee>
	<470DE4B5.3080007@ar.media.kyoto-u.ac.jp>
Message-ID: <470DF0AE.2030206@cens.ioc.ee>



David Cournapeau wrote:
> Pearu Peterson wrote:
>> 2) It seems that scons does not interfare with numpy.distutils much.
>> If this is true and numpy/scipy builds will not break when scons is
>> not installed then I think you could continue the scons support
>> development in trunk.
> It won't break if scons is not installed because scons sources are 
> copied into the branch. Scons developers explicitely support this:
> 
> http://www.scons.org/faq.php#SS_3_3
> 
> (AFAIK, it does not pose any problem license-wise, since scons is new 
> BSD license; it adds ~350 kb of compressed source code to numpy).

I think this is good.
Does scons require python-dev? If not then this will solve one of the
frequent issues that new users may experience: not installed distutils.

> Now, concerning migrating all compiled extensions to scons, I would 
> prefer avoiding doing it in the trunk; but I heard horror stories about 
> subversion and merging, so maybe staying outside the trunk is too risky 
> ?

I think numpy is quite stable now that it's safe to develop in a branch
(if trunk is very actively developed then merging branches
can be a nightmare). However, IMHO using a branch makes other
developers to stay aside from branch development and in time it is
more and more difficult to merge.

> Also, when I said that I don't see big problems to replace distutils 
> for compiled extensions, I lied by over-simplification. If scons is used 
> for compiled extension, with the current design, distutils will call 
> scons for each package. Calling scons is expensive (importing many 
> modules, etc...: this easily takes around one second for each non 
> trivial Sconscript), and also, because each of them is independent, we 
> may check several times for the same thing (fortran compiler, etc...), 
> which would really add up to the build time.
 >
> I see two approaches here:
>     - do not care about it because numpy is unlikely to become really bigger
>     - considering that numpy is really one big package (contrary to 
> scipy which, at least in my understanding, is gearing toward less 
> inter-package dependencies ?), we should only have one big sconscript 
> for configuration (checking blas/lapack, compilers, etc...), and use 
> scons recursively. In this case, it should not be much slower than the 
> current system.

Note that building a subpackage in subpackage directory must be 
supported. So a big sconscript may not be an option.

The third approach would be to cache those checks that are called
frequently to, say, $HOME/.numpy/scons.

numpy.distutils setup.py script gathers the information from
subpackage setup.py scripts recursively and then passes all the
information to one setup function call.
I think setupscons.py should do the same. If scons does not support
recursive reading of scons scripts then the corresponding feature
should be implemented, I guess it would not be difficult.

>> 4) though, we cannot remove numpy.distutils for backward compatibility
>> with software using numpy.distutils. However, it should be possible
>> to enhance Configuration class to generate the corresponding SConstruct
>> scripts. It will save some work when converting configuration() 
>> functions to SConstruct scripts.
> Could you elaborate on this point ? I am not sure what you mean by 
> generating Configuration class ?

I meant that Configuration class could have a method, say 
toscons(<filename>),  that will generate SConstruct script
from the information that Configuration instance holds. I thought that 
this would just ease creating SConstruct scripts from
existing setup.py files.

Pearu


From david at ar.media.kyoto-u.ac.jp  Thu Oct 11 06:12:22 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 11 Oct 2007 19:12:22 +0900
Subject: [Numpy-discussion] For review: first milestone of scons support
 in	numpy
In-Reply-To: <470DF0AE.2030206@cens.ioc.ee>
References: <470D9EC2.5000209@ar.media.kyoto-u.ac.jp>	<470DD809.5050308@cens.ioc.ee>	<470DE4B5.3080007@ar.media.kyoto-u.ac.jp>
	<470DF0AE.2030206@cens.ioc.ee>
Message-ID: <470DF706.90309@ar.media.kyoto-u.ac.jp>

Pearu Peterson wrote:
>
> I think this is good.
> Does scons require python-dev? If not then this will solve one of the
> frequent issues that new users may experience: not installed distutils.
Isn't distutils included in python library ? Anyway, scons does not 
require anything else than a python interpreter. Actually, an explicit 
requirement of scons is to support any python starting at version 1.5.2 
(this is another important point which I consider important for a 
replacement of numpy.distutils).

But please remember that this work is not intended at replacing 
distutils entirely, only the build_ext/build_lib commands. All the build 
process would still be driven by distutils (in particular, supporting 
distutils is a requirement to support setuptools and eggs).
> I think numpy is quite stable now that it's safe to develop in a branch
> (if trunk is very actively developed then merging branches
> can be a nightmare). However, IMHO using a branch makes other
> developers to stay aside from branch development and in time it is
> more and more difficult to merge.
I don't have strong experience in subversion, so I was afraid of that. 
Do I understand correctly that you suggest opening a new dev branch, and 
then do all subsequent dev (including non distutils/scons related ones) 
there ?

>
> The third approach would be to cache those checks that are called
> frequently to, say, $HOME/.numpy/scons.
>
> numpy.distutils setup.py script gathers the information from
> subpackage setup.py scripts recursively and then passes all the
> information to one setup function call.
> I think setupscons.py should do the same. If scons does not support
> recursive reading of scons scripts then the corresponding feature
> should be implemented, I guess it would not be difficult.
Scons supports recursive calls. To be more clear about the possibilities 
of scons wrt to this, let's take a simplified example:

root/Sconstruct
root/numpy/Sconscript
root/numpy/core/Sconscript
root/numpy/linalg/Sconscript

If you are in root and call scons (> is a shell prompt):

root > scons

Then it will call recursively all the sconscript (as long as you request 
it in the sconscript files). The other supported method is

root/numpy/linalg > scons -u

this will look every parent directory to find the "root" Sconstruct. So 
as long as we configure everything (by everything, I mean checking for 
libraries and compilers) in the root Sconscript, this should work. To 
sum it up: the build process would be more like projects using 
autotools; a configure step, and a build step (this would be internal; 
there is no need to expose this mechanism to the user).
>
> I meant that Configuration class could have a method, say 
> toscons(<filename>),  that will generate SConstruct script
> from the information that Configuration instance holds. I thought that 
> this would just ease creating SConstruct scripts from
> existing setup.py files.
I don't think it would worth the effort for numpy (the main work really 
is to implement and test all the checkers: blas/lapack, fortran). Now, 
as a general migration tool, this may be useful. But since we would 
still use distutils, it would only be useful if it is easy to develop 
such as tool.

cheers,

David

P.S: would it be easy for you to make a list of requirements for fortran 
? By requirement, I mean things like name mangling and so on ? Something 
like the autoconf macros: 
http://www.gnu.org/software/autoconf/manual/autoconf-2.57/html_node/autoconf_66.html


From matthieu.brucher at gmail.com  Thu Oct 11 07:09:04 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Thu, 11 Oct 2007 13:09:04 +0200
Subject: [Numpy-discussion] numpy/Windows shared arrays between
	processes?
In-Reply-To: <470DD2FA.3070507@ar.media.kyoto-u.ac.jp>
References: <mailman.13.1192035604.17671.numpy-discussion@scipy.org>
	<6.2.3.4.2.20071010113132.059f04e0@rjs.org>
	<470DA57C.7090001@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710110027ib729996g6a161993f47d592a@mail.gmail.com>
	<470DD2FA.3070507@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0710110409k22b03990qe443154c5bbc7140@mail.gmail.com>

>
> > I don't what he meant by a broken libc, if it is the fact that there
> > is a lot of deprecated standard functions, I don't call it broken
> > (besides, this deprecation follows a technical paper that describe the
> > new safe functions, although it does not deprecate these functions).
> If unilaterally deprecating standard functions which exist for years is
> not broken, I really wonder what is :)
>

They are deprecated (although a simple flag can get rid of those
deprecation) not removed. Besides, the deprecated functions are in fact
functions that can lead to security issues (for the first time Microsoft did
something not completely stupid about this topic), so telling that the
programmer should not use them but more secure one may be seen as a good
idea (from a certain point of view).

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071011/2aaf6aa6/attachment.html>

From pearu at cens.ioc.ee  Thu Oct 11 07:25:11 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Thu, 11 Oct 2007 13:25:11 +0200
Subject: [Numpy-discussion] For review: first milestone of scons support
 in	numpy
In-Reply-To: <470DF706.90309@ar.media.kyoto-u.ac.jp>
References: <470D9EC2.5000209@ar.media.kyoto-u.ac.jp>	<470DD809.5050308@cens.ioc.ee>	<470DE4B5.3080007@ar.media.kyoto-u.ac.jp>	<470DF0AE.2030206@cens.ioc.ee>
	<470DF706.90309@ar.media.kyoto-u.ac.jp>
Message-ID: <470E0817.10909@cens.ioc.ee>

David Cournapeau wrote:
> Pearu Peterson wrote:
>> I think this is good.
>> Does scons require python-dev? If not then this will solve one of the
>> frequent issues that new users may experience: not installed distutils.
> Isn't distutils included in python library ?

Not always. For example, in debian one has to install python-dev 
separately from python.

> Anyway, scons does not 
> require anything else than a python interpreter. Actually, an explicit 
> requirement of scons is to support any python starting at version 1.5.2 
> (this is another important point which I consider important for a 
> replacement of numpy.distutils).

Though, it is irrelevant as numpy/scipy packages require python
versions starting at 2.3.

>> I think numpy is quite stable now that it's safe to develop in a branch
>> (if trunk is very actively developed then merging branches
>> can be a nightmare). However, IMHO using a branch makes other
>> developers to stay aside from branch development and in time it is
>> more and more difficult to merge.
> I don't have strong experience in subversion, so I was afraid of that. 
> Do I understand correctly that you suggest opening a new dev branch, and 
> then do all subsequent dev (including non distutils/scons related ones) 
> there ?

No, my original suggestion was that I don't mind if you would develop
scons support in trunk as it does not affect the current state
of numpy/scipy builds. Don't know if other developers would have
objections in that.

>> numpy.distutils setup.py script gathers the information from
>> subpackage setup.py scripts recursively and then passes all the
>> information to one setup function call.
>> I think setupscons.py should do the same. If scons does not support
>> recursive reading of scons scripts then the corresponding feature
>> should be implemented, I guess it would not be difficult.
> Scons supports recursive calls. To be more clear about the possibilities 
> of scons wrt to this, let's take a simplified example:
> 
> root/Sconstruct
> root/numpy/Sconscript
> root/numpy/core/Sconscript
> root/numpy/linalg/Sconscript
> 
> If you are in root and call scons (> is a shell prompt):
> 
> root > scons
> 
> Then it will call recursively all the sconscript (as long as you request 
> it in the sconscript files). The other supported method is
> 
> root/numpy/linalg > scons -u
> 
> this will look every parent directory to find the "root" Sconstruct.

My point was that

   root/numpy/linalg > scons

should work (without the -u option). A subpackage may not require all
the stuff that other subpackages require and therefore scons should
not configure everything - it's a waste of time and efforts - especially
if something is broken in upper packages but not in the subpackage.

>> I meant that Configuration class could have a method, say 
>> toscons(<filename>),  that will generate SConstruct script
>> from the information that Configuration instance holds. I thought that 
>> this would just ease creating SConstruct scripts from
>> existing setup.py files.
> I don't think it would worth the effort for numpy (the main work really 
> is to implement and test all the checkers: blas/lapack, fortran). Now, 
> as a general migration tool, this may be useful. But since we would 
> still use distutils, it would only be useful if it is easy to develop 
> such as tool.

Yes, that's a low priority feature.

> P.S: would it be easy for you to make a list of requirements for fortran 
> ? By requirement, I mean things like name mangling and so on ? Something 
> like the autoconf macros: 
> http://www.gnu.org/software/autoconf/manual/autoconf-2.57/html_node/autoconf_66.html

To use f2py succesfully, a fortran compiler must support flags that make
fortran symbol names lowercase and with exactly one underscore at the 
end of a name. This is required when using numpy.distutils.

f2py generated modules make use of the following CPP macros:
-DPREPEND_FORTRAN -DNO_APPEND_FORTRAN -DUPPERCASE_FORTRAN -DUNDERSCORE_G77
and therefore the above requirement would not be needed if
scons could figure out how some particular compiler mangles
the names of fortran symbols. This would be especially useful
since some fortran compiler vendors change the compiler flags
between compiler versions and one has to update numpy.distutils
files accordingly.

Note that using hardcoded name mangeling flags may be still required
for certian Fortran 90 compilers (which ones exactly, I don't member 
now) that by default produce symbol names with special characters like $ 
or . for F90 modules and making these names unaccessible to C programs.

Pearu


From tim.hochberg at ieee.org  Thu Oct 11 10:02:04 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Thu, 11 Oct 2007 07:02:04 -0700
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <ce557a360710102158p61c5e64cldbda7fffc27b8e8@mail.gmail.com>
References: <799406d60710102136n3bf3906pd2b6be00472cb1ca@mail.gmail.com>
	<470DAC11.9090205@gmail.com>
	<ce557a360710102158p61c5e64cldbda7fffc27b8e8@mail.gmail.com>
Message-ID: <e4412d6b0710110702y25c5f0fem48148f724b2e9d41@mail.gmail.com>

On 10/10/07, Anne Archibald <peridot.faceted at gmail.com> wrote:
>
> On 11/10/2007, Robert Kern <robert.kern at gmail.com> wrote:
>
> > Appending to a list then converting the list to an array is the most
> > straightforward way to do it. If the performance of this isn't a
> problem, I
> > recommend leaving it alone.
>
> Just a speculation:
>
> Python strings have a similar problem - they're immutable, and so are
> even more resistant to growth than numpy arrays. For those situations
> where you really really want to grow a srting, python provides
> StringIO, where you keep efficiently adding to the string, then
> finalize it to get the real string out. Would something analogous be
> interesting for arrays?


Have you looked at array.array?

>>> import array
>>> a = array.array('i')
>>> a.append(2)
>>> a.append(7)
>>> a
array('i', [2, 7])




-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071011/34b2f572/attachment.html>

From ramercer at gmail.com  Thu Oct 11 10:41:45 2007
From: ramercer at gmail.com (Adam Mercer)
Date: Thu, 11 Oct 2007 10:41:45 -0400
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <470DAC11.9090205@gmail.com>
References: <799406d60710102136n3bf3906pd2b6be00472cb1ca@mail.gmail.com>
	<470DAC11.9090205@gmail.com>
Message-ID: <799406d60710110741k1937e8a9te6887d200e28dc7c@mail.gmail.com>

On 11/10/2007, Robert Kern <robert.kern at gmail.com> wrote:

> Appending to a list then converting the list to an array is the most
> straightforward way to do it. If the performance of this isn't a problem, I
> recommend leaving it alone.

Thanks, I'll leave it as is - I was just wondering if there was a
better way to do it.

Cheers

Adam


From Norbert.Nemec.list at gmx.de  Thu Oct 11 11:34:41 2007
From: Norbert.Nemec.list at gmx.de (Norbert Nemec)
Date: Thu, 11 Oct 2007 17:34:41 +0200
Subject: [Numpy-discussion] diff of find of diff trick
In-Reply-To: <b14b627d0710091055x5d9f844dw4f8e065f72c1b14e@mail.gmail.com>
References: <b14b627d0710091055x5d9f844dw4f8e065f72c1b14e@mail.gmail.com>
Message-ID: <470E4291.60000@gmx.de>

Basically, what you want to do is a histogram. Numpy has that
functionality built in. However: the version built in to numpy is about
as suboptimal as yours. The problem is the unnecessary sorting of the data.

In principle, a histogram does not need any sorting, so it could be done
in strictly O(N). I don't see any simply way of doing so efficiently in
numpy without resorting to a costly loop. We had a discussion about this
a while ago but I cannot remember any resolution. The proper thing to do
would probably be a C routine.

If performance is crucial to you, I would suggest using weave. A minimal
example as starting point is given below.

Greetings,
Norbert

---------------------------
#!/usr/bin/env python

import numpy
import scipy.weave
import scipy.weave.converters

def histogram(data, nR):
    res = numpy.zeros(nR,int)
    dataflat = data.flatten()
    Ndata = len(dataflat)
    scipy.weave.inline("""
        for(int i=0;i<Ndata;i++) {
            int d = dataflat(i);
            if(d < nR && d >= 0) {
                res(d) += 1;
            }
        }
        """,
        arg_names = ["dataflat","res","Ndata","nR"],
        type_converters = scipy.weave.converters.blitz,
    )
    return res



a = numpy.array([0,3,1,2,3,5,2,3,2,4,5,1,3,4,2])
print a
print histogram(a,4)
---------------------------





Robin wrote:
> Hello,
>
> As a converting MATLAB user I am bringing over some of my code. A core
> function I need is to sample the probability distribution of a given
> set of data. Sometimes these sets are large so I would like to make it
> as efficient as possible. (the data are integers representing members
> of a discrete space)
>
> In MATLAB the best way I found was the "diff of find of diff" trick
> which resulted in the completely vectorised solution (below). Does it
> make sense to translate this into numpy? I don't have a feel yet for
> what is fast/slow - are the functions below built in and so quick (I
> thought diff might not be).
>
> Is there a better/more pythonic way to do it?
>
> --------
> function Pr=prob(data, nR)
>
> Pr    = zeros(nR,1);
> % diff of find of diff trick for counting number of elements
> temp     = sort(data(data>0));    % want vector excluding P(0)
> dtemp    = diff([temp;max(temp)+1]);
> count     = diff(find([1;dtemp]));
> indx     = temp(dtemp>0);
> Pr(indx)= count ./ numel(data);    % probability
> --------
>
> Thanks
>
> Robin
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From mjanikas at esri.com  Thu Oct 11 12:42:02 2007
From: mjanikas at esri.com (Mark Janikas)
Date: Thu, 11 Oct 2007 09:42:02 -0700
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <799406d60710110741k1937e8a9te6887d200e28dc7c@mail.gmail.com>
Message-ID: <627102C921CD9745B070C3B10CB8199B049D53CB@hardwire.esri.com>

If you do not know the size of your array before you finalize it, then
you should use lists whenever you can.  I just cooked up a short
example:

######################################################################
import timeit
import numpy as N

values = range(10000)

def appendArray(values):
    result = N.array([], dtype="int")
    for value in values:
        result = N.append(result, value)
    return result

def appendList(values):
    result = []
    for value in values:
        result.append(value)
    return N.array(result)

test = timeit.Timer('appendArray(values)',
        'from __main__ import appendArray, values')
t1 = test.timeit(number=10)

test2 = timeit.Timer('appendList(values)',
        'from __main__ import appendList, values')
t2 = test2.timeit(number=10)

print "Total Time with array: " + str(t1)
print "Total Time with list: " + str(t2)

##### Result #####
Total Time with array: 2.12951189331
Total Time with list: 0.0469707035741
########################################################################
####

Hope this helps,

MJ

-----Original Message-----
From: numpy-discussion-bounces at scipy.org
[mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Adam Mercer
Sent: Thursday, October 11, 2007 7:42 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] appending extra items to arrays

On 11/10/2007, Robert Kern <robert.kern at gmail.com> wrote:

> Appending to a list then converting the list to an array is the most
> straightforward way to do it. If the performance of this isn't a
problem, I
> recommend leaving it alone.

Thanks, I'll leave it as is - I was just wondering if there was a
better way to do it.

Cheers

Adam
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion



From robert.kern at gmail.com  Thu Oct 11 13:36:07 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 11 Oct 2007 12:36:07 -0500
Subject: [Numpy-discussion] For review: first milestone of scons support
 in	numpy
In-Reply-To: <470E0817.10909@cens.ioc.ee>
References: <470D9EC2.5000209@ar.media.kyoto-u.ac.jp>	<470DD809.5050308@cens.ioc.ee>	<470DE4B5.3080007@ar.media.kyoto-u.ac.jp>	<470DF0AE.2030206@cens.ioc.ee>	<470DF706.90309@ar.media.kyoto-u.ac.jp>
	<470E0817.10909@cens.ioc.ee>
Message-ID: <470E5F07.7010601@gmail.com>

Pearu Peterson wrote:
> David Cournapeau wrote:
>> Pearu Peterson wrote:
>>> I think this is good.
>>> Does scons require python-dev? If not then this will solve one of the
>>> frequent issues that new users may experience: not installed distutils.
>> Isn't distutils included in python library ?
> 
> Not always. For example, in debian one has to install python-dev 
> separately from python.

You'd still need python-dev(el) for the headers if not distutils.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From ramercer at gmail.com  Thu Oct 11 14:53:31 2007
From: ramercer at gmail.com (Adam Mercer)
Date: Thu, 11 Oct 2007 14:53:31 -0400
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <627102C921CD9745B070C3B10CB8199B049D53CB@hardwire.esri.com>
References: <799406d60710110741k1937e8a9te6887d200e28dc7c@mail.gmail.com>
	<627102C921CD9745B070C3B10CB8199B049D53CB@hardwire.esri.com>
Message-ID: <799406d60710111153w496563cbwe65f0aaffd0d7d1@mail.gmail.com>

On 11/10/2007, Mark Janikas <mjanikas at esri.com> wrote:
> If you do not know the size of your array before you finalize it, then
> you should use lists whenever you can.  I just cooked up a short
> example:

<snip>

> ##### Result #####
> Total Time with array: 2.12951189331
> Total Time with list: 0.0469707035741
> ########################################################################
> ####
>
> Hope this helps,

That is helpful, I thought that using arrays would be much faster but
its clearly not in this case.

Thanks

Adam


From aisaac at american.edu  Thu Oct 11 20:36:47 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Thu, 11 Oct 2007 20:36:47 -0400
Subject: [Numpy-discussion] numpy.distutils.cpuinfo bugs?
In-Reply-To: <20071002192054.GA30238@arbutus.physics.mcmaster.ca>
References: <Mahogany-0.67.0-1068-20071001-214026.00@american.edu><20071002192054.GA30238@arbutus.physics.mcmaster.ca>
Message-ID: <Mahogany-0.67.0-1524-20071011-203647.00@american.edu>

On Tue, 2 Oct 2007, "David M. Cooke" apparently wrote:
> Should be fixed now. 

The update seems to work for all my students who
were having problems.

Thank you,
Alan Isaac






From david at ar.media.kyoto-u.ac.jp  Thu Oct 11 23:02:36 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 12 Oct 2007 12:02:36 +0900
Subject: [Numpy-discussion] [OT]numpy/Windows shared arrays
	between	processes?
In-Reply-To: <e76aa17f0710110409k22b03990qe443154c5bbc7140@mail.gmail.com>
References: <mailman.13.1192035604.17671.numpy-discussion@scipy.org>	<6.2.3.4.2.20071010113132.059f04e0@rjs.org>	<470DA57C.7090001@ar.media.kyoto-u.ac.jp>	<e76aa17f0710110027ib729996g6a161993f47d592a@mail.gmail.com>	<470DD2FA.3070507@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710110409k22b03990qe443154c5bbc7140@mail.gmail.com>
Message-ID: <470EE3CC.5080500@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
>     > I don't what he meant by a broken libc, if it is the fact that there
>     > is a lot of deprecated standard functions, I don't call it broken
>     > (besides, this deprecation follows a technical paper that
>     describe the
>     > new safe functions, although it does not deprecate these functions).
>     If unilaterally deprecating standard functions which exist for
>     years is
>     not broken, I really wonder what is :)
>
>
> They are deprecated (although a simple flag can get rid of those 
> deprecation) not removed. 
deprecated means that they will disappear soon, and (worse), people will 
chose to use the non deprecated, hence programming non portable code 
(those functions are neither portable or standard; different OS have 
different implementation).
> Besides, the deprecated functions are in fact functions that can lead 
> to security issues 
This is far from obvious. Actually, numerous people (who I trust more 
than MS) have raised their concern on those functions: they add 
complexity for no obvious advantage.

http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1106.txt

cheers,

Ddavid


From david at ar.media.kyoto-u.ac.jp  Thu Oct 11 23:34:56 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 12 Oct 2007 12:34:56 +0900
Subject: [Numpy-discussion] For review: first milestone of scons support
 in	numpy
In-Reply-To: <470E0817.10909@cens.ioc.ee>
References: <470D9EC2.5000209@ar.media.kyoto-u.ac.jp>	<470DD809.5050308@cens.ioc.ee>	<470DE4B5.3080007@ar.media.kyoto-u.ac.jp>	<470DF0AE.2030206@cens.ioc.ee>	<470DF706.90309@ar.media.kyoto-u.ac.jp>
	<470E0817.10909@cens.ioc.ee>
Message-ID: <470EEB60.3090800@ar.media.kyoto-u.ac.jp>

Pearu Peterson wrote:
>
>> Anyway, scons does not 
>> require anything else than a python interpreter. Actually, an explicit 
>> requirement of scons is to support any python starting at version 1.5.2 
>> (this is another important point which I consider important for a 
>> replacement of numpy.distutils).
>
> Though, it is irrelevant as numpy/scipy packages require python
> versions starting at 2.3.
Sure, I just meant that we don't have to worry that scons may depend on 
2.4 or 2.5 features (I am not sure waf have such a requirement, for 
example).
>
> No, my original suggestion was that I don't mind if you would develop
> scons support in trunk as it does not affect the current state
> of numpy/scipy builds. Don't know if other developers would have
> objections in that.
Ah, ok. I am not sure I can guarantee it will never affect the build 
process. As this is a pretty fundamental change, I though about doing 
that in a 1.1 branch of numpy. But I do not have strong opinion on that, 
nor much experience in that area.
>
> My point was that
>
>    root/numpy/linalg > scons
>
> should work (without the -u option). 
Since scons is called by distutils, and never by the user, I don't see 
how this can be a problem ? distutils would have to figure out when to 
use -u, not the user.
> A subpackage may not require all
> the stuff that other subpackages require and therefore scons should
> not configure everything - it's a waste of time and efforts - especially
> if something is broken in upper packages but not in the subpackage.
>
Note that scons caches the configuration, so I don't think speed will be 
an issue here (except maybe if you keep changing the configuration 
before building). What I could do is for each subpackage, to declare the 
tests to use in each subpackage, but then, what to do if two packages 
have some common tests ? I cannot just remove double tests, because the 
order is significant (because of link options, for example).

The way I see it, either we keep the current behaviour (each package is 
totally independant, scons is called for each subpackage, and scons has 
no way to know about other subpackages; this has the disadvantage of 
being slower: this will be significant of many subpackages use costly 
checks like fortran mangling and so on), or we have a main sconscript 
with the configuration, which does not prevent building subpackages, but 
which requires a global configuration.

If none of those approach seems right to you, I will see if I can come 
up with something better, but this will certainly add some complexity 
(or I am just stupid to see an obvious solution :) ).
> To use f2py succesfully, a fortran compiler must support flags that make
> fortran symbol names lowercase and with exactly one underscore at the 
> end of a name. This is required when using numpy.distutils.
>
> f2py generated modules make use of the following CPP macros:
> -DPREPEND_FORTRAN -DNO_APPEND_FORTRAN -DUPPERCASE_FORTRAN -DUNDERSCORE_G77
> and therefore the above requirement would not be needed if
> scons could figure out how some particular compiler mangles
> the names of fortran symbols. This would be especially useful
> since some fortran compiler vendors change the compiler flags
> between compiler versions and one has to update numpy.distutils
> files accordingly.
>
Thank you for those information. Do I understand correctly (sorry for 
being slow, but I don't know anything about fortran) that what you need 
is macro like:
    - AC_F77_WRAPPERS (which defines C macros for converting C functions 
to fortran compiler mangling)
    - AC_F77_FUNC (which retrieves the name mangled by fortran linked 
from the 'canonical' name: this is already implemented, as it is 
necessary for checking blas/lapack)

And that's it ?

cheers,

David


From pearu at cens.ioc.ee  Fri Oct 12 03:53:07 2007
From: pearu at cens.ioc.ee (Pearu Peterson)
Date: Fri, 12 Oct 2007 09:53:07 +0200
Subject: [Numpy-discussion] For review: first milestone of scons support
 in	numpy
In-Reply-To: <470EEB60.3090800@ar.media.kyoto-u.ac.jp>
References: <470D9EC2.5000209@ar.media.kyoto-u.ac.jp>	<470DD809.5050308@cens.ioc.ee>	<470DE4B5.3080007@ar.media.kyoto-u.ac.jp>	<470DF0AE.2030206@cens.ioc.ee>	<470DF706.90309@ar.media.kyoto-u.ac.jp>	<470E0817.10909@cens.ioc.ee>
	<470EEB60.3090800@ar.media.kyoto-u.ac.jp>
Message-ID: <470F27E3.8070000@cens.ioc.ee>



David Cournapeau wrote:
> Pearu Peterson wrote:

>> My point was that
>>
>>    root/numpy/linalg > scons
>>
>> should work (without the -u option). 
> Since scons is called by distutils, and never by the user, I don't see 
> how this can be a problem ? distutils would have to figure out when to 
> use -u, not the user.

Yes. I was considering the situations where distutils in not used at 
all. May be it will be relevant in future..

> The way I see it, either we keep the current behaviour (each package is 
> totally independant, scons is called for each subpackage, and scons has 
> no way to know about other subpackages; this has the disadvantage of 
> being slower: this will be significant of many subpackages use costly 
> checks like fortran mangling and so on), or we have a main sconscript 
> with the configuration, which does not prevent building subpackages, but 
> which requires a global configuration.
> 
> If none of those approach seems right to you, I will see if I can come 
> up with something better, but this will certainly add some complexity 
> (or I am just stupid to see an obvious solution :) ).

I would prefer the former solution where each package is totally
independent. Let's first see what is the acctual performance and then
maybe consider more complex solutions if the build is really slow.

>> To use f2py succesfully, a fortran compiler must support flags that make
>> fortran symbol names lowercase and with exactly one underscore at the 
>> end of a name. This is required when using numpy.distutils.
>>
>> f2py generated modules make use of the following CPP macros:
>> -DPREPEND_FORTRAN -DNO_APPEND_FORTRAN -DUPPERCASE_FORTRAN -DUNDERSCORE_G77
>> and therefore the above requirement would not be needed if
>> scons could figure out how some particular compiler mangles
>> the names of fortran symbols. This would be especially useful
>> since some fortran compiler vendors change the compiler flags
>> between compiler versions and one has to update numpy.distutils
>> files accordingly.
>>
> Thank you for those information. Do I understand correctly (sorry for 
> being slow, but I don't know anything about fortran) that what you need 
> is macro like:
>     - AC_F77_WRAPPERS (which defines C macros for converting C functions 
> to fortran compiler mangling)
>     - AC_F77_FUNC (which retrieves the name mangled by fortran linked 
> from the 'canonical' name: this is already implemented, as it is 
> necessary for checking blas/lapack)

These macros are fine. May be we need more such macros for F90 compilers
but we can discuss about these when the corresponding issues will emerge.

However, if you could also provide the macros
that f2py uses then everything would work with no change in f2py part.
f2py uses the following defintion of F_FUNC, for example:

#if defined(PREPEND_FORTRAN)
#if defined(NO_APPEND_FORTRAN)
#if defined(UPPERCASE_FORTRAN)
#define F_FUNC(f,F) _##F
#else
#define F_FUNC(f,F) _##f
#endif
#else
#if defined(UPPERCASE_FORTRAN)
#define F_FUNC(f,F) _##F##_
#else
#define F_FUNC(f,F) _##f##_
#endif
#endif
#else
#if defined(NO_APPEND_FORTRAN)
#if defined(UPPERCASE_FORTRAN)
#define F_FUNC(f,F) F
#else
#define F_FUNC(f,F) f
#endif
#else
#if defined(UPPERCASE_FORTRAN)
#define F_FUNC(f,F) F##_
#else
#define F_FUNC(f,F) f##_
#endif
#endif
#endif
#if defined(UNDERSCORE_G77)
#define F_FUNC_US(f,F) F_FUNC(f##_,F##_)
#else
#define F_FUNC_US(f,F) F_FUNC(f,F)
#endif

> And that's it ?

I don't recall that there would be more issues that would prevent
using f2py (we'll see if I am going to be wrong here;).

system_info has a feature of providing architecture specific
and optimization flags for fortran compilers. Could scons
support such a feature?

Pearu


From david at ar.media.kyoto-u.ac.jp  Fri Oct 12 04:33:53 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 12 Oct 2007 17:33:53 +0900
Subject: [Numpy-discussion] For review: first milestone of scons support
 in	numpy
In-Reply-To: <470F27E3.8070000@cens.ioc.ee>
References: <470D9EC2.5000209@ar.media.kyoto-u.ac.jp>	<470DD809.5050308@cens.ioc.ee>	<470DE4B5.3080007@ar.media.kyoto-u.ac.jp>	<470DF0AE.2030206@cens.ioc.ee>	<470DF706.90309@ar.media.kyoto-u.ac.jp>	<470E0817.10909@cens.ioc.ee>	<470EEB60.3090800@ar.media.kyoto-u.ac.jp>
	<470F27E3.8070000@cens.ioc.ee>
Message-ID: <470F3171.9040301@ar.media.kyoto-u.ac.jp>

Pearu Peterson wrote:
>
> Yes. I was considering the situations where distutils in not used at 
> all. May be it will be relevant in future..
>
If distutils is not used at all, there will be more difficult problems 
to solve than this one, then. I think it is safe to assume that 
distutils will stay there for quite some time, no ?

Note that I try hard in numpy/scons to separate the scons specific from 
the parts which are not specific to scons. If in the future, we decide 
to go on the waf route, or whatever, I hope that most of the logic would 
be easily reused: for example, parsing the fortran link output to guess 
the link options for C/Fortran is totally independant from scons (this 
has the nice advantage of making regression tests possible on all 
platforms).
>
> I would prefer the former solution where each package is totally
> independent. Let's first see what is the acctual performance and then
> maybe consider more complex solutions if the build is really slow.
I measured the initialization time of scons if called for every 
subpackage in numpy/: this adds less than two percents of total build 
time, so this is not as bad as I feared (specially since only a few 
package would need scons: pure python ones do not need them). And scons 
can also give some speed advantage, because it explicitely support in a 
totally safe way parallel builds.

So basically, I went to the same conclusion than you. I have started 
porting numpy.core to scons (by far the most difficult subpackage, at 
least a priori) following this route.
>
> These macros are fine. May be we need more such macros for F90 compilers
> but we can discuss about these when the corresponding issues will emerge.
Oh, I forgot to say it, but the macro would be implemented for both F77 
and F90 (and F2003 or whatever should we need it). As most of the logic 
is independent of the version used, this is not a problem.

>
> However, if you could also provide the macros
> that f2py uses then everything would work with no change in f2py part.
> f2py uses the following defintion of F_FUNC, for example:
>
I can of course change the name of the macro (this could be an option, 
even). I guess it is better to follow f2py convention than autotools 
here, in our case.
>
> system_info has a feature of providing architecture specific
> and optimization flags for fortran compilers. Could scons
> support such a feature?
scons uses the concept of Environment objects: they have sensible 
defaults (compiler/platform combination), but you can override anything 
you want, and they are used for compilation.

env = Environment()
env.AppendUnique(F77FLAGS = "-O3")
env.Program('hello', source = ['hello.f'])

Of course, in our case, we should provide some support such as instead 
of adding -O3, we have something like a FULL_OPTIMIZED variable which 
contains the optimization flags tweaked for a particular 
compiler/platform. Better, we can check that a given flag is actually 
supported. This is something I have not done yet, but that's the kind of 
things scons does really well, so I don't see any problems here.

The main problem right now is to convert all the checks in 
numpy/core/setup.py, which is the only non trivial setup.py file as far 
as I can tell in numpy. This will take a few hours to get it right on 
major platforms :)

cheers,

David

P.S If you have access to some fortran compiler you would like to check, 
you can check my bzr branch on launchpad (this is done outside numpy 
because some things may be merged to scons sources at some point):

https://code.launchpad.net/~david-ar/numpy.scons.support/dev

And test it by: scons CC=ccompiler F77=f77compiler F90=f90compiler. For 
example: scons CC=icc F77=g77 F90=ifort.

(If you change compiler, you may need to add the option --config=force).

This is a "fake" sconscript which tests the following:
    - does the fortran compiler works (e.g. can compile a dummy program)
    - find the dummy main
    - find the verbose flag (useful to find informations at runtime)
    - find the name mangling
    - find the necessary options for C/Fortran mix (only implemented for 
F77 for now).

There is a pretty good chance that you will be able to break it (in 
particular finding the options for C/Fortran: I have not yet implemented 
all the necessary parsing), but in this case, please provides me with 
the config.log (which logs all the configuration dones by scons).


From matthieu.brucher at gmail.com  Fri Oct 12 05:56:30 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 12 Oct 2007 11:56:30 +0200
Subject: [Numpy-discussion] Use of PyArray_FromAny() with a PyArrayInterface
	object
Message-ID: <e76aa17f0710120256m3570acefo39a4838392db850a@mail.gmail.com>

Hi,

I'm trying to understand (but perhaps everything is in the numpy book in
which case I'd rather buy the book immediately) how to use the
PyArray_FromAny() function.
Suppose I have a C object (created by Pyrex or SWIG or Boost.Python) that
has the __array_struct__ attribute. Can I pass this object to the
PyArray_FromAny() function to get a PyArrayObject ?
If it is not the case, if I have a PyCObject created with a
PyArrayInterface, is it possible ?
I read on a mailing-list that there is the context argument that can be
passed to an __array__ method. What is the purpose of this __array__ method
? I don't recollect anything regarding this method :(

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071012/684f06d7/attachment.html>

From renato.serodio at gmail.com  Fri Oct 12 08:54:31 2007
From: renato.serodio at gmail.com (Renato Serodio)
Date: Fri, 12 Oct 2007 14:54:31 +0200
Subject: [Numpy-discussion] abstraction and interfaces
Message-ID: <4f4101bf0710120554y1b57d351n9157663db993916a@mail.gmail.com>

Hello there,

I use NumPy/SciPy to process telemetry, producing some metrics that I
plot with matplotlib. Well, I want to package my scripts into a sort
of all-wrapping application, i.e, the user just calls the app, selects
which metrics to produce and waits to obtain the plots, without being
exposed to anything more dangerous than a zoom button..

The scripts that produce these metrics use Scipy/Numpy functions that
operate on data conveniently converted to numpy arrays. They're quite
specific, and I tend to produce/tweak a lot of them. So, to fit in
this application someone suggested I programmed 'interfaces' (in java
jargon) to them - that way I could develop the whole wrapper
application without giving much thought to the actual number-crunching
bits.

Problem is, I learnt numerics using Fortran, and though I understand
the idea of objects, I have quite some difficulty imagining the
'patterns' that allow OO to be so flexible. So I was wondering if
anyone experienced with OO Numerical computing could provide some
pointers. For context, here are my thoughts:

1. the core of the application is responsible for interaction with the
outside world. Crunching blocks are provided data and return
matplotlib objects that go into GTK. To write procedures that abide to
these parameters is my first (fortranic) impulse. But then I need to
change the core everytime I add a procedure, and I'm limited to
numpy-IN/matplolib-OUT.

2. if I could force every analysis procedure to be a class that
implements its own data retrieval and outputs matplolib objects, my
core no longer controls the whole communication with the outside
world. And i still have to edit the core everytime a new analysis is
required. A very interesting thing is that with these classes I could
create new objects with different attributes, rather than pass a host
of parameters to a procedure..

So, I'm a bit at odds with this brave new world I discovered. I'd be
quite grateful for pointers on literature or examples that more or
less lead to a solution. So far my searches have justified the need
for objects, but not helped address this modularity requirement...

Cheers,

Renato


From aisaac at american.edu  Fri Oct 12 09:20:50 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 12 Oct 2007 09:20:50 -0400
Subject: [Numpy-discussion] abstraction and interfaces
In-Reply-To: <4f4101bf0710120554y1b57d351n9157663db993916a@mail.gmail.com>
References: <4f4101bf0710120554y1b57d351n9157663db993916a@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-1524-20071012-092050.00@american.edu>

On Fri, 12 Oct 2007, Renato Serodio apparently wrote:
> The scripts that produce these metrics use Scipy/Numpy 
> functions that operate on data conveniently converted to 
> numpy arrays. They're quite specific, and I tend to 
> produce/tweak a lot of them. So, to fit in this 
> application someone suggested I programmed 'interfaces' 
> (in java jargon) to them - that way I could develop the 
> whole wrapper application without giving much thought to 
> the actual number-crunching bits. 

That sounds quite right.  Check out
<URL:https://projects.scipy.org/scipy/scikits/browser/trunk/openopt/scikits/openopt/solvers/optimizers/optimizer>
<URL:http://svn.scipy.org/svn/scipy/trunk/scipy/stats/models/>
for examples that may be relevant to your project.

Python does not have interfaces per se, but that does
not stop you from designing interface-like classes and
inheriting from them.

fwiw,
Alan Isaac




From aisaac at american.edu  Fri Oct 12 09:30:25 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Fri, 12 Oct 2007 09:30:25 -0400
Subject: [Numpy-discussion]
	=?iso-8859-1?q?Use_of_PyArray_FromAny=28=29_wi?=
	=?iso-8859-1?q?th_a_PyArrayInterface=09object?=
In-Reply-To: <e76aa17f0710120256m3570acefo39a4838392db850a@mail.gmail.com>
References: <e76aa17f0710120256m3570acefo39a4838392db850a@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-1524-20071012-093025.00@american.edu>

On Fri, 12 Oct 2007, Matthieu Brucher apparently wrote:
> I'm trying to understand (but perhaps everything is in the 
> numpy book in which case I'd rather buy the book 
> immediately) how to use the PyArray_FromAny() function. 

This function is discussed in the NumPy Book.
I see perhaps a dozen references to the function
in the Complete API chapter.

Even apart from that chapter, which is useful to you but
not to me, I have found the book to be excellent.

Cheers,
Alan Isaac





From matthieu.brucher at gmail.com  Fri Oct 12 09:29:13 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 12 Oct 2007 15:29:13 +0200
Subject: [Numpy-discussion] Use of PyArray FromAny() with a
	PyArrayInterface object
In-Reply-To: <Mahogany-0.67.0-1524-20071012-093025.00@american.edu>
References: <e76aa17f0710120256m3570acefo39a4838392db850a@mail.gmail.com>
	<Mahogany-0.67.0-1524-20071012-093025.00@american.edu>
Message-ID: <e76aa17f0710120629m5f9e65bdjc9a6f7e7289db15e@mail.gmail.com>

2007/10/12, Alan G Isaac <aisaac at american.edu>:
>
> On Fri, 12 Oct 2007, Matthieu Brucher apparently wrote:
> > I'm trying to understand (but perhaps everything is in the
> > numpy book in which case I'd rather buy the book
> > immediately) how to use the PyArray_FromAny() function.
>
> This function is discussed in the NumPy Book.
> I see perhaps a dozen references to the function
> in the Complete API chapter.
>
> Even apart from that chapter, which is useful to you but
> not to me, I have found the book to be excellent.


Excellent, thank you for the tip, the book is now ordered :)

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071012/029d251d/attachment.html>

From ndbecker2 at gmail.com  Fri Oct 12 11:01:33 2007
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 12 Oct 2007 11:01:33 -0400
Subject: [Numpy-discussion] Use of PyArray FromAny() with a
	PyArrayInterface object
References: <e76aa17f0710120256m3570acefo39a4838392db850a@mail.gmail.com>
	<Mahogany-0.67.0-1524-20071012-093025.00@american.edu>
	<e76aa17f0710120629m5f9e65bdjc9a6f7e7289db15e@mail.gmail.com>
Message-ID: <feo2al$sti$1@sea.gmane.org>

Matthieu Brucher wrote:

> 2007/10/12, Alan G Isaac <aisaac at american.edu>:
>>
>> On Fri, 12 Oct 2007, Matthieu Brucher apparently wrote:
>> > I'm trying to understand (but perhaps everything is in the
>> > numpy book in which case I'd rather buy the book
>> > immediately) how to use the PyArray_FromAny() function.
>>
>> This function is discussed in the NumPy Book.
>> I see perhaps a dozen references to the function
>> in the Complete API chapter.
>>
>> Even apart from that chapter, which is useful to you but
>> not to me, I have found the book to be excellent.
> 
> 
> Excellent, thank you for the tip, the book is now ordered :)
> 
> Matthieu
I don't see the text string '__array' anywhere in this book.



From matthieu.brucher at gmail.com  Fri Oct 12 11:08:13 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 12 Oct 2007 17:08:13 +0200
Subject: [Numpy-discussion] Use of PyArray FromAny() with a
	PyArrayInterface object
In-Reply-To: <feo2al$sti$1@sea.gmane.org>
References: <e76aa17f0710120256m3570acefo39a4838392db850a@mail.gmail.com>
	<Mahogany-0.67.0-1524-20071012-093025.00@american.edu>
	<e76aa17f0710120629m5f9e65bdjc9a6f7e7289db15e@mail.gmail.com>
	<feo2al$sti$1@sea.gmane.org>
Message-ID: <e76aa17f0710120808l410de541q72d1640154ff07f7@mail.gmail.com>

>
>
> I don't see the text string '__array' anywhere in this book.
>

I saw something a subchapter with the PyArrayInterface, nothing around it
with __array__ or __array_struct__ ?

Matthieu (wating for the book :))
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071012/9d492832/attachment.html>

From oliphant at enthought.com  Fri Oct 12 13:03:59 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 12 Oct 2007 12:03:59 -0500
Subject: [Numpy-discussion] Use of PyArray FromAny() with
 a	PyArrayInterface object
In-Reply-To: <feo2al$sti$1@sea.gmane.org>
References: <e76aa17f0710120256m3570acefo39a4838392db850a@mail.gmail.com>	<Mahogany-0.67.0-1524-20071012-093025.00@american.edu>	<e76aa17f0710120629m5f9e65bdjc9a6f7e7289db15e@mail.gmail.com>
	<feo2al$sti$1@sea.gmane.org>
Message-ID: <470FA8FF.7050502@enthought.com>

Neal Becker wrote:
> Matthieu Brucher wrote:
>
>   
>> 2007/10/12, Alan G Isaac <aisaac at american.edu>:
>>     
>>> On Fri, 12 Oct 2007, Matthieu Brucher apparently wrote:
>>>       
>>>> I'm trying to understand (but perhaps everything is in the
>>>> numpy book in which case I'd rather buy the book
>>>> immediately) how to use the PyArray_FromAny() function.
>>>>         
>>> This function is discussed in the NumPy Book.
>>> I see perhaps a dozen references to the function
>>> in the Complete API chapter.
>>>
>>> Even apart from that chapter, which is useful to you but
>>> not to me, I have found the book to be excellent.
>>>       
>> Excellent, thank you for the tip, the book is now ordered :)
>>
>> Matthieu
>>     
> I don't see the text string '__array' anywhere in this book.
>   

Perhaps PDF searching cannot see "bold" text.   But, take a look at page 74.

-Travis



From yves.revaz at obspm.fr  Sun Oct 14 10:44:14 2007
From: yves.revaz at obspm.fr (Yves Revaz)
Date: Sun, 14 Oct 2007 16:44:14 +0200
Subject: [Numpy-discussion] .bytes
Message-ID: <47122B3E.8010703@obspm.fr>

Dear list,

I'm translating codes from numarray to numpy.
Unfortunately, I'm unable to find the equivalent of the command
that give the number of bytes for a given type :
using numarray I used :

 >>> Float32.bytes
4

I'm sure there is a solution in numpy,
but I'm unable to find it.

Thanks,

Yves

-- 
                                                (o o)
--------------------------------------------oOO--(_)--OOo-------
  Yves Revaz
  Lerma Batiment A           Tel : ++ 33 (0) 1 40 51 20 79
  Observatoire de Paris      Fax : ++ 33 (0) 1 40 51 20 02 
  77 av Denfert-Rochereau    e-mail : yves.revaz at obspm.fr
  F-75014 Paris              Web : http://www.lunix.ch/revaz/
  FRANCE             
----------------------------------------------------------------



From matthieu.brucher at gmail.com  Sun Oct 14 10:48:04 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Sun, 14 Oct 2007 16:48:04 +0200
Subject: [Numpy-discussion] .bytes
In-Reply-To: <47122B3E.8010703@obspm.fr>
References: <47122B3E.8010703@obspm.fr>
Message-ID: <e76aa17f0710140748i301580eal87527a80c8125299@mail.gmail.com>

Hi,

In the description field, you have itemsize which is what you want.

Matthieu

2007/10/14, Yves Revaz <yves.revaz at obspm.fr>:
>
> Dear list,
>
> I'm translating codes from numarray to numpy.
> Unfortunately, I'm unable to find the equivalent of the command
> that give the number of bytes for a given type :
> using numarray I used :
>
> >>> Float32.bytes
> 4
>
> I'm sure there is a solution in numpy,
> but I'm unable to find it.
>
> Thanks,
>
> Yves
>
> --
>                                                 (o o)
> --------------------------------------------oOO--(_)--OOo-------
>   Yves Revaz
>   Lerma Batiment A           Tel : ++ 33 (0) 1 40 51 20 79
>   Observatoire de Paris      Fax : ++ 33 (0) 1 40 51 20 02
>   77 av Denfert-Rochereau    e-mail : yves.revaz at obspm.fr
>   F-75014 Paris              Web : http://www.lunix.ch/revaz/
>   FRANCE
> ----------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071014/4c9930dc/attachment.html>

From nadavh at visionsense.com  Sun Oct 14 11:11:13 2007
From: nadavh at visionsense.com (Nadav Horesh)
Date: Sun, 14 Oct 2007 17:11:13 +0200
Subject: [Numpy-discussion] .bytes
Message-ID: <710F2847B0018641891D9A21602763600B6DFE@ex3.envision.co.il>


array(1, dtype=float32).itemsize

  Nadav

-----Original Message-----
From: numpy-discussion-bounces at scipy.org on behalf of Yves Revaz
Sent: Sun 14-Oct-07 16:44
To: numpy-discussion at scipy.org
Subject: [Numpy-discussion] .bytes
 
Dear list,

I'm translating codes from numarray to numpy.
Unfortunately, I'm unable to find the equivalent of the command
that give the number of bytes for a given type :
using numarray I used :

 >>> Float32.bytes
4

I'm sure there is a solution in numpy,
but I'm unable to find it.

Thanks,

Yves

-- 
                                                (o o)
--------------------------------------------oOO--(_)--OOo-------
  Yves Revaz
  Lerma Batiment A           Tel : ++ 33 (0) 1 40 51 20 79
  Observatoire de Paris      Fax : ++ 33 (0) 1 40 51 20 02 
  77 av Denfert-Rochereau    e-mail : yves.revaz at obspm.fr
  F-75014 Paris              Web : http://www.lunix.ch/revaz/
  FRANCE             
----------------------------------------------------------------

_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 2834 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071014/82dd4b16/attachment.bin>

From david at ar.media.kyoto-u.ac.jp  Mon Oct 15 06:29:09 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 15 Oct 2007 19:29:09 +0900
Subject: [Numpy-discussion] Partially merging scons into trunk ?
Message-ID: <471340F5.4060302@ar.media.kyoto-u.ac.jp>

Hi,

    Following the previous thread on using scons within distutils in 
numpy, I would like to know if it is ok to merge some of the code from 
numpy.scons branch into the trunk ? I don't want to merge everything, 
just the part which has little chance to break anything (basically up to 
rev 4178 in numpy.scons branch; the code has been sucessfully tested on 
most major platforms). The only downside I can see is that it adds ~ 1 
Mb of code into numpy because I included a local copy on scons.

    cheers,

    David
   


From yves.revaz at obspm.fr  Mon Oct 15 06:46:01 2007
From: yves.revaz at obspm.fr (Yves Revaz)
Date: Mon, 15 Oct 2007 12:46:01 +0200
Subject: [Numpy-discussion] .bytes
In-Reply-To: <710F2847B0018641891D9A21602763600B6DFE@ex3.envision.co.il>
References: <710F2847B0018641891D9A21602763600B6DFE@ex3.envision.co.il>
Message-ID: <471344E9.7090509@obspm.fr>

Nadav Horesh wrote:

>array(1, dtype=float32).itemsize
>
>  
>
ok, it will work fine for my purpose.
In numpy, is there any reason to supress the attribute .bytes from the 
type object itself ?
Is it simply because the native python types (int, float, complex, etc.) 
do not have this attribute ?

>  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Numpy-discussion mailing list
>Numpy-discussion at scipy.org
>http://projects.scipy.org/mailman/listinfo/numpy-discussion
>  
>


-- 
                                                (o o)
--------------------------------------------oOO--(_)--OOo-------
  Yves Revaz
  Lerma Batiment A           Tel : ++ 33 (0) 1 40 51 20 79
  Observatoire de Paris      Fax : ++ 33 (0) 1 40 51 20 02 
  77 av Denfert-Rochereau    e-mail : yves.revaz at obspm.fr
  F-75014 Paris              Web : http://obswww.unige.ch/~revaz/
  FRANCE             
----------------------------------------------------------------



From matthieu.brucher at gmail.com  Mon Oct 15 09:02:31 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 15 Oct 2007 15:02:31 +0200
Subject: [Numpy-discussion] Error in deallocation ?
Message-ID: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>

Hi

I keep on getting this error :

*** Reference count error detected:
an attempt was made to deallocate 7 (l) ***

It happens on numpy calls (multiplications, call to inner(), ...), but I
didn't find the real reason. I'm using Numpy '1.0.4.dev3875' with Python
2.5.1.

Someone has a hint to solve this ?

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071015/b0f002e0/attachment.html>

From oliphant at enthought.com  Mon Oct 15 10:33:18 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Mon, 15 Oct 2007 09:33:18 -0500
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
Message-ID: <47137A2E.6030006@enthought.com>

Matthieu Brucher wrote:
> Hi
>
> I keep on getting this error :
>
> *** Reference count error detected:
> an attempt was made to deallocate 7 (l) ***
>
> It happens on numpy calls (multiplications, call to inner(), ...), but 
> I didn't find the real reason. I'm using Numpy ' 1.0.4.dev3875' with 
> Python 2.5.1.
>
> Someone has a hint to solve this ?

The problem is that there is a data-type reference counting error some 
where that is attempting to deallocate the built-in data-type 'l' 

It's not really a Python error but a logging.  The code won't let you 
deallocate the built-ins, but it will tell you that something tried to.

Reference counting on data-types is easy to get wrong (particularly with 
Pyrex extension modules) because most calls consume a reference to the 
data-type (if they return an object that  contains a reference to the 
data-type).

It is a bug, and it would be nice to figure it out, but that would 
require the code that caused it.

Thanks,

-Travis O.



From matthieu.brucher at gmail.com  Mon Oct 15 10:39:54 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 15 Oct 2007 16:39:54 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <47137A2E.6030006@enthought.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
Message-ID: <e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>

>
> The problem is that there is a data-type reference counting error some
> where that is attempting to deallocate the built-in data-type 'l'



That's what I supposed, but I couldn't find the reason why it wanted to do
this


It's not really a Python error but a logging.  The code won't let you
> deallocate the built-ins, but it will tell you that something tried to.
>
> Reference counting on data-types is easy to get wrong (particularly with
> Pyrex extension modules) because most calls consume a reference to the
> data-type (if they return an object that  contains a reference to the
> data-type).
>
> It is a bug, and it would be nice to figure it out, but that would
> require the code that caused it.


I've updated my numpy version to the latest svn, the behaviour seems to be
different (more warnings), I'll try to give more information about the
error, but giving the whole code will not be simple (it uses a big data file
that seems to trigger the error as with other data files, the error didn't
show up :()

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071015/7f394700/attachment.html>

From robert.kern at gmail.com  Mon Oct 15 12:42:53 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 15 Oct 2007 11:42:53 -0500
Subject: [Numpy-discussion] .bytes
In-Reply-To: <471344E9.7090509@obspm.fr>
References: <710F2847B0018641891D9A21602763600B6DFE@ex3.envision.co.il>
	<471344E9.7090509@obspm.fr>
Message-ID: <4713988D.70004@gmail.com>

Yves Revaz wrote:
> Nadav Horesh wrote:
> 
>> array(1, dtype=float32).itemsize
>>
> ok, it will work fine for my purpose.
> In numpy, is there any reason to supress the attribute .bytes from the 
> type object itself ?
> Is it simply because the native python types (int, float, complex, etc.) 
> do not have this attribute ?

The problem is that the instances of the scalar types do have the itemsize
attribute. The implementation of type objects is such that the type object will
also have that attribute, but it will be a stub:

In [15]: float64.itemsize
Out[15]: <attribute 'itemsize' of 'numpy.generic' objects>

A more straightforward way to get the itemsize is this:

In [17]: dtype(float64).itemsize
Out[17]: 8

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From bhendrix at enthought.com  Mon Oct 15 13:08:52 2007
From: bhendrix at enthought.com (Bryce Hendrix)
Date: Mon, 15 Oct 2007 12:08:52 -0500
Subject: [Numpy-discussion] Partially merging scons into trunk ?
In-Reply-To: <471340F5.4060302@ar.media.kyoto-u.ac.jp>
References: <471340F5.4060302@ar.media.kyoto-u.ac.jp>
Message-ID: <47139EA4.5080409@enthought.com>

I've only been following this thread on the perimeter, so I'm not sure
if "makefile" migration has been discussed. I have a script I wrote
about a year ago when we (Enthought) were looking at using SCons for our
internal builds. The script is capable of generating SConscript files
from setup.py scripts. It works by faking a build with distutils and
uses those build steps to generate the scons targets.

The script is not complete, for example 'weave' confuses it, but it
could be a good starting point. If anyone is interested in it, I'll post
it to the list.

Bryce

David Cournapeau wrote:
> Hi,
>
>     Following the previous thread on using scons within distutils in 
> numpy, I would like to know if it is ok to merge some of the code from 
> numpy.scons branch into the trunk ? I don't want to merge everything, 
> just the part which has little chance to break anything (basically up to 
> rev 4178 in numpy.scons branch; the code has been sucessfully tested on 
> most major platforms). The only downside I can see is that it adds ~ 1 
> Mb of code into numpy because I included a local copy on scons.
>
>     cheers,
>
>     David
>    
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   



From giles.thomas at resolversystems.com  Mon Oct 15 14:21:35 2007
From: giles.thomas at resolversystems.com (Giles Thomas)
Date: Mon, 15 Oct 2007 19:21:35 +0100
Subject: [Numpy-discussion] Request for advice: project to get NumPy working
	in IronPython
Message-ID: <4713AFAF.10201@resolversystems.com>

Hi,

At Resolver Systems, we have a product that is written in IronPython - 
the .NET Python implementation - and allows users to use that language 
to script a spreadsheet-like interface.  Because they're using 
IronPython, they can access their existing .NET objects and libraries, 
which has worked out really well for us and for them.  But there's an 
increasing number of users who would like access to CPython C extensions 
- in particular, NumPy.

We *don't* want to just hack something together that would just allow 
users of our own product to access NumPy; what we'd really like to do is 
work on an OSS solution that will allow all IronPython users to access 
NumPy transparently, just as if they were using CPython.

My question is, is this something that the readers of this list would 
like to get involved in, either in terms of contibuting, or even in 
terms on advice?  We're under no illusions; this is a significant 
undertaking.  And we're very keen on hearing from those who know NumPy 
better about how this should be structured, even if people don't have 
the time to write code for it.  In particular:

    * How do people feel about a source-code-compatible solution, where
      (perhaps) we would maintain a project that basically provided an
      alternative makefile for NumPy - or, even better, we could work
      with the NumPy developers to contribute a .NET/IronPython package.
    * Would it be better to try for some kind of "binary compatibility",
      where we'd write some kind of "glue" that sat between the existing
      C extension .pyd files and the IronPython engine?  Our gut feeling
      is that this would be much more work, but we might be missing
      something.
    * What should the work's relationship to the NumPy project be?

Any thoughts, comments, or advice would be very much appreciated.


Regards,

Giles

-- 
Giles Thomas
MD & CTO, Resolver Systems Ltd.
giles.thomas at resolversystems.com
+44 (0) 20 7253 6372

We're hiring! http://www.resolversystems.com/jobs/ 

17a Clerkenwell Road, London EC1M 5RD, UK
VAT No.: GB 893 5643 79 
Registered in England and Wales as company number 5467329.
Registered address: 843 Finchley Road, London NW11 8NA, UK

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071015/04297f6c/attachment.html>

From fullung at gmail.com  Mon Oct 15 14:49:32 2007
From: fullung at gmail.com (Albert Strasheim)
Date: Mon, 15 Oct 2007 20:49:32 +0200
Subject: [Numpy-discussion] Request for advice: project to get
	NumPy	working in IronPython
In-Reply-To: <4713AFAF.10201@resolversystems.com>
References: <4713AFAF.10201@resolversystems.com>
Message-ID: <20071015184932.GA4159@dogbert.sdsl.sun.ac.za>

Hello

On Mon, 15 Oct 2007, Giles Thomas wrote:

>    * Would it be better to try for some kind of "binary compatibility",
>      where we'd write some kind of "glue" that sat between the existing
>      C extension .pyd files and the IronPython engine?  Our gut feeling
>      is that this would be much more work, but we might be missing
>      something.

Along these lines, what ever happened to Sanghyeon Seo's CPython 
reflector?

http://lists.ironpython.com/pipermail/users-ironpython.com/2006-June/002503.html

I think it might be an avenue worth exploring, since this would also 
make other native modules available as a by-product. Last I heard, the 
JRuby folks were also trying something these lines for native MRI 
modules, using JNA (essentially, a Java equivalent to ctypes).

Good luck with your project. I hope it succeeds.

Cheers,

Albert


From oliphant at enthought.com  Mon Oct 15 15:35:45 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Mon, 15 Oct 2007 14:35:45 -0500
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
Message-ID: <4713C111.5010609@enthought.com>

Matthieu Brucher wrote:
>
>     The problem is that there is a data-type reference counting error some
>     where that is attempting to deallocate the built-in data-type 'l' 
>
>
>
> That's what I supposed, but I couldn't find the reason why it wanted 
> to do this
>  
>
>     It's not really a Python error but a logging.  The code won't let you
>     deallocate the built-ins, but it will tell you that something
>     tried to.
>
>     Reference counting on data-types is easy to get wrong
>     (particularly with
>     Pyrex extension modules) because most calls consume a reference to the
>     data-type (if they return an object that  contains a reference to the
>     data-type).
>
>     It is a bug, and it would be nice to figure it out, but that would
>     require the code that caused it.
>
>
> I've updated my numpy version to the latest svn, the behaviour seems 
> to be different (more warnings), I'll try to give more information 
> about the error, but giving the whole code will not be simple (it uses 
> a big data file that seems to trigger the error as with other data 
> files, the error didn't show up :()
>

There are two types of errors that can occur with reference counting on 
data-types. 

1) There are too many DECREF's --- this gets us to the error quickly and 
is usually easy to reproduce
2) There are too many INCREF's (the reference count keeps going up until 
the internal counter wraps around to 0 and deallocation is attempted) 
--- this error is harder to reproduce and usually takes a while before 
it happens in the code.


-Travis



From oliphant at enthought.com  Mon Oct 15 15:44:52 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Mon, 15 Oct 2007 14:44:52 -0500
Subject: [Numpy-discussion] Request for advice: project to get NumPy
 working in IronPython
In-Reply-To: <4713AFAF.10201@resolversystems.com>
References: <4713AFAF.10201@resolversystems.com>
Message-ID: <4713C334.2000407@enthought.com>

Giles Thomas wrote:
> Hi,
>
> At Resolver Systems, we have a product that is written in IronPython - 
> the .NET Python implementation - and allows users to use that language 
> to script a spreadsheet-like interface.  Because they're using 
> IronPython, they can access their existing .NET objects and libraries, 
> which has worked out really well for us and for them.  But there's an 
> increasing number of users who would like access to CPython C 
> extensions - in particular, NumPy.
An IronPython compatible version of NumPy would be great.    Of course 
it could be done by using C# to write NumPy, but I'm not sure that this 
would really be any less work than creating a "glue" layer that allowed 
most (or all) C-Python extensions to work with IronPython.


>     * How do people feel about a source-code-compatible solution,
>       where (perhaps) we would maintain a project that basically
>       provided an alternative makefile for NumPy - or, even better, we
>       could work with the NumPy developers to contribute a
>       .NET/IronPython package.
>
Hmm.. I don't know enough about C# I guess, but would a different 
setup.py file really be enough?  Would this have some sort of C->C# 
translator.  I would be surprised if that actually worked, though.
>
>    *
>
>
>     * Would it be better to try for some kind of "binary
>       compatibility", where we'd write some kind of "glue" that sat
>       between the existing C extension .pyd files and the IronPython
>       engine?  Our gut feeling is that this would be much more work,
>       but we might be missing something.
>
I'm not sure that this would really be any more work than the C->C# 
translator that you are talking about above, but then again I've never 
done any C to C# translation.
>
>     * What should the work's relationship to the NumPy project be?
>
This is flexible.   It could be distributed with NumPy or else simply 
advertised by NumPy.


Best regards,

-Travis Oliphant



From Chris.Barker at noaa.gov  Mon Oct 15 16:05:14 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 15 Oct 2007 13:05:14 -0700
Subject: [Numpy-discussion] Request for advice: project to get NumPy
 working in IronPython
In-Reply-To: <4713C334.2000407@enthought.com>
References: <4713AFAF.10201@resolversystems.com>
	<4713C334.2000407@enthought.com>
Message-ID: <4713C7FA.3080205@noaa.gov>

Is there a C# <-> C (or, I guess, really a .net <-> C ) binding 
mechanism? Like JNI for Java?

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From david at ar.media.kyoto-u.ac.jp  Mon Oct 15 23:28:03 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 16 Oct 2007 12:28:03 +0900
Subject: [Numpy-discussion] Request for advice: project to get NumPy
 working in IronPython
In-Reply-To: <4713C7FA.3080205@noaa.gov>
References: <4713AFAF.10201@resolversystems.com>	<4713C334.2000407@enthought.com>
	<4713C7FA.3080205@noaa.gov>
Message-ID: <47142FC3.9050207@ar.media.kyoto-u.ac.jp>

Christopher Barker wrote:
> Is there a C# <-> C (or, I guess, really a .net <-> C ) binding 
> mechanism? Like JNI for Java?
I don't know much about the whole .Net thing (CLR, in this case), but 
there is the p-invoke mechanism for that:

http://en.wikipedia.org/wiki/Platform_Invocation_Services

I don't know the performance impact. According to pypy developers, the 
best way to extend python in C is to use ctypes: this makes interop 
easier for different VM (Cpython vs CLR in this case). But in the case 
of numpy, this would mean quite a lot of work, I guess, because numpy is 
more than just a python extension, it provides a big C Api (which is the 
real problem in the case of VM adaptation). The best would be of course 
a solution which enables interop, so that if later, people want to use 
the work gone into numpy for IronPython, people do not have to start 
everything from scratch for say Jython or something else. But maybe this 
is too big of a task.

Giles, maybe you could see with the pypy people if they can give you 
some advice, too (pypy is an implementation of python in python, which 
can translate into different platforms, including the CLI):

http://codespeak.net/pypy/dist/pypy/doc/news.html
http://lists.ironpython.com/pipermail/users-ironpython.com/2007-March/004685.html

cheers,

David



From david at ar.media.kyoto-u.ac.jp  Tue Oct 16 00:58:41 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 16 Oct 2007 13:58:41 +0900
Subject: [Numpy-discussion] Partially merging scons into trunk ?
In-Reply-To: <47139EA4.5080409@enthought.com>
References: <471340F5.4060302@ar.media.kyoto-u.ac.jp>
	<47139EA4.5080409@enthought.com>
Message-ID: <47144501.3090608@ar.media.kyoto-u.ac.jp>

Bryce Hendrix wrote:
> I've only been following this thread on the perimeter, so I'm not sure
> if "makefile" migration has been discussed. I have a script I wrote
> about a year ago when we (Enthought) were looking at using SCons for our
> internal builds. The script is capable of generating SConscript files
> from setup.py scripts. It works by faking a build with distutils and
> uses those build steps to generate the scons targets.
>
> The script is not complete, for example 'weave' confuses it, but it
> could be a good starting point. If anyone is interested in it, I'll post
> it to the list.
>   
That sounds interesting. Do I understand correctly that you ditched 
distutils entirely ? That's not the approach I have taken, because I 
thought it would be too big (before inclusion in numpy, I need to show 
something remotely impressive, to convince the change worths it), and 
would prevent doing things iteratively (e.g. building numpy wo distutils 
would be a huge task). For most subpackages of scipy, which have small 
setup.py, this would be useful, I guess (for the packages I know, they 
are independant, at least as far as build issues are concerned).

I don't think any automated task would work for numpy:  I spent a good 
deal of time last WE to be able to build all extensions in numpy with 
scons, and it is a lot of work, because of bootstrapping issues. If you 
take into account the code generators, the numpy.core "Makefile" is more 
than 1000 lines of non trivial code. I finally got my head around it, 
but I don't think any automated scheme would work in this case.

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Tue Oct 16 06:00:51 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 16 Oct 2007 19:00:51 +0900
Subject: [Numpy-discussion] Avoiding distributing config.h in numpy.core ?
Message-ID: <47148BD3.9030405@ar.media.kyoto-u.ac.jp>

Hi,

   I spent the last WE trying to understand the whole numpy build 
process (all this related to my project of "sconsification" of numpy). 
One thing which I came across, and is a bit problematic is the fact that 
numpy/core build process relies on a config.h which is meant to be 
distributed (by distributed, I mean that the config,h is an installed 
file). Would a patch to remove config.h from the generated files be 
considered useful ? (this change would of course be 100 % backward 
compatible: it would be totally transparent to users of numpy headers).

   To be more precise: this config.h is obviously inspired by autoheader 
facilities, but the config.h generated by the autotools should not be 
distributed:

http://sourceware.org/autobook/autobook/autobook_96.html#SEC96

Incidently, the problems mentioned in the above link happen in numpy, 
where we conditionally define some symbols (various SIZEOF_*), because 
python itself does not follow the rule. A config.h should really only 
contains #define/#undef lines, nothing else; it would also avoid 
polluting the namespace (the config.h symbols should not be visible from 
users of numpy headers as they are now). Finally, this would also make 
the sconsification process easier,

cheers,

David


From giles.thomas at resolversystems.com  Tue Oct 16 09:17:43 2007
From: giles.thomas at resolversystems.com (Giles Thomas)
Date: Tue, 16 Oct 2007 14:17:43 +0100
Subject: [Numpy-discussion] Request for advice: project to get NumPy
 working in IronPython
In-Reply-To: <47142FC3.9050207@ar.media.kyoto-u.ac.jp>
References: <4713AFAF.10201@resolversystems.com>	<4713C334.2000407@enthought.com>	<4713C7FA.3080205@noaa.gov>
	<47142FC3.9050207@ar.media.kyoto-u.ac.jp>
Message-ID: <4714B9F7.8050205@resolversystems.com>

We've created a Google Group to discuss this project, and if anyone's 
interested in contributing or advising, you're very welcome to sign up: 
<http://groups.google.com/group/c-extensions-for-ironpython/>

My replies to everyone's comments below:

David Cournapeau wrote:
 > Giles, maybe you could see with the pypy people if they can give you 
some advice, too

Thanks for the pointer - I'd heard of PyPy, but it hadn't occurred to me 
that they have valuable knowledge about how to do this kind of thing - 
obvious now you point it out :-).  I'll get in touch with them.

 > According to pypy developers, the best way to extend python in C is 
to use ctypes:

There was an effort to get ctypes working with IronPython a while back - 
the CPython Reflector by Sanghyeon Seo that Albert Strasheim mentioned 
-  Seo has just joined our mailing list, so we'll certainly start a 
discussion about that.

 > Christopher Barker wrote:
 > > Is there a C# <-> C (or, I guess, really a .net <-> C ) binding
 > > mechanism? Like JNI for Java?
 > I don't know much about the whole .Net thing (CLR, in this case), but
 > there is the p-invoke mechanism for that

P/Invoke is indeed the normal way to link C# with C; we use it in our 
software on those (luckily rare) occasions when we have to step outside 
the .NET world and talk to the basic Win32 APIs.  It's pretty easy to 
do, and our initial thoughts about the best way to integrate IronPython 
with NumPy were that we would use P/Invoke to call into the C library (I 
agree with Travis E. Oliphant that trying to convert the existing C to 
C# would be a Bad Idea).  Using P/Invoke would give us a very 
rough-and-ready .NET API, which we could then wrap with a Python library 
that <handwaving>interfaced with the existing CPython library in some 
manner</handwaving>.

 > I don't know the performance impact.

I don't think P/Invoke slows down the running of the C code, but of 
course calls from .NET to native code are quite slow.  My expectation so 
far has been with most Python code using NumPy, the number-crunching 
will happen at the native code level, with relatively few switches 
between languages - that is, it is a "chunky" rather than a "chatty" 
API.  Is that a reasonable assumption?

> The best would be of course a solution which enables interop, 
> so that if later, people want to use the work gone into numpy 
> for IronPython, people do not have to start everything from 
> scratch for say Jython or something else. But maybe this is 
> too big of a task.


We're aiming to try to solve the "simple" case of one Python 
implementation, one C extension to start off with.  Our intention was to 
then look at handling a couple of other extensions, to broaden the 
solution more toward a general solution for loading C extensions into 
IronPython - I agree that we could usefully also broaden it to handle 
other Python implementations.


Regards,

Giles

-- 
Giles Thomas
MD & CTO, Resolver Systems Ltd.
giles.thomas at resolversystems.com
+44 (0) 20 7253 6372

We're hiring! http://www.resolversystems.com/jobs/ 

17a Clerkenwell Road, London EC1M 5RD, UK
VAT No.: GB 893 5643 79 
Registered in England and Wales as company number 5467329.
Registered address: 843 Finchley Road, London NW11 8NA, UK



From haase at msg.ucsf.edu  Tue Oct 16 11:47:35 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Tue, 16 Oct 2007 17:47:35 +0200
Subject: [Numpy-discussion] setting the attributes of an array of object
In-Reply-To: <loom.20070929T085848-518@post.gmane.org>
References: <loom.20070929T085848-518@post.gmane.org>
Message-ID: <bc657ead0710160847n50b89c03ga783923b0751fe1a@mail.gmail.com>

Hi,
there is a way of doing this. As far as I know, you have to create
your own "version of numpy arrays". E. g. try this:
class myNumpy(numpy.ndarray):
     pass

Then creating an instance as in
a = myNumpy(otherNumpyArray)

would make `a` behave just like any other array, except that you CAN
attach attributes to it.

Be carefull that some (many / most ?) operations on that array will
return you a normal numpy array again.

I don't understand the reasons behind all of this.

HTH,
Sebastian Haase


On 9/29/07, jelle <jelleferinga at gmail.com> wrote:
> Hi,
>
> I'm wondering whether i can re-write the following idiom with numpy arrays:
>
> for i in some_list:
>   i.some_attr = some_value
>
> it would be wonderful if one was able to write this idiom as
>
> arr[just_these].some_attr = some_value
> or
> setattr(arr[just_these], 'some_attr', some_value)
>
> since often expensive loops through lists of object could be avoided.
>
> any thoughts on this are much appreciated,
>
> thanks,
>
> -jelle
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From pgmdevlist at gmail.com  Tue Oct 16 12:06:33 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 16 Oct 2007 12:06:33 -0400
Subject: [Numpy-discussion] setting the attributes of an array of object
In-Reply-To: <bc657ead0710160847n50b89c03ga783923b0751fe1a@mail.gmail.com>
References: <loom.20070929T085848-518@post.gmane.org>
	<bc657ead0710160847n50b89c03ga783923b0751fe1a@mail.gmail.com>
Message-ID: <200710161206.33717.pgmdevlist@gmail.com>

On Tuesday 16 October 2007 11:47:35 Sebastian Haase wrote:
> Hi,
> there is a way of doing this. As far as I know, you have to create
> your own "version of numpy arrays". E. g. try this:
...
> Be carefull that some (many / most ?) operations on that array will
> return you a normal numpy array again.
>
> I don't understand the reasons behind all of this.

Maybe
http://www.scipy.org/Subclasses
could give you some hints ?


From oliphant at enthought.com  Tue Oct 16 12:17:29 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Tue, 16 Oct 2007 11:17:29 -0500
Subject: [Numpy-discussion] setting the attributes of an array of object
In-Reply-To: <bc657ead0710160847n50b89c03ga783923b0751fe1a@mail.gmail.com>
References: <loom.20070929T085848-518@post.gmane.org>
	<bc657ead0710160847n50b89c03ga783923b0751fe1a@mail.gmail.com>
Message-ID: <4714E419.1040804@enthought.com>

Sebastian Haase wrote:
> Hi,
> there is a way of doing this. As far as I know, you have to create
> your own "version of numpy arrays". E. g. try this:
> class myNumpy(numpy.ndarray):
>      pass
>
> Then creating an instance as in
> a = myNumpy(otherNumpyArray)
>
> would make `a` behave just like any other array, except that you CAN
> attach attributes to it.
>
> Be carefull that some (many / most ?) operations on that array will
> return you a normal numpy array again.
>
> I don't understand the reasons behind all of this.
>   

The __array_priority__  attribute determines who "wins" when two 
different sub-classes are involved in an operation.   The ndarray has a 
priority of 0.0.

-Travis



From bhendrix at enthought.com  Tue Oct 16 12:15:39 2007
From: bhendrix at enthought.com (Bryce Hendrix)
Date: Tue, 16 Oct 2007 11:15:39 -0500
Subject: [Numpy-discussion] Partially merging scons into trunk ?
In-Reply-To: <47144501.3090608@ar.media.kyoto-u.ac.jp>
References: <471340F5.4060302@ar.media.kyoto-u.ac.jp>	<47139EA4.5080409@enthought.com>
	<47144501.3090608@ar.media.kyoto-u.ac.jp>
Message-ID: <4714E3AB.5010701@enthought.com>

David Cournapeau wrote:
> Bryce Hendrix wrote:
>   
>> I've only been following this thread on the perimeter, so I'm not sure
>> if "makefile" migration has been discussed. I have a script I wrote
>> about a year ago when we (Enthought) were looking at using SCons for our
>> internal builds. The script is capable of generating SConscript files
>> from setup.py scripts. It works by faking a build with distutils and
>> uses those build steps to generate the scons targets.
>>
>> The script is not complete, for example 'weave' confuses it, but it
>> could be a good starting point. If anyone is interested in it, I'll post
>> it to the list.
>>   
>>     
> That sounds interesting. Do I understand correctly that you ditched 
> distutils entirely ? That's not the approach I have taken, because I 
> thought it would be too big (before inclusion in numpy, I need to show 
> something remotely impressive, to convince the change worths it), and 
> would prevent doing things iteratively (e.g. building numpy wo distutils 
> would be a huge task). For most subpackages of scipy, which have small 
> setup.py, this would be useful, I guess (for the packages I know, they 
> are independant, at least as far as build issues are concerned).
>
>   

No, what I was investigating was whether we could auto-generate
Sconscript files from setup.py scripts so our builds could reliably only
build/test/document what was necessary. For example, our docs would take
> 10 minutes to re-generate the whole thing, but if only 1 file changed,
building the docs should take < 10 seconds. On top of that, there was a
summer of code project in 2006 to add packaging (msi and rpm were the
targeted formats), and we were looking at packaging in msi, rpm, and egg
formats. I eventually got frustrated with the speed at which the SCons
developers were willing to roll this change into the trunk, so I gave up
and we moved on to a different solution.

> I don't think any automated task would work for numpy:  I spent a good 
> deal of time last WE to be able to build all extensions in numpy with 
> scons, and it is a lot of work, because of bootstrapping issues. If you 
> take into account the code generators, the numpy.core "Makefile" is more 
> than 1000 lines of non trivial code. I finally got my head around it, 
> but I don't think any automated scheme would work in this case.
>   

Understood, especially since we could never auto-generate all of ours.
Just thought I'd offer it, since its just bit-rotting in our internal
repository right now.

Bryce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071016/59f89d80/attachment.html>

From julien.hillairet at gmail.com  Tue Oct 16 14:14:35 2007
From: julien.hillairet at gmail.com (Julien Hillairet)
Date: Tue, 16 Oct 2007 20:14:35 +0200
Subject: [Numpy-discussion] A basic question on the dot function
Message-ID: <8742df0c0710161114x552007eeya186b5f1ff4e4fa2@mail.gmail.com>

Hello,


First of all, I'm sorry if this question had already been asked. I've
searched on the gmane archive and elsewhere on internet, but I didn't found
the answer to my question.

As expected, the dot product of 2 'classical' vectors works fine :

In [50]: a0 = numpy.array([1,2,3])
In [51]: numpy.dot(a0,a0)
Out[51]: 14

What I don't understand, is why the dot product of a (3,N) vector gives an
error :

In [52]: a1 = numpy.array([[1,2,3],[1,2,3],[1,2,3],[1,2,3]])
In [54]: numpy.dot(a1,a1)
<type 'exceptions.ValueError'>            Traceback (most recent call last)
<type 'exceptions.ValueError'>: objects are not aligned

instead of what I've expected ; an array([14 14 14 14]).

Of course, I've found an alternative in doing :

In [61]: numpy.sum(a1*a1,axis=1)
Out[61]: array([14, 14, 14, 14])

But I like to understand my mistakes or misunderstands :) So, could someone
explained me what I've missed ? Thanks in advance.

Best regards,

JH
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071016/2bd8310f/attachment.html>

From wbaxter at gmail.com  Tue Oct 16 14:20:49 2007
From: wbaxter at gmail.com (Bill Baxter)
Date: Wed, 17 Oct 2007 03:20:49 +0900
Subject: [Numpy-discussion] A basic question on the dot function
In-Reply-To: <8742df0c0710161114x552007eeya186b5f1ff4e4fa2@mail.gmail.com>
References: <8742df0c0710161114x552007eeya186b5f1ff4e4fa2@mail.gmail.com>
Message-ID: <e86a5fd00710161120u38b188bfwf9f5d4370a822694@mail.gmail.com>

On 10/17/07, Julien Hillairet <julien.hillairet at gmail.com> wrote:
> Hello,
>
>
> First of all, I'm sorry if this question had already been asked. I've
> searched on the gmane archive and elsewhere on internet, but I didn't found
> the answer to my question.
>
> As expected, the dot product of 2 'classical' vectors works fine :
>
> In [50]: a0 = numpy.array([1,2,3])
> In [51]: numpy.dot(a0,a0)
> Out[51]: 14
>
> What I don't understand, is why the dot product of a (3,N) vector gives an
> error :

dot() also serves as Numpy's matrix multiply function.  So it's trying
to interpret that as a (3,N) matrix times a (3,N) matrix.

See examples here:
http://www.scipy.org/Numpy_Example_List_With_Doc#head-2a810f7dccd3f7c700d1076f15078ad1fe3c6d0d

--bb


From charlesr.harris at gmail.com  Tue Oct 16 14:23:36 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 16 Oct 2007 12:23:36 -0600
Subject: [Numpy-discussion] A basic question on the dot function
In-Reply-To: <8742df0c0710161114x552007eeya186b5f1ff4e4fa2@mail.gmail.com>
References: <8742df0c0710161114x552007eeya186b5f1ff4e4fa2@mail.gmail.com>
Message-ID: <e06186140710161123w2448fe38s5bbc9be35ba85eb2@mail.gmail.com>

On 10/16/07, Julien Hillairet <julien.hillairet at gmail.com> wrote:
>
> Hello,
>
>
> First of all, I'm sorry if this question had already been asked. I've
> searched on the gmane archive and elsewhere on internet, but I didn't found
> the answer to my question.
>
> As expected, the dot product of 2 'classical' vectors works fine :
>
> In [50]: a0 = numpy.array([1,2,3])
> In [51]: numpy.dot(a0,a0)
> Out[51]: 14
>
> What I don't understand, is why the dot product of a (3,N) vector gives an
> error :
>
> In [52]: a1 = numpy.array([[1,2,3],[1,2,3],[1,2,3],[1,2,3]])
> In [54]: numpy.dot(a1,a1)
> <type 'exceptions.ValueError'>            Traceback (most recent call
> last)
> <type 'exceptions.ValueError'>: objects are not aligned
>
> instead of what I've expected ; an array([14 14 14 14]).
>
> Of course, I've found an alternative in doing :
>
> In [61]: numpy.sum(a1*a1,axis=1)
> Out[61]: array([14, 14, 14, 14])
>
> But I like to understand my mistakes or misunderstands :) So, could
> someone explained me what I've missed ? Thanks in advance.


Dot is matrix multiplication, not the "dot" product you were expecting. It
is also a bit ambiguous, as you see with the 1-D vectors, where you got what
you expected.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071016/853db2e2/attachment.html>

From robert.kern at gmail.com  Tue Oct 16 14:24:44 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 16 Oct 2007 13:24:44 -0500
Subject: [Numpy-discussion] A basic question on the dot function
In-Reply-To: <8742df0c0710161114x552007eeya186b5f1ff4e4fa2@mail.gmail.com>
References: <8742df0c0710161114x552007eeya186b5f1ff4e4fa2@mail.gmail.com>
Message-ID: <471501EC.5020203@gmail.com>

Julien Hillairet wrote:
> Hello,
> 
> 
> First of all, I'm sorry if this question had already been asked. I've
> searched on the gmane archive and elsewhere on internet, but I didn't
> found the answer to my question.
> 
> As expected, the dot product of 2 'classical' vectors works fine :
> 
> In [50]: a0 = numpy.array([1,2,3])
> In [51]: numpy.dot(a0,a0)
> Out[51]: 14
> 
> What I don't understand, is why the dot product of a (3,N) vector gives
> an error :
> 
> In [52]: a1 = numpy.array([[1,2,3],[1,2,3],[1,2,3],[1,2,3]])
> In [54]: numpy.dot(a1,a1)
> <type 'exceptions.ValueError'>            Traceback (most recent call last)
> <type 'exceptions.ValueError'>: objects are not aligned
> 
> instead of what I've expected ; an array([14 14 14 14]).

When given two 2-D arrays, dot() essentially does matrix multiplication. The
last dimension of the first argument is matched with the next-to-last dimension
of the second argument.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From julien.hillairet at gmail.com  Tue Oct 16 15:55:06 2007
From: julien.hillairet at gmail.com (Julien Hillairet)
Date: Tue, 16 Oct 2007 21:55:06 +0200
Subject: [Numpy-discussion] A basic question on the dot function
In-Reply-To: <471501EC.5020203@gmail.com>
References: <8742df0c0710161114x552007eeya186b5f1ff4e4fa2@mail.gmail.com>
	<471501EC.5020203@gmail.com>
Message-ID: <8742df0c0710161255kce738cfgf21b8fd43e2dcd84@mail.gmail.com>

2007/10/16, Bill Baxter <wbaxter at gmail.com>:
>
> dot() also serves as Numpy's matrix multiply function.  So it's trying
> to interpret that as a (3,N) matrix times a (3,N) matrix.
>
> See examples here:
>
> http://www.scipy.org/Numpy_Example_List_With_Doc#head-2a810f7dccd3f7c700d1076f15078ad1fe3c6d0d
>
> --bb
>

2007/10/16, Charles R Harris <charlesr.harris at gmail.com>:
>
>
>
> Dot is matrix multiplication, not the "dot" product you were expecting. It
> is also a bit ambiguous, as you see with the 1-D vectors, where you got what
> you expected.
>
> Chuck
>


2007/10/16, Robert Kern <robert.kern at gmail.com>:
>
> When given two 2-D arrays, dot() essentially does matrix multiplication.
> The
> last dimension of the first argument is matched with the next-to-last
> dimension
> of the second argument.
>
> --
> Robert Kern


Thank you for your answers. So, is there a "proper" solution to do the dot
product as I had expected it ?

Cheers,

JH
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071016/f2f4f66c/attachment.html>

From tim.hochberg at ieee.org  Tue Oct 16 16:08:30 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Tue, 16 Oct 2007 13:08:30 -0700
Subject: [Numpy-discussion] A basic question on the dot function
In-Reply-To: <8742df0c0710161255kce738cfgf21b8fd43e2dcd84@mail.gmail.com>
References: <8742df0c0710161114x552007eeya186b5f1ff4e4fa2@mail.gmail.com>
	<471501EC.5020203@gmail.com>
	<8742df0c0710161255kce738cfgf21b8fd43e2dcd84@mail.gmail.com>
Message-ID: <e4412d6b0710161308j57310257t913cf8a1dbdaa108@mail.gmail.com>

On 10/16/07, Julien Hillairet <julien.hillairet at gmail.com> wrote:
>
> 2007/10/16, Bill Baxter <wbaxter at gmail.com>:
> >
> > dot() also serves as Numpy's matrix multiply function.  So it's trying
> > to interpret that as a (3,N) matrix times a (3,N) matrix.
> >
> > See examples here:
> >
> > http://www.scipy.org/Numpy_Example_List_With_Doc#head-2a810f7dccd3f7c700d1076f15078ad1fe3c6d0d
> >
> > --bb
> >
>
> 2007/10/16, Charles R Harris < charlesr.harris at gmail.com>:
> >
> >
> >
> > Dot is matrix multiplication, not the "dot" product you were expecting.
> > It is also a bit ambiguous, as you see with the 1-D vectors, where you got
> > what you expected.
> >
> > Chuck
> >
>
>
> 2007/10/16, Robert Kern <robert.kern at gmail.com>:
> >
> > When given two 2-D arrays, dot() essentially does matrix multiplication.
> > The
> > last dimension of the first argument is matched with the next-to-last
> > dimension
> > of the second argument.
> >
> > --
> > Robert Kern
>
>
> Thank you for your answers. So, is there a "proper" solution to do the dot
> product as I had expected it ?
>

You might try tensordot. Without thinking it through too much:
numpy.tensordot(a0, a1, axes=[-1,-1])
seems to do what you want.



-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071016/57572164/attachment.html>

From renato.serodio at gmail.com  Wed Oct 17 03:33:36 2007
From: renato.serodio at gmail.com (Renato Serodio)
Date: Wed, 17 Oct 2007 09:33:36 +0200
Subject: [Numpy-discussion] abstraction and interfaces
In-Reply-To: <Mahogany-0.67.0-1524-20071012-092050.00@american.edu>
References: <4f4101bf0710120554y1b57d351n9157663db993916a@mail.gmail.com>
	<Mahogany-0.67.0-1524-20071012-092050.00@american.edu>
Message-ID: <4f4101bf0710170033l1d5d5306maf8a0f4fb024e997@mail.gmail.com>

Hello there,

thanks to your pointer, I've progressed further on the OO concept, and
am currently building analysis, inputData and outputResults interfaces
that should add some flexibility to my program.

On the other hand, pulling the OO and interfaces string opened a box
full of little critters, and now I'm learning about UML, some advanced
class usage and so on. My brain hurts..

Cheers,

Renato


On 12/10/2007, Alan G Isaac <aisaac at american.edu> wrote:
> On Fri, 12 Oct 2007, Renato Serodio apparently wrote:
> > The scripts that produce these metrics use Scipy/Numpy
> > functions that operate on data conveniently converted to
> > numpy arrays. They're quite specific, and I tend to
> > produce/tweak a lot of them. So, to fit in this
> > application someone suggested I programmed 'interfaces'
> > (in java jargon) to them - that way I could develop the
> > whole wrapper application without giving much thought to
> > the actual number-crunching bits.
>
> That sounds quite right.  Check out
> <URL:https://projects.scipy.org/scipy/scikits/browser/trunk/openopt/scikits/openopt/solvers/optimizers/optimizer>
> <URL:http://svn.scipy.org/svn/scipy/trunk/scipy/stats/models/>
> for examples that may be relevant to your project.
>
> Python does not have interfaces per se, but that does
> not stop you from designing interface-like classes and
> inheriting from them.
>
> fwiw,
> Alan Isaac
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From matthieu.brucher at gmail.com  Wed Oct 17 03:59:22 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 17 Oct 2007 09:59:22 +0200
Subject: [Numpy-discussion] abstraction and interfaces
In-Reply-To: <4f4101bf0710170033l1d5d5306maf8a0f4fb024e997@mail.gmail.com>
References: <4f4101bf0710120554y1b57d351n9157663db993916a@mail.gmail.com>
	<Mahogany-0.67.0-1524-20071012-092050.00@american.edu>
	<4f4101bf0710170033l1d5d5306maf8a0f4fb024e997@mail.gmail.com>
Message-ID: <e76aa17f0710170059r68a99a2cq835b76ccf4ee215a@mail.gmail.com>

What is more, with the concept of duck-typing, you just have to provide the
interface without inheriting from a mother class (which is what I do for the
optimizer framework, the other sub-modules do not derive from a common
ancestor).

Matthieu

2007/10/17, Renato Serodio <renato.serodio at gmail.com>:
>
> Hello there,
>
> thanks to your pointer, I've progressed further on the OO concept, and
> am currently building analysis, inputData and outputResults interfaces
> that should add some flexibility to my program.
>
> On the other hand, pulling the OO and interfaces string opened a box
> full of little critters, and now I'm learning about UML, some advanced
> class usage and so on. My brain hurts..
>
> Cheers,
>
> Renato
>
>
> On 12/10/2007, Alan G Isaac <aisaac at american.edu> wrote:
> > On Fri, 12 Oct 2007, Renato Serodio apparently wrote:
> > > The scripts that produce these metrics use Scipy/Numpy
> > > functions that operate on data conveniently converted to
> > > numpy arrays. They're quite specific, and I tend to
> > > produce/tweak a lot of them. So, to fit in this
> > > application someone suggested I programmed 'interfaces'
> > > (in java jargon) to them - that way I could develop the
> > > whole wrapper application without giving much thought to
> > > the actual number-crunching bits.
> >
> > That sounds quite right.  Check out
> > <URL:
> https://projects.scipy.org/scipy/scikits/browser/trunk/openopt/scikits/openopt/solvers/optimizers/optimizer
> >
> > <URL:http://svn.scipy.org/svn/scipy/trunk/scipy/stats/models/>
> > for examples that may be relevant to your project.
> >
> > Python does not have interfaces per se, but that does
> > not stop you from designing interface-like classes and
> > inheriting from them.
> >
> > fwiw,
> > Alan Isaac
> >
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/632895b9/attachment.html>

From julien.hillairet at gmail.com  Wed Oct 17 04:11:36 2007
From: julien.hillairet at gmail.com (Julien Hillairet)
Date: Wed, 17 Oct 2007 10:11:36 +0200
Subject: [Numpy-discussion] A basic question on the dot function
In-Reply-To: <e4412d6b0710161308j57310257t913cf8a1dbdaa108@mail.gmail.com>
References: <8742df0c0710161114x552007eeya186b5f1ff4e4fa2@mail.gmail.com>
	<471501EC.5020203@gmail.com>
	<8742df0c0710161255kce738cfgf21b8fd43e2dcd84@mail.gmail.com>
	<e4412d6b0710161308j57310257t913cf8a1dbdaa108@mail.gmail.com>
Message-ID: <8742df0c0710170111v3323c63bm2e6e5d23ff7be9a3@mail.gmail.com>

2007/10/16, Timothy Hochberg <tim.hochberg at ieee.org>:
>
>
> You might try tensordot. Without thinking it through too much:
> numpy.tensordot(a0, a1, axes=[-1,-1])
> seems to do what you want.
>
>
Thank you.

However, it works only for this simple example, where a0 and a1 are similar.
The tensor product increase the rank of the output, doesn't it ?
Although the dot product decrease the rank. Is there a ?proper" solution if
a and b are general (3,N) vectors ?  By example :

In [16]: a = random.random_sample((3,5))
In [17]: b = random.random_sample((3,5))

what I'm searching for is :

In [18]: dotprod2(a,b)
Out[18]: array([ 0.28354876,  0.54474092,  0.22986942,  0.42822669,
0.98179793])

where I defined a "classical" (in the way I understand it. I may not
understand it properly ?) dot product between these 2 vectors.

def dotprod2(a,b):
  return sum(a*b,axis=0)

or in maths notation : c_j = \sum_i a_{ij} b_{ij}

Thank in advance,

JH
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/ff4b5ead/attachment.html>

From matthieu.brucher at gmail.com  Wed Oct 17 04:16:43 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 17 Oct 2007 10:16:43 +0200
Subject: [Numpy-discussion] A basic question on the dot function
In-Reply-To: <8742df0c0710170111v3323c63bm2e6e5d23ff7be9a3@mail.gmail.com>
References: <8742df0c0710161114x552007eeya186b5f1ff4e4fa2@mail.gmail.com>
	<471501EC.5020203@gmail.com>
	<8742df0c0710161255kce738cfgf21b8fd43e2dcd84@mail.gmail.com>
	<e4412d6b0710161308j57310257t913cf8a1dbdaa108@mail.gmail.com>
	<8742df0c0710170111v3323c63bm2e6e5d23ff7be9a3@mail.gmail.com>
Message-ID: <e76aa17f0710170116t3be12164ubd18aa6ef971cfb4@mail.gmail.com>

>
> what I'm searching for is :
>
> In [18]: dotprod2(a,b)
> Out[18]: array([ 0.28354876,  0.54474092,  0.22986942,  0.42822669,
> 0.98179793])
>
> where I defined a "classical" (in the way I understand it. I may not
> understand it properly ?) dot product between these 2 vectors.
>
> def dotprod2(a,b):
>   return sum(a*b,axis=0)
>

I think this is your best choice ;)

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/349d26c0/attachment.html>

From markbak at gmail.com  Wed Oct 17 08:16:55 2007
From: markbak at gmail.com (mark)
Date: Wed, 17 Oct 2007 12:16:55 -0000
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <799406d60710111153w496563cbwe65f0aaffd0d7d1@mail.gmail.com>
References: <799406d60710110741k1937e8a9te6887d200e28dc7c@mail.gmail.com>
	<627102C921CD9745B070C3B10CB8199B049D53CB@hardwire.esri.com>
	<799406d60710111153w496563cbwe65f0aaffd0d7d1@mail.gmail.com>
Message-ID: <1192623415.011110.306040@v23g2000prn.googlegroups.com>

So it seems like lists are the way to grow an array.

Interestingly enough, it is very easy to grown an array in Matlab.
Any idea how they do that (or are they slow as well?).

Mark

On Oct 11, 8:53 pm, "Adam Mercer" <ramer... at gmail.com> wrote:
> On 11/10/2007, Mark Janikas <mjani... at esri.com> wrote:
>
> > If you do not know the size of your array before you finalize it, then
> > you should use lists whenever you can.  I just cooked up a short
> > example:
>
> <snip>
>
> > ##### Result #####
> > Total Time with array: 2.12951189331
> > Total Time with list: 0.0469707035741
> > ########################################################################
> > ####
>
> > Hope this helps,
>
> That is helpful, I thought that using arrays would be much faster but
> its clearly not in this case.
>
> Thanks
>
> Adam
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion



From matthieu.brucher at gmail.com  Wed Oct 17 08:34:17 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 17 Oct 2007 14:34:17 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <4713C111.5010609@enthought.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
	<4713C111.5010609@enthought.com>
Message-ID: <e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>

>
> There are two types of errors that can occur with reference counting on
> data-types.
>
> 1) There are too many DECREF's --- this gets us to the error quickly and
> is usually easy to reproduce
> 2) There are too many INCREF's (the reference count keeps going up until
> the internal counter wraps around to 0 and deallocation is attempted)
> --- this error is harder to reproduce and usually takes a while before
> it happens in the code.
>

The error showed up again with a simple operation :
  File "/home/brucher/local/src/include/toolbox/stats/kernels/gaussian.py",
line 60, in __call__
    xp = (x-self.loc) * (1/self.scale)
Where x and self.loc are arrays abd self.scale is an integer.

The error as it was given :

*** Reference count error detected:
an attempt was made to deallocate 7 (l) ***
*** Reference count error detected:
an attempt was made to deallocate 7 (l) ***
*** Reference count error detected:
an attempt was made to deallocate 7 (l) ***

I don't know if it is too many INCREF, but I doubt it. It happens in a loop
where I treat a bunch of 3D images each time (64*64*64*3), and it crashes
after some dozens of them. I have no problem when I treat small 2D images
(128*128), but far more samples.


Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/88441726/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Wed Oct 17 08:45:37 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 17 Oct 2007 21:45:37 +0900
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>	<47137A2E.6030006@enthought.com>	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>	<4713C111.5010609@enthought.com>
	<e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>
Message-ID: <471603F1.9090700@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
>     There are two types of errors that can occur with reference
>     counting on
>     data-types.
>
>     1) There are too many DECREF's --- this gets us to the error
>     quickly and
>     is usually easy to reproduce
>     2) There are too many INCREF's (the reference count keeps going up
>     until
>     the internal counter wraps around to 0 and deallocation is attempted)
>     --- this error is harder to reproduce and usually takes a while before
>     it happens in the code.
>
>
> The error showed up again with a simple operation :
>   File 
> "/home/brucher/local/src/include/toolbox/stats/kernels/gaussian.py", 
> line 60, in __call__
>     xp = (x-self.loc) * (1/self.scale)
> Where x and self.loc are arrays abd self.scale is an integer.
>
> The error as it was given :
>
> *** Reference count error detected:
> an attempt was made to deallocate 7 (l) ***
> *** Reference count error detected:
> an attempt was made to deallocate 7 (l) ***
> *** Reference count error detected:
> an attempt was made to deallocate 7 (l) ***
>
> I don't know if it is too many INCREF, but I doubt it. It happens in a 
> loop where I treat a bunch of 3D images each time (64*64*64*3), and it 
> crashes after some dozens of them. I have no problem when I treat 
> small 2D images (128*128), but far more samples.
Can't you emulate this behaviour with signals different than images ? 
(say random signal of 64*64*64*3 samples). If the process does not 
require a long processing time (say a couple of minutes), then you may 
be able to use massif tool from valgrind, which may be helpful to detect 
too many INCREF. For DECREF, then the default memory checker from 
valgrind should be useful as well.

If it does take only a couple of minutes, it is then relatively easy to 
"bisect" the code to spot the error (once you get through the C level, 
it is easy to see the problem, if you can reproduce it, in my experience).

cheers,

David


From cckidder at gmail.com  Wed Oct 17 08:58:50 2007
From: cckidder at gmail.com (Chad Kidder)
Date: Wed, 17 Oct 2007 07:58:50 -0500
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <1192623415.011110.306040@v23g2000prn.googlegroups.com>
References: <799406d60710110741k1937e8a9te6887d200e28dc7c@mail.gmail.com>
	<627102C921CD9745B070C3B10CB8199B049D53CB@hardwire.esri.com>
	<799406d60710111153w496563cbwe65f0aaffd0d7d1@mail.gmail.com>
	<1192623415.011110.306040@v23g2000prn.googlegroups.com>
Message-ID: <64AA0379-5277-4395-B14C-E5AFE2BC5169@gmail.com>

Growing an array by appending it is the slow way in matlab.  The  
suggested way to do things there is preallocate the array by saying  
x=zeros(....) and then referencing the elements in the array and  
inserting the correct value.


--Chad Kidder


On Oct 17, 2007, at 7:16 AM, mark wrote:

> So it seems like lists are the way to grow an array.
>
> Interestingly enough, it is very easy to grown an array in Matlab.
> Any idea how they do that (or are they slow as well?).
>
> Mark
>
> On Oct 11, 8:53 pm, "Adam Mercer" <ramer... at gmail.com> wrote:
>> On 11/10/2007, Mark Janikas <mjani... at esri.com> wrote:
>>
>>> If you do not know the size of your array before you finalize it,  
>>> then
>>> you should use lists whenever you can.  I just cooked up a short
>>> example:
>>
>> <snip>
>>
>>> ##### Result #####
>>> Total Time with array: 2.12951189331
>>> Total Time with list: 0.0469707035741
>>> #################################################################### 
>>> ####
>>> ####
>>
>>> Hope this helps,
>>
>> That is helpful, I thought that using arrays would be much faster but
>> its clearly not in this case.
>>
>> Thanks
>>
>> Adam
>> _______________________________________________
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/87df94e6/attachment.html>

From matthieu.brucher at gmail.com  Wed Oct 17 08:59:45 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 17 Oct 2007 14:59:45 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <471603F1.9090700@ar.media.kyoto-u.ac.jp>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
	<4713C111.5010609@enthought.com>
	<e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>
	<471603F1.9090700@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>

>
> Can't you emulate this behaviour with signals different than images ?
> (say random signal of 64*64*64*3 samples).



I wish I could, but this behaviour only shows up on this peculiar data set
:(


If the process does not
> require a long processing time (say a couple of minutes), then you may
> be able to use massif tool from valgrind, which may be helpful to detect
> too many INCREF. For DECREF, then the default memory checker from
> valgrind should be useful as well.
>
> If it does take only a couple of minutes, it is then relatively easy to
> "bisect" the code to spot the error (once you get through the C level,
> it is easy to see the problem, if you can reproduce it, in my experience).


Unfortunately, the process is very long, there are several optimizations in
the process, the whole thing in a EM-like algorithm, and the crash does not
occur with the first image, it is later.

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/387fcc50/attachment.html>

From matthieu.brucher at gmail.com  Wed Oct 17 09:04:12 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 17 Oct 2007 15:04:12 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
	<4713C111.5010609@enthought.com>
	<e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>
	<471603F1.9090700@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>
Message-ID: <e76aa17f0710170604y16f00552ybac7d075f46175bc@mail.gmail.com>

Oups, I'm wrong, the process happened this time with the 128? images and not
the 3D images. I'll try to check a little bit further.

Matthieu

2007/10/17, Matthieu Brucher <matthieu.brucher at gmail.com>:
>
> Can't you emulate this behaviour with signals different than images ?
> > (say random signal of 64*64*64*3 samples).
>
>
>
> I wish I could, but this behaviour only shows up on this peculiar data set
> :(
>
>
> If the process does not
> > require a long processing time (say a couple of minutes), then you may
> > be able to use massif tool from valgrind, which may be helpful to detect
> > too many INCREF. For DECREF, then the default memory checker from
> > valgrind should be useful as well.
> >
> > If it does take only a couple of minutes, it is then relatively easy to
> > "bisect" the code to spot the error (once you get through the C level,
> > it is easy to see the problem, if you can reproduce it, in my
> > experience).
>
>
> Unfortunately, the process is very long, there are several optimizations
> in the process, the whole thing in a EM-like algorithm, and the crash does
> not occur with the first image, it is later.
>
> Matthieu
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/c9b48110/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Wed Oct 17 08:56:30 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 17 Oct 2007 21:56:30 +0900
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>	<47137A2E.6030006@enthought.com>	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>	<4713C111.5010609@enthought.com>	<e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>	<471603F1.9090700@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>
Message-ID: <4716067E.10400@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
>     Can't you emulate this behaviour with signals different than images ?
>     (say random signal of 64*64*64*3 samples). 
>
>
>
> I wish I could, but this behaviour only shows up on this peculiar data 
> set :(
It always happens at the same time ?
>
> Unfortunately, the process is very long, there are several 
> optimizations in the process, the whole thing in a EM-like algorithm, 
> and the crash does not occur with the first image, it is later.
Can you observe some 'funky' memory behaviour while the process is 
running ? For example, does the memory keeps increasing after each dataset ?



From matthieu.brucher at gmail.com  Wed Oct 17 09:20:45 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 17 Oct 2007 15:20:45 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <4716067E.10400@ar.media.kyoto-u.ac.jp>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
	<4713C111.5010609@enthought.com>
	<e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>
	<471603F1.9090700@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>
	<4716067E.10400@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0710170620v40139afbjb498f6c811561c25@mail.gmail.com>

>
> > I wish I could, but this behaviour only shows up on this peculiar data
> > set :(
> It always happens at the same time ?


Yes.
Perhaps debugging the process will sort things out ?


> Unfortunately, the process is very long, there are several
> > optimizations in the process, the whole thing in a EM-like algorithm,
> > and the crash does not occur with the first image, it is later.
> Can you observe some 'funky' memory behaviour while the process is
> running ? For example, does the memory keeps increasing after each dataset
> ?
>

Good question, in fact it should because I'm saving the result in a list
before I pickle it (but in fact I have to change this process as pickle
cannot handle the images I'm using, but that's another problem). So it
should increase, but not too much. I'll check this.

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/9cc79b51/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Wed Oct 17 09:27:24 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 17 Oct 2007 22:27:24 +0900
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710170620v40139afbjb498f6c811561c25@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>	<47137A2E.6030006@enthought.com>	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>	<4713C111.5010609@enthought.com>	<e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>	<471603F1.9090700@ar.media.kyoto-u.ac.jp>	<e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>	<4716067E.10400@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170620v40139afbjb498f6c811561c25@mail.gmail.com>
Message-ID: <47160DBC.6050405@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
>     > I wish I could, but this behaviour only shows up on this
>     peculiar data
>     > set :(
>     It always happens at the same time ?
>
>
> Yes.
> Perhaps debugging the process will sort things out ?
As said, my approach to debugging this kind of thing is to get out of 
python ASAP. And once you manage to reproduce the result when calling 
only a couple of python functions, then you use massif or memcheck.

The reason to get out of python is that tools like valgrind can only 
give you meaningful informations at the C level, and it is difficult to 
make the link between C and python calls, if not impossible in all but 
trivial cases.

But when you can get this kind of graphs:

http://valgrind.org/docs/manual/ms-manual.html

Then the problem is "solved", at least in my limited experience.

David


From matthieu.brucher at gmail.com  Wed Oct 17 09:44:39 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 17 Oct 2007 15:44:39 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <47160DBC.6050405@ar.media.kyoto-u.ac.jp>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
	<4713C111.5010609@enthought.com>
	<e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>
	<471603F1.9090700@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>
	<4716067E.10400@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170620v40139afbjb498f6c811561c25@mail.gmail.com>
	<47160DBC.6050405@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0710170644o434fa4daq4cda035f32889c7b@mail.gmail.com>

>
> As said, my approach to debugging this kind of thing is to get out of
> python ASAP. And once you manage to reproduce the result when calling
> only a couple of python functions, then you use massif or memcheck.


I agree with you, the problem is that I do not use directly C functions and
that I do not know how I can reproduce the result with a minimal example.
I know I've had once a Numpy problem that was solved by recompiling Numpy
completely, but here, I do not possess a first clue to help me track this
bug.

The reason to get out of python is that tools like valgrind can only
> give you meaningful informations at the C level, and it is difficult to
> make the link between C and python calls, if not impossible in all but
> trivial cases.
>
> But when you can get this kind of graphs:
>
> http://valgrind.org/docs/manual/ms-manual.html
>
> Then the problem is "solved", at least in my limited experience.


Compltely agreed.

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/618cb989/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Wed Oct 17 09:42:32 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 17 Oct 2007 22:42:32 +0900
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710170644o434fa4daq4cda035f32889c7b@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>	<47137A2E.6030006@enthought.com>	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>	<4713C111.5010609@enthought.com>	<e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>	<471603F1.9090700@ar.media.kyoto-u.ac.jp>	<e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>	<4716067E.10400@ar.media.kyoto-u.ac.jp>	<e76aa17f0710170620v40139afbjb498f6c811561c25@mail.gmail.com>	<47160DBC.6050405@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170644o434fa4daq4cda035f32889c7b@mail.gmail.com>
Message-ID: <47161148.70705@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
>     As said, my approach to debugging this kind of thing is to get out of
>     python ASAP. And once you manage to reproduce the result when calling
>     only a couple of python functions, then you use massif or memcheck.
>
>
> I agree with you, the problem is that I do not use directly C 
> functions and that I do not know how I can reproduce the result with a 
> minimal example.
Yes, that's why I suggested looking at the memory usage (using top or 
something else). Because maybe the problem can be spotted long before 
seeing it happening, hence having smaller code (which you could e.g. 
post, etc...)

But if you can reproduce it with 128*128 images, then maybe it is quick 
enough to run the whole thing under massif ?

cheers,

David


From matthieu.brucher at gmail.com  Wed Oct 17 10:00:01 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 17 Oct 2007 16:00:01 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <47161148.70705@ar.media.kyoto-u.ac.jp>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<4713C111.5010609@enthought.com>
	<e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>
	<471603F1.9090700@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>
	<4716067E.10400@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170620v40139afbjb498f6c811561c25@mail.gmail.com>
	<47160DBC.6050405@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170644o434fa4daq4cda035f32889c7b@mail.gmail.com>
	<47161148.70705@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0710170700n276f46f2i1ff38cf9484b0153@mail.gmail.com>

>
> > I agree with you, the problem is that I do not use directly C
> > functions and that I do not know how I can reproduce the result with a
> > minimal example.
> Yes, that's why I suggested looking at the memory usage (using top or
> something else). Because maybe the problem can be spotted long before
> seeing it happening, hence having smaller code (which you could e.g.
> post, etc...)



For the moment nothing unusual, the memory use seems to be stable.


But if you can reproduce it with 128*128 images, then maybe it is quick
> enough to run the whole thing under massif ?


I'll try it this night (because it is very very long, so with the
simulator...)

Thanks for the help ;)

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/78d25918/attachment.html>

From rbastian at free.fr  Wed Oct 17 09:56:51 2007
From: rbastian at free.fr (=?iso-8859-1?q?Ren=E9=20Bastian?=)
Date: Wed, 17 Oct 2007 15:56:51 +0200
Subject: [Numpy-discussion] appending extra items to arrays
In-Reply-To: <64AA0379-5277-4395-B14C-E5AFE2BC5169@gmail.com>
References: <799406d60710110741k1937e8a9te6887d200e28dc7c@mail.gmail.com>
	<1192623415.011110.306040@v23g2000prn.googlegroups.com>
	<64AA0379-5277-4395-B14C-E5AFE2BC5169@gmail.com>
Message-ID: <07101715565103.00752@rbastian>

Hello,

I work with numarray.zeros(n, numarray.Float64) as sound mixers ;
there are huge number of datas : 44 000 .. 192 000 /second

Operations : add, append, add & append (if the mixing begins
on the existing part of the array + append if the array has to
be prolonged)

I do never use list-appending but I concatenate a zero-array
with a fixed length (which can be given e.g. sr/10 )

This way is much more faster then appending with lists
if there are huge number of datas (for small arrays,
lists are faster, but I have never a sound of 10 samples :-)


 
-- 
Ren? Bastian
http://www.musiques-rb.org
http://pythoneon.musiques-rb.org




From david at ar.media.kyoto-u.ac.jp  Wed Oct 17 10:14:28 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 17 Oct 2007 23:14:28 +0900
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710170700n276f46f2i1ff38cf9484b0153@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>	<4713C111.5010609@enthought.com>	<e76aa17f0710170534l6537d6a9qcfcbe8f3d4ad1069@mail.gmail.com>	<471603F1.9090700@ar.media.kyoto-u.ac.jp>	<e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>	<4716067E.10400@ar.media.kyoto-u.ac.jp>	<e76aa17f0710170620v40139afbjb498f6c811561c25@mail.gmail.com>	<47160DBC.6050405@ar.media.kyoto-u.ac.jp>	<e76aa17f0710170644o434fa4daq4cda035f32889c7b@mail.gmail.com>	<47161148.70705@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170700n276f46f2i1ff38cf9484b0153@mail.gmail.com>
Message-ID: <471618C4.9090000@ar.media.kyoto-u.ac.jp>

Matthieu Brucher wrote:
>
>     > I agree with you, the problem is that I do not use directly C
>     > functions and that I do not know how I can reproduce the result
>     with a
>     > minimal example.
>     Yes, that's why I suggested looking at the memory usage (using top or
>     something else). Because maybe the problem can be spotted long before
>     seeing it happening, hence having smaller code (which you could e.g.
>     post, etc...)
>
>
>
> For the moment nothing unusual, the memory use seems to be stable.
>  
>
>     But if you can reproduce it with 128*128 images, then maybe it is
>     quick
>     enough to run the whole thing under massif ?
>
>
> I'll try it this night (because it is very very long, so with the 
> simulator...)
Yes, that's the worse cases, of course. For those cases, I wish we could 
use numpy with COUNT_ALLOCS. Unfortunately, using numpy in this case is 
impossible (it crashes at import, and I've never tracked down the 
problem; maybe your problem may be enough of an incentive to start again):

http://mail.python.org/pipermail/python-list/2003-May/206256.html

cheers,

David


From matthieu.brucher at gmail.com  Wed Oct 17 10:56:50 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 17 Oct 2007 16:56:50 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <471618C4.9090000@ar.media.kyoto-u.ac.jp>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<471603F1.9090700@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>
	<4716067E.10400@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170620v40139afbjb498f6c811561c25@mail.gmail.com>
	<47160DBC.6050405@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170644o434fa4daq4cda035f32889c7b@mail.gmail.com>
	<47161148.70705@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170700n276f46f2i1ff38cf9484b0153@mail.gmail.com>
	<471618C4.9090000@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0710170756u5c119e1dje42c63e17734eedc@mail.gmail.com>

>
> > I'll try it this night (because it is very very long, so with the
> > simulator...)
> Yes, that's the worse cases, of course. For those cases, I wish we could
> use numpy with COUNT_ALLOCS. Unfortunately, using numpy in this case is
> impossible (it crashes at import, and I've never tracked down the
> problem; maybe your problem may be enough of an incentive to start again):
>
> http://mail.python.org/pipermail/python-list/2003-May/206256.html



The problem is I don't have time for this at the moment, I have to develop
my algorithm for my PhD, and if one does not work, I'll try another one, but
this is strange because this algorithm worked in the past...

BTW, I can't use massif. Although the normal algorithm does not use more
than 150MB, with massif, it goes higher than 2.7GB, and this is before the
really hungry process.

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/e9da8978/attachment.html>

From renato.serodio at gmail.com  Wed Oct 17 11:11:22 2007
From: renato.serodio at gmail.com (Renato Serodio)
Date: Wed, 17 Oct 2007 17:11:22 +0200
Subject: [Numpy-discussion] design patterns for computing
Message-ID: <4f4101bf0710170811j7f8c9ad5nc9e3ac8b5d361443@mail.gmail.com>

Hello there,

some helpful pointers provided in the list have put me on track to ask
further questions. Now that I'm acquainted with simulating interfaces
and duck-typing (which is still a bit fuzzy to me) I think the
information I've been looking for all along are the so-called 'design
patterns'.

Looking at the Optimizer and Stats examples, I could see how new
objects were created that built upon a basis class. Stats even had a
class to contain calculation results, which brought me even closer to
the point. But now I'm stuck again..

I imagined three interfaces to be designed, through which the main
program controls the number-crunching without constraining its
implementation. These would be:
- getInputs: responsible for passing/preparing data and parameters for
the analysis/computation
- Analysis: which implements all the crunching from FFTs to esoteric
stuff like genetic algorithms (and which should be considered opaque
from the main program POV)
- AnalysisResults: to somehow get the results back into the main program.

An additional class would cater for plotting properties, which are
usually associated with a      particular analysis. Finally,
anticipating that I may want to cascade analyses, the inputs and
results must be made uniform - must be of the same form, likely numpy
arrays.

Well, with this pattern, to implement an analysis, I'd have to define
derivated getInputs and AnalysisResults classes, and code Analysis to
accept and return arrays. A flag to the analysis object could be used
to generate plot information or not.

(Hey, writing this email is already helping, I'll put the gist in my
documentation!)

So, after this brainstorming I'll do some more on the framework. I'd
be most grateful for comments on the design pattern - is it sensible,
could it be better? Are there any good literature sources on patterns
for this type of work?

Thanks,

Renato


From cournape at gmail.com  Wed Oct 17 11:13:34 2007
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 18 Oct 2007 00:13:34 +0900
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710170756u5c119e1dje42c63e17734eedc@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<e76aa17f0710170559v73332fa1n5a6840d78b14579b@mail.gmail.com>
	<4716067E.10400@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170620v40139afbjb498f6c811561c25@mail.gmail.com>
	<47160DBC.6050405@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170644o434fa4daq4cda035f32889c7b@mail.gmail.com>
	<47161148.70705@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170700n276f46f2i1ff38cf9484b0153@mail.gmail.com>
	<471618C4.9090000@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170756u5c119e1dje42c63e17734eedc@mail.gmail.com>
Message-ID: <5b8d13220710170813g16a2178aq939604029d87f7b8@mail.gmail.com>

On 10/17/07, Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
>
> > > I'll try it this night (because it is very very long, so with the
> > > simulator...)
> > Yes, that's the worse cases, of course. For those cases, I wish we could
> > use numpy with COUNT_ALLOCS. Unfortunately, using numpy in this case is
> > impossible (it crashes at import, and I've never tracked down the
> > problem; maybe your problem may be enough of an incentive to start again):
> >
> >
> http://mail.python.org/pipermail/python-list/2003-May/206256.html
>
>
> The problem is I don't have time for this at the moment, I have to develop
> my algorithm for my PhD, and if one does not work, I'll try another one, but
> this is strange because this algorithm worked in the past...

By incentive, I meant incentive for me, not for you :) I think this
could be really helpful for other problems if this kind as well.

>
> BTW, I can't use massif. Although the normal algorithm does not use more
> than 150MB, with massif, it goes higher than 2.7GB, and this is before the
> really hungry process.
Using massif with code using a large number of python code is clearly
a no no, since it uses so much malloc. Sometimes, you can just stop
the thing when it is growing up, and watching the resulting graphs.
But this is no fun, and if you have no time...

David


From matthieu.brucher at gmail.com  Wed Oct 17 11:21:15 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Wed, 17 Oct 2007 17:21:15 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <5b8d13220710170813g16a2178aq939604029d87f7b8@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<4716067E.10400@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170620v40139afbjb498f6c811561c25@mail.gmail.com>
	<47160DBC.6050405@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170644o434fa4daq4cda035f32889c7b@mail.gmail.com>
	<47161148.70705@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170700n276f46f2i1ff38cf9484b0153@mail.gmail.com>
	<471618C4.9090000@ar.media.kyoto-u.ac.jp>
	<e76aa17f0710170756u5c119e1dje42c63e17734eedc@mail.gmail.com>
	<5b8d13220710170813g16a2178aq939604029d87f7b8@mail.gmail.com>
Message-ID: <e76aa17f0710170821q79be9169gb9b7cf2139895530@mail.gmail.com>

>
> > The problem is I don't have time for this at the moment, I have to
> develop
> > my algorithm for my PhD, and if one does not work, I'll try another one,
> but
> > this is strange because this algorithm worked in the past...
>
> By incentive, I meant incentive for me, not for you :) I think this
> could be really helpful for other problems if this kind as well.



But as you have the same problems than me ;)


> BTW, I can't use massif. Although the normal algorithm does not use more
> > than 150MB, with massif, it goes higher than 2.7GB, and this is before
> the
> > really hungry process.
> Using massif with code using a large number of python code is clearly
> a no no, since it uses so much malloc. Sometimes, you can just stop
> the thing when it is growing up, and watching the resulting graphs.
> But this is no fun, and if you have no time...



It's manily that it crashes when the second file is loaded into memory, the
model that will be used for this data isn't even loaded and thus the value
of the result is zero :( Strange though (IMHM) that massif needs to allocate
so much memory for this little data.

Matthieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071017/1e5ac10c/attachment.html>

From aisaac at american.edu  Wed Oct 17 11:36:25 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 17 Oct 2007 11:36:25 -0400
Subject: [Numpy-discussion] design patterns for computing
In-Reply-To: <4f4101bf0710170811j7f8c9ad5nc9e3ac8b5d361443@mail.gmail.com>
References: <4f4101bf0710170811j7f8c9ad5nc9e3ac8b5d361443@mail.gmail.com>
Message-ID: <Mahogany-0.67.0-1220-20071017-113625.00@american.edu>

It sounds to me like you have something closer to the 
following.

class Problem
  - initialized with an Input instance and an Analysis instance
  - has a ``get_results`` method that asks the Analysis instance to
        - call ``get_input`` on Input instance
        - run analysis on the provided input
        - create and return and instance of a Results class

Instances of the Input class can then produce input anyway 
you like---e.g., as data containers or as data fetchers---as 
long as they provide it in a standard format
in response to ``get_input``.

fwiw,
Alan Isaac





From david at ar.media.kyoto-u.ac.jp  Thu Oct 18 09:30:20 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 18 Oct 2007 22:30:20 +0900
Subject: [Numpy-discussion] Scons and numpy,
	second milestone: all numpy built with scons.
Message-ID: <47175FEC.1080608@ar.media.kyoto-u.ac.jp>

Hi,

   I am happy to announce a second milestone in scons support in numpy. 
This one is much bigger than the first one, since now I can build the 
whole numpy using scons (e.g. distutils is not used at all to build non 
python code).

How to use it:
==============

just do as before for installing numpy: python setup.py build

To use parrallel builds: python setup scons --jobs=4 build

Windows build does not work yet, but this should be easy to fix. Do not 
use it for practical purpose, though: I have not yet tackled the problem 
of optimization flags, nor the detection of blas/lapack.

What has been done:
===================

Some of the things which are now possible:
   - any module can be customized wrt build options (warning flags, 
etc...). This can be done per object file, also, with a bit some work.
   - for numpy.core (which took 95 % of my time for this milestone), all 
the generated code is integrated in the scons dependency graph. I don't 
know how distutils handled this, but if you change a txt file, or a 
header, scons will rebuilt the necessary things (the dependencies are 
found automatically). I also cleanup up some things (done in a separate 
svn branch).
   - Numpy configuration is also totally integrated into scons config 
framework, which makes the thing more understandable, I hope.

and of course, everything possible with scons is now available. Some 
other nice advantages which I did not think about at first:

   - parallel builds: since scons supports parrallel builds, I thought 
it would be interesting to support it in distutils. This was trivial to 
add, and this largely alleviates the cost added by using scons in the 
first place. For scipy, my guess would be that it could significantly 
reduce the build time (whereas numpy build time is mostly spent 
compiling one or two big files).

So basically, I believe most of the things planned in 
http://projects.scipy.org/scipy/numpy/wiki/DistutilsRevamp are now 
available because they are available in scons, if numpy developers 
decide to follow the scons route. Before being usable, I need to finish 
fortran support and blas/lapack/atlas detection: once this is done, 
numpy should be able to support some platforms not supported yet (intel 
compiler on windows, sun compiler with performance libraries, etc...), 
which was the reason I started this work in the first place.

cheers,

David


From fullung at gmail.com  Thu Oct 18 09:52:48 2007
From: fullung at gmail.com (Albert Strasheim)
Date: Thu, 18 Oct 2007 15:52:48 +0200
Subject: [Numpy-discussion] Scons and numpy,
	second milestone: all numpy built with scons.
In-Reply-To: <47175FEC.1080608@ar.media.kyoto-u.ac.jp>
References: <47175FEC.1080608@ar.media.kyoto-u.ac.jp>
Message-ID: <5eec5f300710180652v31c60107o75aeac328504af10@mail.gmail.com>

Hello

Firstly, great work. I always thought SCons was the way to go for
NumPy and SciPy, and you've pulled it off.

> So basically, I believe most of the things planned in
> http://projects.scipy.org/scipy/numpy/wiki/DistutilsRevamp are now
> available because they are available in scons, if numpy developers
> decide to follow the scons route. Before being usable, I need to finish
> fortran support and blas/lapack/atlas detection: once this is done,
> numpy should be able to support some platforms not supported yet (intel
> compiler on windows, sun compiler with performance libraries, etc...),
> which was the reason I started this work in the first place.

I don't think you should make the autodetection of BLAS and LAPACK too
auto. A .cfg file like mpi4py's mpi.cfg (almost like NumPy's site.cfg,
but simpler) would be great. This way, you can provide a few default
build configurations for common platforms, but still make it easy for
people to configure their build exactly the way they want it, if they
so choose.

Cheers,

Albert


From basilisk96 at gmail.com  Fri Oct  5 00:13:34 2007
From: basilisk96 at gmail.com (Basilisk96)
Date: Fri, 05 Oct 2007 04:13:34 -0000
Subject: [Numpy-discussion] problem converting to matrix from Unicode input
	string
Message-ID: <1191557614.414259.56430@g4g2000hsf.googlegroups.com>

Hello all,

I have the following function, with print statements inserted for
debugging:

import numpy
def file2mat(inFile, sep=None, T=True):
    try:
        input = inFile.readlines()
        print "input=%s" % input
    except:
        raise
    finally:
        inFile.close()
    data = [line.split(sep) for line in input]
    print "data=%s" % data
    if T==True:
        return numpy.mat(data).astype(numpy.float64).T
    else:
        return numpy.mat(data).astype(numpy.float64)

which is then tested as follows:

>>> s = "-0.500 -0.500\n0.500 -0.500\n-0.500 0.500\n0.500 0.500"
>>> u = unicode(s)
>>> from cStringIO import StringIO
>>> file2mat(StringIO(s))
input=['-0.500 -0.500\n', '0.500 -0.500\n', '-0.500 0.500\n', '0.500
0.500']
data=[['-0.500', '-0.500'], ['0.500', '-0.500'], ['-0.500', '0.500'],
['0.500', '0.500']]
matrix([[-0.5,  0.5, -0.5,  0.5],
        [-0.5, -0.5,  0.5,  0.5]])

This is the expected result matrix... But now:

>>> file2mat(StringIO(u))
input=['-\x000\x00.\x005\x000\x000\x00 \x00-
\x000\x00.\x005\x000\x000\x00\n', '\x000\x00.\x005\x000\x000\x00 \x00-
\x000\x00.\x005\x000\x000\x00\n', '\x00-\x000\x00.\x005\x000\x000\x00
\x000\x00.\x005\x000\x000\x00\n', '\x000\x00.\x005\x000\x000\x00
\x000\x00.\x005\x000\x000\x00']
data=[['-\x000\x00.\x005\x000\x000\x00', '\x00-
\x000\x00.\x005\x000\x000\x00'], ['\x000\x00.\x005\x000\x000\x00',
'\x00-\x000\x00.\x005\x000\x000\x00'], ['\x00-
\x000\x00.\x005\x000\x000\x00', '\x000\x00.\x005\x000\x000\x00'],
['\x000\x00.\x005\x000\x000\x00', '\x000\x00.\x005\x000\x000\x00']]
Traceback (most recent call last):
    ...
ValueError: invalid literal for float(): -


When I explicitly cast the input to 'string', I get the expected
result:

>>> file2mat(StringIO(str(u)))
input=['-0.500 -0.500\n', '0.500 -0.500\n', '-0.500 0.500\n', '0.500
0.500']
data=[['-0.500', '-0.500'], ['0.500', '-0.500'], ['-0.500', '0.500'],
['0.500', '0.500']]
matrix([[-0.5,  0.5, -0.5,  0.5],
        [-0.5, -0.5,  0.5,  0.5]])

Any suggestions on how to improve my code?
Is this a Unicode issue, numpy issue, or both?
The input string can come from an ASCII file or a GUI text control. In
the case of a GUI, the control returns a Unicode string, so for now I
am casting it to str(), but it seems like a hack..

BTW, the reason that I am using the astype() method of numpy.matrix is
that I get a "ValueError: setting an array element with a sequence"
when trying to use
    return numpy.mat(data, numpy.float64)
in the above function.

Thank you,
-Basilisk96



From danielhuang at hotmail.com  Tue Oct  9 14:47:58 2007
From: danielhuang at hotmail.com (danielhuang)
Date: Tue, 9 Oct 2007 14:47:58 -0400
Subject: [Numpy-discussion] Install Numpy with Lapack library -- compiling
	error on solaris 9
Message-ID: <BAY122-DS18E706D45E9E6BDA0FCEEB1A50@phx.gbl>

Hi,

I am trying to install numpy with lapack/altas library support and got 
following error on a solaris 9 box.

http://www.scipy.org/Download

The error I got during compiling is as following:


**********
ATL_dset                            0x1494 
/usr/local/lib/atlas/libatlas.a(ATL_zCtrsmKL.o)
ATL_dset                            0x14ac 
/usr/local/lib/atlas/libatlas.a(ATL_zCtrsmKL.o)
ATL_dset                            0xdec 
/usr/local/lib/atlas/libatlas.a(ATL_zCtrsmKR.o)
ATL_dset                            0xe04 
/usr/local/lib/atlas/libatlas.a(ATL_zCtrsmKR.o)
ATL_dset_xp0yp0aXbX                 0xd4 
/usr/local/lib/atlas/libatlas.a(ATL_dset.o)
ld: fatal: relocations remain against allocatable but non-writable 
sectionscollect2: ld returned 1 exit status error: Command "gcc -shared 
build/temp.solaris-2.9-sun4u-2.5/numpy/core/blasdot/_dotblas.o -L/usr/local/lib/atlas 
 -lptf77blas -lptcblas -latlas -o build/lib.sol
aris-2.9-sun4u-2.5/numpy/core/_dotblas.so" failed with exit status 1
***********

I am using gcc 3.4.6, any one has insight what might cause this error ?

thanks a lot.

- Daniel 



From axel.breuer at bnpparibas.com  Thu Oct 11 05:29:29 2007
From: axel.breuer at bnpparibas.com (axel.breuer at bnpparibas.com)
Date: Thu, 11 Oct 2007 11:29:29 +0200
Subject: [Numpy-discussion] Numpy problem with SVD
Message-ID: <OFDA918661.4693E494-ONC1257371.00338A84-C1257371.00341F08@bnpparibas.com>


Hi,

I tried the function numpy.linalg.svd(X) on an array X with 100 rows and 50
columns.

I got an Error message:
** On entry to DGESDD parameter number 12 has an illegal value
Then I got kicked out of python.

I tried numarray.mlab.svd(X) and it worked fine.

Is there a workaround to perform svd with numpy on reasonably sized arrays
?

Thanks in advance for your support

Axel Breuer


This message and any attachments (the "message") is
intended solely for the addressees and is confidential. 
If you receive this message in error, please delete it and 
immediately notify the sender. Any use not in accord with 
its purpose, any dissemination or disclosure, either whole 
or partial, is prohibited except formal approval. The internet
can not guarantee the integrity of this message. 
BNP PARIBAS (and its subsidiaries) shall (will) not 
therefore be liable for the message if modified. 

                ---------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le 
"message") sont etablis a l'intention exclusive de ses 
destinataires et sont confidentiels. Si vous recevez ce 
message par erreur, merci de le detruire et d'en avertir 
immediatement l'expediteur. Toute utilisation de ce 
message non conforme a sa destination, toute diffusion 
ou toute publication, totale ou partielle, est interdite, sauf 
autorisation expresse. L'internet ne permettant pas 
d'assurer l'integrite de ce message, BNP PARIBAS (et ses
filiales) decline(nt) toute responsabilite au titre de ce 
message, dans l'hypothese ou il aurait ete modifie.



From charlesr.harris at gmail.com  Thu Oct 18 11:56:17 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 18 Oct 2007 09:56:17 -0600
Subject: [Numpy-discussion] Numpy problem with SVD
In-Reply-To: <OFDA918661.4693E494-ONC1257371.00338A84-C1257371.00341F08@bnpparibas.com>
References: <OFDA918661.4693E494-ONC1257371.00338A84-C1257371.00341F08@bnpparibas.com>
Message-ID: <e06186140710180856y234bd3b8o352cc1b5436510f5@mail.gmail.com>

Hi Axel,

On 10/11/07, axel.breuer at bnpparibas.com <axel.breuer at bnpparibas.com> wrote:
>
>
> Hi,
>
> I tried the function numpy.linalg.svd(X) on an array X with 100 rows and
> 50
> columns.
>
> I got an Error message:
> ** On entry to DGESDD parameter number 12 has an illegal value
> Then I got kicked out of python.
>
> I tried numarray.mlab.svd(X) and it worked fine.
>
> Is there a workaround to perform svd with numpy on reasonably sized arrays
> ?


Works here. This sounds like an old bug. What version of numpy are you
using? Also, information on the OS, hardware (32/64 bit), and whether you
are using ATLAS will help.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071018/92327cec/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Thu Oct 18 18:38:46 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 19 Oct 2007 07:38:46 +0900
Subject: [Numpy-discussion] Scons and numpy,
 second milestone: all numpy built with scons.
In-Reply-To: <5eec5f300710180652v31c60107o75aeac328504af10@mail.gmail.com>
References: <47175FEC.1080608@ar.media.kyoto-u.ac.jp>
	<5eec5f300710180652v31c60107o75aeac328504af10@mail.gmail.com>
Message-ID: <4717E076.4030508@ar.media.kyoto-u.ac.jp>

Albert Strasheim wrote:
> Hello
>
> Firstly, great work. I always thought SCons was the way to go for
> NumPy and SciPy, and you've pulled it off.
>
>> So basically, I believe most of the things planned in
>> http://projects.scipy.org/scipy/numpy/wiki/DistutilsRevamp are now
>> available because they are available in scons, if numpy developers
>> decide to follow the scons route. Before being usable, I need to finish
>> fortran support and blas/lapack/atlas detection: once this is done,
>> numpy should be able to support some platforms not supported yet (intel
>> compiler on windows, sun compiler with performance libraries, etc...),
>> which was the reason I started this work in the first place.
>
> I don't think you should make the autodetection of BLAS and LAPACK too
> auto. A .cfg file like mpi4py's mpi.cfg (almost like NumPy's site.cfg,
> but simpler) would be great. This way, you can provide a few default
> build configurations for common platforms, but still make it easy for
> people to configure their build exactly the way they want it, if they
> so choose.
>
I agree it should be overridable (I have implemented a NumpyCheckLib, 
which is just a CheckLib overridable with a site.cfg; I still have to 
implement the others checkers). But I think it should work out of the 
box, that is if you have a usable blas and lapack in standard 
directories, then you should not have to do anything; for this, I really 
think setting default configuration is NOT the way to go. The autoconf 
way is much better IMHO, because it would be able to use other compilers 
without trouble. It would also detect configuration problems before 
compilation in most cases,

cheers,

David


From zyzhu2000 at gmail.com  Thu Oct 18 19:00:56 2007
From: zyzhu2000 at gmail.com (Geoffrey Zhu)
Date: Thu, 18 Oct 2007 18:00:56 -0500
Subject: [Numpy-discussion] Sum of the product of three or more arrays
Message-ID: <fcedc2970710181600t2210c1e4qd3038c858612711a@mail.gmail.com>

Hi All,

Given three vectors of the same lengths, X, Y, and Z, I am looking for
an efficient way to calculate the following:

sum(x[i]*y[i]*z[i], for i=1..n )


I am not sure if there is a vectorized way to do this.

Thanks,
Geoffrey


From zyzhu2000 at gmail.com  Thu Oct 18 19:07:46 2007
From: zyzhu2000 at gmail.com (Geoffrey Zhu)
Date: Thu, 18 Oct 2007 18:07:46 -0500
Subject: [Numpy-discussion] Sum of the product of three or more arrays
In-Reply-To: <fcedc2970710181600t2210c1e4qd3038c858612711a@mail.gmail.com>
References: <fcedc2970710181600t2210c1e4qd3038c858612711a@mail.gmail.com>
Message-ID: <fcedc2970710181607l284b07c6w78b3d9a01082756b@mail.gmail.com>

I think I figured out: sum(X*Y*Z).

Never mind.

On 10/18/07, Geoffrey Zhu <zyzhu2000 at gmail.com> wrote:
> Hi All,
>
> Given three vectors of the same lengths, X, Y, and Z, I am looking for
> an efficient way to calculate the following:
>
> sum(x[i]*y[i]*z[i], for i=1..n )
>
>
> I am not sure if there is a vectorized way to do this.
>
> Thanks,
> Geoffrey
>


From robert.kern at gmail.com  Thu Oct 18 19:08:06 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 18 Oct 2007 18:08:06 -0500
Subject: [Numpy-discussion] Sum of the product of three or more arrays
In-Reply-To: <fcedc2970710181600t2210c1e4qd3038c858612711a@mail.gmail.com>
References: <fcedc2970710181600t2210c1e4qd3038c858612711a@mail.gmail.com>
Message-ID: <4717E756.3030904@gmail.com>

Geoffrey Zhu wrote:
> Hi All,
> 
> Given three vectors of the same lengths, X, Y, and Z, I am looking for
> an efficient way to calculate the following:
> 
> sum(x[i]*y[i]*z[i], for i=1..n )

(x*y*z).sum()

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From millman at berkeley.edu  Thu Oct 18 23:35:59 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Thu, 18 Oct 2007 20:35:59 -0700
Subject: [Numpy-discussion] NumPy 1.0.4 release
Message-ID: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>

Hi all,

After speaking with Travis, I think that we can release NumPy 1.0.4 by
the end of the month.  1.0.3 came out almost 5 months ago and there
have been a number of bug-fixes and other improvements since then.

Please take a look at the 1.0.4 roadmap:
http://scipy.org/scipy/numpy/milestone/1.0.4
There are currently 23 active tickets that could use some attention.
Hopefully, we can close as many of these as possible before the
release (some need fixes, some can just be closed if appropriate).  If
you see something that must be addressed before the release, mark it
with a severity level of 'blocker'.

Here is the proposed release schedule:

Monday (10/22)
  * Development Freeze.  After this point, only bug fixes should be checked in.

Sunday (10/28)
  * Release Tagged.  At this point, I will be asking for help from
packagers and testers.

Thursday (11/1)
   * Release Announced.

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From david at ar.media.kyoto-u.ac.jp  Fri Oct 19 00:07:12 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 19 Oct 2007 13:07:12 +0900
Subject: [Numpy-discussion] NumPy 1.0.4 release
In-Reply-To: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>
References: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>
Message-ID: <47182D70.7060304@ar.media.kyoto-u.ac.jp>

Jarrod Millman wrote:
> Hi all,
>
> After speaking with Travis, I think that we can release NumPy 1.0.4 by
> the end of the month.  1.0.3 came out almost 5 months ago and there
> have been a number of bug-fixes and other improvements since then.
>
> Please take a look at the 1.0.4 roadmap:
> http://scipy.org/scipy/numpy/milestone/1.0.4
> There are currently 23 active tickets that could use some attention.
> Hopefully, we can close as many of these as possible before the
> release (some need fixes, some can just be closed if appropriate).  If
> you see something that must be addressed before the release, mark it
> with a severity level of 'blocker'.
>
> Here is the proposed release schedule:
>
> Monday (10/22)
>   * Development Freeze.  After this point, only bug fixes should be checked in.
>
> Sunday (10/28)
>   * Release Tagged.  At this point, I will be asking for help from
> packagers and testers.
>
> Thursday (11/1)
>    * Release Announced.
>
> Thanks,
>
>   
Hi Jarrod,

    Would it be possible to merge some of the work I have done recently 
concerning cleaning configuration and so on (If nobody is against it, of 
course) ? If this is considerer too big of a change, what is the plan 
for a 1.1 release, if any ?

cheers,

David


From oliphant at enthought.com  Fri Oct 19 00:50:39 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Thu, 18 Oct 2007 23:50:39 -0500
Subject: [Numpy-discussion] NumPy 1.0.4 release
In-Reply-To: <47182D70.7060304@ar.media.kyoto-u.ac.jp>
References: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>
	<47182D70.7060304@ar.media.kyoto-u.ac.jp>
Message-ID: <4718379F.806@enthought.com>

David Cournapeau wrote:
>
>>
>>   
>>     
> Hi Jarrod,
>
>     Would it be possible to merge some of the work I have done recently 
> concerning cleaning configuration and so on (If nobody is against it, of 
> course) ? If this is considerer too big of a change, what is the plan 
> for a 1.1 release, if any ?
>   
Could you review what the disadvantages are for including your changes 
into the trunk? 

Would there be any code breakage?    What is the approximate size 
increase of the resulting NumPy?

I've tried to follow the discussion, but haven't kept up with all of it. 

Thanks,

-Travis



From david at ar.media.kyoto-u.ac.jp  Fri Oct 19 01:09:36 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 19 Oct 2007 14:09:36 +0900
Subject: [Numpy-discussion] NumPy 1.0.4 release
In-Reply-To: <4718379F.806@enthought.com>
References: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>	<47182D70.7060304@ar.media.kyoto-u.ac.jp>
	<4718379F.806@enthought.com>
Message-ID: <47183C10.5050100@ar.media.kyoto-u.ac.jp>

Travis E. Oliphant wrote:
> David Cournapeau wrote:
>>>   
>>>     
>> Hi Jarrod,
>>
>>     Would it be possible to merge some of the work I have done recently 
>> concerning cleaning configuration and so on (If nobody is against it, of 
>> course) ? If this is considerer too big of a change, what is the plan 
>> for a 1.1 release, if any ?
>>   
> Could you review what the disadvantages are for including your changes 
> into the trunk? 
>
> Would there be any code breakage?    What is the approximate size 
> increase of the resulting NumPy?
>
> I've tried to follow the discussion, but haven't kept up with all of it. 
There are been several things. I tried to keep different things 
separated for separate merge, and to make the review easier.

cleanconfig branch
==================

For the cleanconfig branch, I tried to be detailed enough in the ticket 
where I posted the patch: the goal is to separate the config.h into two 
separate headers, one public, one private. The point is to avoid 
polluting the C namespace when using numpy (the current config.h define 
HAVE_* and SIZEOF_* symbols which are used by autotools). This also 
makes it more similar to autoheader, so that if/when my scons work is 
integrated, replacing the custom config generators by scons will be 
trivial. The patch being actually generated from the branch, it does not 
really make sense to apply the patch, just merge the branch (I have also 
updated the code since I posted the patch).

http://scipy.org/scipy/numpy/ticket/593

This is purely internal, this should not change anything to anyone on 
any platform, otherwise, it is a bug.

numpy.scons branch
==================

This is a much more massive change. Scons itself adds something like 350 
kb to a bzip tarball. There are two milestones already in this branch.
    - The first one just adds some facilities to numpy.distutils, and it 
should not break nor change anything: this would enable people to build 
ctypes-based extension in a portable way, though.
    - The second milestone: I don't think it would be appropriate to 
merge this at this point for a 1.0.x release. Normally, nothing is used 
by default (e.g. you have to use a different setup.py to build with the 
new method), but it is so easy to screw things up that with such a short 
window (a few days), I would prefer avoiding potential breakage.

The first milestone, if merged, consist in all revision up to 4177 (this 
one has been tested on many platforms, including Mac OS X intel, Windows 
with gnu compilers, linux with gnu compilers, and more obscure 
configurations as well; a breakage should be trivial to fix anyway).

cheers,

David


From oliphant at enthought.com  Fri Oct 19 01:26:34 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 19 Oct 2007 00:26:34 -0500
Subject: [Numpy-discussion] NumPy 1.0.4 release
In-Reply-To: <47182D70.7060304@ar.media.kyoto-u.ac.jp>
References: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>
	<47182D70.7060304@ar.media.kyoto-u.ac.jp>
Message-ID: <4718400A.4060405@enthought.com>


I've finally caught up with the discussion on aligned allocators for 
NumPy.   In general I'm favorable to the idea, although it is not as 
easy to implement in 1.0.X because of the need to possibly change the C-API.

The Python solution is workable and would just require a function call 
on the Python side (which one could call from the C-side as well with 
little difficulty, I believe Chuck Harris already suggested such a 
function).   So, I think at least the Python functions are an easy 
addition for 1.0.4 (along with simple tests for alignment --- although 
a.ctypes.data % 16 is pretty simple and probably doesn't warrant a new 
function)

I'm a bit more resistant to the more involved C-code in the patch 
provided with #568, because of the requested new additions to the C-API, 
but I understand the need.   I'm currently also thinking heavily about 
using SIMD intrinsics in ufunc inner loops but will not likely get those 
in before 1.0.4.   Unfortunately, all ufuncs that take advantage of SIMD 
instructions will have to handle the unaligned portions which may occur 
even if the start of the array is aligned, so the problem of thinking 
about alignment does not go away there with a simplified function call.

A simple addition is an NPY_ALIGNED_16 and NPY_ALIGNED_32 flag for the 
PyArray_From_Any that could adjust the data-pointer as needed to get at 
least those kinds of alignment.

We can't change the C-API for PyArray_FromAny to accept an alignment 
flag, and I'm pretty loath to do that even for 1.1.

Is there a consensus?  What do others think of the patch in ticket 
#568?  Is there a need to add general-purpose aligned memory allocators 
to NumPy without a corresponding array_allocator? 

I would think the PyArray_FromAny and PyArray_NewFromDescr with aligned 
memory is more important which I think we could do with flag bits.

-Travis O.










From oliphant at enthought.com  Fri Oct 19 01:30:49 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 19 Oct 2007 00:30:49 -0500
Subject: [Numpy-discussion] NumPy 1.0.4 release
In-Reply-To: <4718400A.4060405@enthought.com>
References: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>	<47182D70.7060304@ar.media.kyoto-u.ac.jp>
	<4718400A.4060405@enthought.com>
Message-ID: <47184109.2010808@enthought.com>

Travis E. Oliphant wrote:
> We can't change the C-API for PyArray_FromAny to accept an alignment 
> flag, and I'm pretty loath to do that even for 1.1.
>   
Ooops!  Pleas read that as can't accept an alignment *argument*

-Travis




From david at ar.media.kyoto-u.ac.jp  Fri Oct 19 01:33:09 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 19 Oct 2007 14:33:09 +0900
Subject: [Numpy-discussion] NumPy 1.0.4 release
In-Reply-To: <4718400A.4060405@enthought.com>
References: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>	<47182D70.7060304@ar.media.kyoto-u.ac.jp>
	<4718400A.4060405@enthought.com>
Message-ID: <47184195.6040801@ar.media.kyoto-u.ac.jp>

Travis E. Oliphant wrote:
> I've finally caught up with the discussion on aligned allocators for 
> NumPy.   In general I'm favorable to the idea, although it is not as 
> easy to implement in 1.0.X because of the need to possibly change the C-API.
>
> The Python solution is workable and would just require a function call 
> on the Python side (which one could call from the C-side as well with 
> little difficulty, I believe Chuck Harris already suggested such a 
> function).   So, I think at least the Python functions are an easy 
> addition for 1.0.4 (along with simple tests for alignment --- although 
> a.ctypes.data % 16 is pretty simple and probably doesn't warrant a new 
> function)
>
> I'm a bit more resistant to the more involved C-code in the patch 
> provided with #568, because of the requested new additions to the C-API, 
> but I understand the need.   I'm currently also thinking heavily about 
> using SIMD intrinsics in ufunc inner loops but will not likely get those 
> in before 1.0.4.   Unfortunately, all ufuncs that take advantage of SIMD 
> instructions will have to handle the unaligned portions which may occur 
> even if the start of the array is aligned, so the problem of thinking 
> about alignment does not go away there with a simplified function call.
>   
I don't know anything about the ufunc machinery yet, but I guess you 
need to know the alignement of a given buffer, right ? This can be done 
easily.

Actually, one problem I encountered (If I remember correctly) was that 
there is no pure C library facility in numpy: by that, I mean a simple C 
library, independant of python, which could be reusable by C code using 
numpy. For example, if we want to start thinking about using SIMD, I 
think it would be good to support basics in a pure C library.

I don't see any downside to this approach ?
> A simple addition is an NPY_ALIGNED_16 and NPY_ALIGNED_32 flag for the 
> PyArray_From_Any that could adjust the data-pointer as needed to get at 
> least those kinds of alignment.
>
> We can't change the C-API for PyArray_FromAny to accept an alignment 
> flag, and I'm pretty loath to do that even for 1.1.
>
> Is there a consensus?  What do others think of the patch in ticket 
> #568?  Is there a need to add general-purpose aligned memory allocators 
> to NumPy without a corresponding array_allocator? 
>   
Having the NPY_ALIGNED_* flags would already be enough for most cases 
(for SIMD, you rarely, if ever needs more than 32 bytes alignment 
AFAIK). Those flags + general purposes memory allocators (in a C support 
library) would be enough to do everything we need to fft, for example.

cheers,

David


From matthieu.brucher at gmail.com  Fri Oct 19 03:53:08 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 19 Oct 2007 09:53:08 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <4713C111.5010609@enthought.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
	<4713C111.5010609@enthought.com>
Message-ID: <e76aa17f0710190053v272ae632y8dec40eec05980a3@mail.gmail.com>

Here is an excerpt of the stack on the numpy svn of wednesday :

#0  0x40000402 in __kernel_vsyscall ()
#1  0x00b8e382 in sem_post at GLIBC_2.0 () from /lib/libpthread.so.0
#2  0x080fe5b7 in PyThread_release_lock (lock=0x80d0fb2) at
Python/thread_pthread.h:374
#3  0x080d0fb2 in PyEval_SaveThread () at Python/ceval.c:299
#4  0x4064ec7a in dotblas_innerproduct (dummy=0x0, args=0x960522c) at
numpy/core/blasdot/_dotblas.c:835
#5  0x0811ecf1 in PyCFunction_Call (func=0x960522c, arg=0x80cfec7,
kw=0x942068c) at Objects/methodobject.c:73
#6  0x080cfec7 in call_function (pp_stack=0x0, oparg=1) at
Python/ceval.c:3564
#7  0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
#8  0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
#9  0x0811de5b in function_call (func=0x0, arg=0x960516c, kw=0x965b148) at
Objects/funcobject.c:517
#10 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960516c, kw=0x8061a66) at
Objects/abstract.c:1860
#11 0x08061a66 in instancemethod_call (func=0x95f16ac, arg=0x94f5824,
kw=0x805bdb6) at Objects/classobject.c:2497
#12 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac, kw=0x80a86e9) at
Objects/abstract.c:1860
#13 0x080a86e9 in slot_tp_call (self=0xbf975c78, args=0x0, kwds=0x805bdb6)
at Objects/typeobject.c:4633
#14 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac, kw=0x80d017b) at
Objects/abstract.c:1860
#15 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=154955456) at
Python/ceval.c:3775
#16 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
Python/ceval.c:3587
#17 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
#18 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
#19 0x0811de5b in function_call (func=0x0, arg=0x960520c, kw=0x30) at
Objects/funcobject.c:517
#20 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960520c, kw=0x8061a66) at
Objects/abstract.c:1860
#21 0x08061a66 in instancemethod_call (func=0x95ecbac, arg=0x94f57d4,
kw=0x805bdb6) at Objects/classobject.c:2497
#22 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac, kw=0x80a86e9) at
Objects/abstract.c:1860
#23 0x080a86e9 in slot_tp_call (self=0xbf97609c, args=0x0, kwds=0x805bdb6)
at Objects/typeobject.c:4633
#24 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac, kw=0x80d017b) at
Objects/abstract.c:1860
#25 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=156957888) at
Python/ceval.c:3775
#26 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
Python/ceval.c:3587
#27 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
#28 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
#29 0x0811de5b in function_call (func=0x0, arg=0x960518c, kw=0x9548a1) at
Objects/funcobject.c:517
#30 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960518c, kw=0x8061a66) at
Objects/abstract.c:1860
#31 0x08061a66 in instancemethod_call (func=0x960526c, arg=0x95fc874,
kw=0x805bdb6) at Objects/classobject.c:2497
#32 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c, kw=0x80a86e9) at
Objects/abstract.c:1860
#33 0x080a86e9 in slot_tp_call (self=0xbf9764c0, args=0x0, kwds=0x805bdb6)
at Objects/typeobject.c:4633
#34 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c, kw=0x80d017b) at
Objects/abstract.c:1860
#35 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=157640136) at
Python/ceval.c:3775
#36 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
Python/ceval.c:3587
#37 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
#38 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
#39 0x0811de5b in function_call (func=0x0, arg=0x95f260c, kw=0x0) at
Objects/funcobject.c:517
#40 0x0805bdb6 in PyObject_Call (func=0x9609934, arg=0x95f260c,
kw=0x8061a66) at Objects/abstract.c:1860

Seems that the bug could be somewhere in the handling of the dot blas module
?

Matthieu

2007/10/15, Travis E. Oliphant <oliphant at enthought.com>:
>
> Matthieu Brucher wrote:
> >
> >     The problem is that there is a data-type reference counting error
> some
> >     where that is attempting to deallocate the built-in data-type 'l'
> >
> >
> >
> > That's what I supposed, but I couldn't find the reason why it wanted
> > to do this
> >
> >
> >     It's not really a Python error but a logging.  The code won't let
> you
> >     deallocate the built-ins, but it will tell you that something
> >     tried to.
> >
> >     Reference counting on data-types is easy to get wrong
> >     (particularly with
> >     Pyrex extension modules) because most calls consume a reference to
> the
> >     data-type (if they return an object that  contains a reference to
> the
> >     data-type).
> >
> >     It is a bug, and it would be nice to figure it out, but that would
> >     require the code that caused it.
> >
> >
> > I've updated my numpy version to the latest svn, the behaviour seems
> > to be different (more warnings), I'll try to give more information
> > about the error, but giving the whole code will not be simple (it uses
> > a big data file that seems to trigger the error as with other data
> > files, the error didn't show up :()
> >
>
> There are two types of errors that can occur with reference counting on
> data-types.
>
> 1) There are too many DECREF's --- this gets us to the error quickly and
> is usually easy to reproduce
> 2) There are too many INCREF's (the reference count keeps going up until
> the internal counter wraps around to 0 and deallocation is attempted)
> --- this error is harder to reproduce and usually takes a while before
> it happens in the code.
>
>
> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071019/56a9099b/attachment.html>

From millman at berkeley.edu  Fri Oct 19 04:49:59 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Fri, 19 Oct 2007 01:49:59 -0700
Subject: [Numpy-discussion] Support universal gfortran compiler on OS X
Message-ID: <c7009a550710190149k2a098aa1t5b194176ec531b0b@mail.gmail.com>

Hello,

Could someone confirm whether this ticket can be closed:
http://projects.scipy.org/scipy/numpy/ticket/571

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From matthieu.brucher at gmail.com  Fri Oct 19 06:27:44 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 19 Oct 2007 12:27:44 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <4713C111.5010609@enthought.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
	<4713C111.5010609@enthought.com>
Message-ID: <e76aa17f0710190327p18a55596q38fada88d23dddb@mail.gmail.com>

Here is an excerpt of the stack on the numpy svn of wednesday :

#0  0x40000402 in __kernel_vsyscall ()
#1  0x00b8e382 in sem_post at GLIBC_2.0 () from /lib/libpthread.so.0
#2  0x080fe5b7 in PyThread_release_lock (lock=0x80d0fb2) at
Python/thread_pthread.h:374
#3  0x080d0fb2 in PyEval_SaveThread () at Python/ceval.c:299
#4  0x4064ec7a in dotblas_innerproduct (dummy=0x0, args=0x960522c) at
numpy/core/blasdot/_dotblas.c:835
#5  0x0811ecf1 in PyCFunction_Call (func=0x960522c, arg=0x80cfec7,
kw=0x942068c) at Objects/methodobject.c:73
#6  0x080cfec7 in call_function (pp_stack=0x0, oparg=1) at
Python/ceval.c:3564
#7  0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
#8  0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
#9  0x0811de5b in function_call (func=0x0, arg=0x960516c, kw=0x965b148) at
Objects/funcobject.c:517
#10 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960516c, kw=0x8061a66) at
Objects/abstract.c:1860
#11 0x08061a66 in instancemethod_call (func=0x95f16ac, arg=0x94f5824,
kw=0x805bdb6) at Objects/classobject.c:2497
#12 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac, kw=0x80a86e9) at
Objects/abstract.c:1860
#13 0x080a86e9 in slot_tp_call (self=0xbf975c78, args=0x0, kwds=0x805bdb6)
at Objects/typeobject.c:4633
#14 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac, kw=0x80d017b) at
Objects/abstract.c:1860
#15 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=154955456) at
Python/ceval.c:3775
#16 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
Python/ceval.c:3587
#17 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
#18 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
#19 0x0811de5b in function_call (func=0x0, arg=0x960520c, kw=0x30) at
Objects/funcobject.c:517
#20 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960520c, kw=0x8061a66) at
Objects/abstract.c:1860
#21 0x08061a66 in instancemethod_call (func=0x95ecbac, arg=0x94f57d4,
kw=0x805bdb6) at Objects/classobject.c:2497
#22 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac, kw=0x80a86e9) at
Objects/abstract.c:1860
#23 0x080a86e9 in slot_tp_call (self=0xbf97609c, args=0x0, kwds=0x805bdb6)
at Objects/typeobject.c:4633
#24 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac, kw=0x80d017b) at
Objects/abstract.c:1860
#25 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=156957888) at
Python/ceval.c:3775
#26 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
Python/ceval.c:3587
#27 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
#28 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
#29 0x0811de5b in function_call (func=0x0, arg=0x960518c, kw=0x9548a1) at
Objects/funcobject.c:517
#30 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960518c, kw=0x8061a66) at
Objects/abstract.c:1860
#31 0x08061a66 in instancemethod_call (func=0x960526c, arg=0x95fc874,
kw=0x805bdb6) at Objects/classobject.c:2497
#32 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c, kw=0x80a86e9) at
Objects/abstract.c:1860
#33 0x080a86e9 in slot_tp_call (self=0xbf9764c0, args=0x0, kwds=0x805bdb6)
at Objects/typeobject.c:4633
#34 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c, kw=0x80d017b) at
Objects/abstract.c:1860
#35 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=157640136) at
Python/ceval.c:3775
#36 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
Python/ceval.c:3587
#37 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
#38 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
#39 0x0811de5b in function_call (func=0x0, arg=0x95f260c, kw=0x0) at
Objects/funcobject.c:517
#40 0x0805bdb6 in PyObject_Call (func=0x9609934, arg=0x95f260c,
kw=0x8061a66) at Objects/abstract.c:1860

Seems that

2007/10/15, Travis E. Oliphant <oliphant at enthought.com>:
>
> Matthieu Brucher wrote:
> >
> >     The problem is that there is a data-type reference counting error
> some
> >     where that is attempting to deallocate the built-in data-type 'l'
> >
> >
> >
> > That's what I supposed, but I couldn't find the reason why it wanted
> > to do this
> >
> >
> >     It's not really a Python error but a logging.  The code won't let
> you
> >     deallocate the built-ins, but it will tell you that something
> >     tried to.
> >
> >     Reference counting on data-types is easy to get wrong
> >     (particularly with
> >     Pyrex extension modules) because most calls consume a reference to
> the
> >     data-type (if they return an object that  contains a reference to
> the
> >     data-type).
> >
> >     It is a bug, and it would be nice to figure it out, but that would
> >     require the code that caused it.
> >
> >
> > I've updated my numpy version to the latest svn, the behaviour seems
> > to be different (more warnings), I'll try to give more information
> > about the error, but giving the whole code will not be simple (it uses
> > a big data file that seems to trigger the error as with other data
> > files, the error didn't show up :()
> >
>
> There are two types of errors that can occur with reference counting on
> data-types.
>
> 1) There are too many DECREF's --- this gets us to the error quickly and
> is usually easy to reproduce
> 2) There are too many INCREF's (the reference count keeps going up until
> the internal counter wraps around to 0 and deallocation is attempted)
> --- this error is harder to reproduce and usually takes a while before
> it happens in the code.
>
>
> -Travis
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071019/2129d3d1/attachment.html>

From matthieu.brucher at gmail.com  Fri Oct 19 06:28:18 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 19 Oct 2007 12:28:18 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710190327p18a55596q38fada88d23dddb@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
	<4713C111.5010609@enthought.com>
	<e76aa17f0710190327p18a55596q38fada88d23dddb@mail.gmail.com>
Message-ID: <e76aa17f0710190328u7b2b950foebd04943fda15bc5@mail.gmail.com>

Oups, sorry for the sending twice, we have some problem with our DNS in
Strasbourg :(

2007/10/19, Matthieu Brucher <matthieu.brucher at gmail.com>:
>
> Here is an excerpt of the stack on the numpy svn of wednesday :
>
> #0  0x40000402 in __kernel_vsyscall ()
> #1  0x00b8e382 in sem_post at GLIBC_2.0 () from /lib/libpthread.so.0
> #2  0x080fe5b7 in PyThread_release_lock (lock=0x80d0fb2) at
> Python/thread_pthread.h:374
> #3  0x080d0fb2 in PyEval_SaveThread () at Python/ceval.c:299
> #4  0x4064ec7a in dotblas_innerproduct (dummy=0x0, args=0x960522c) at
> numpy/core/blasdot/_dotblas.c:835
> #5  0x0811ecf1 in PyCFunction_Call (func=0x960522c, arg=0x80cfec7,
> kw=0x942068c) at Objects/methodobject.c:73
> #6  0x080cfec7 in call_function (pp_stack=0x0, oparg=1) at
> Python/ceval.c:3564
> #7  0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> #8  0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> #9  0x0811de5b in function_call (func=0x0, arg=0x960516c, kw=0x965b148) at
> Objects/funcobject.c:517
> #10 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960516c, kw=0x8061a66) at
> Objects/abstract.c:1860
> #11 0x08061a66 in instancemethod_call (func=0x95f16ac, arg=0x94f5824,
> kw=0x805bdb6) at Objects/classobject.c:2497
> #12 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac, kw=0x80a86e9) at
> Objects/abstract.c:1860
> #13 0x080a86e9 in slot_tp_call (self=0xbf975c78, args=0x0, kwds=0x805bdb6)
> at Objects/typeobject.c:4633
> #14 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac, kw=0x80d017b) at
> Objects/abstract.c:1860
> #15 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=154955456) at
> Python/ceval.c:3775
> #16 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> Python/ceval.c:3587
> #17 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> #18 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> #19 0x0811de5b in function_call (func=0x0, arg=0x960520c, kw=0x30) at
> Objects/funcobject.c:517
> #20 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960520c, kw=0x8061a66) at
> Objects/abstract.c:1860
> #21 0x08061a66 in instancemethod_call (func=0x95ecbac, arg=0x94f57d4,
> kw=0x805bdb6) at Objects/classobject.c:2497
> #22 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac, kw=0x80a86e9) at
> Objects/abstract.c:1860
> #23 0x080a86e9 in slot_tp_call (self=0xbf97609c, args=0x0, kwds=0x805bdb6)
> at Objects/typeobject.c:4633
> #24 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac, kw=0x80d017b) at
> Objects/abstract.c:1860
> #25 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=156957888) at
> Python/ceval.c:3775
> #26 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> Python/ceval.c:3587
> #27 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> #28 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> #29 0x0811de5b in function_call (func=0x0, arg=0x960518c, kw=0x9548a1) at
> Objects/funcobject.c:517
> #30 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960518c, kw=0x8061a66) at
> Objects/abstract.c:1860
> #31 0x08061a66 in instancemethod_call (func=0x960526c, arg=0x95fc874,
> kw=0x805bdb6) at Objects/classobject.c:2497
> #32 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c, kw=0x80a86e9) at
> Objects/abstract.c:1860
> #33 0x080a86e9 in slot_tp_call (self=0xbf9764c0, args=0x0, kwds=0x805bdb6)
> at Objects/typeobject.c:4633
> #34 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c, kw=0x80d017b) at
> Objects/abstract.c:1860
> #35 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=157640136) at
> Python/ceval.c:3775
> #36 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> Python/ceval.c:3587
> #37 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> #38 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> #39 0x0811de5b in function_call (func=0x0, arg=0x95f260c, kw=0x0) at
> Objects/funcobject.c:517
> #40 0x0805bdb6 in PyObject_Call (func=0x9609934, arg=0x95f260c,
> kw=0x8061a66) at Objects/abstract.c:1860
>
> Seems that
>
> 2007/10/15, Travis E. Oliphant <oliphant at enthought.com >:
> >
> > Matthieu Brucher wrote:
> > >
> > >     The problem is that there is a data-type reference counting error
> > some
> > >     where that is attempting to deallocate the built-in data-type 'l'
> > >
> > >
> > >
> > > That's what I supposed, but I couldn't find the reason why it wanted
> > > to do this
> > >
> > >
> > >     It's not really a Python error but a logging.  The code won't let
> > you
> > >     deallocate the built-ins, but it will tell you that something
> > >     tried to.
> > >
> > >     Reference counting on data-types is easy to get wrong
> > >     (particularly with
> > >     Pyrex extension modules) because most calls consume a reference to
> > the
> > >     data-type (if they return an object that  contains a reference to
> > the
> > >     data-type).
> > >
> > >     It is a bug, and it would be nice to figure it out, but that would
> > >     require the code that caused it.
> > >
> > >
> > > I've updated my numpy version to the latest svn, the behaviour seems
> > > to be different (more warnings), I'll try to give more information
> > > about the error, but giving the whole code will not be simple (it uses
> > > a big data file that seems to trigger the error as with other data
> > > files, the error didn't show up :()
> > >
> >
> > There are two types of errors that can occur with reference counting on
> > data-types.
> >
> > 1) There are too many DECREF's --- this gets us to the error quickly and
> >
> > is usually easy to reproduce
> > 2) There are too many INCREF's (the reference count keeps going up until
> > the internal counter wraps around to 0 and deallocation is attempted)
> > --- this error is harder to reproduce and usually takes a while before
> > it happens in the code.
> >
> >
> > -Travis
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071019/b2ec892a/attachment.html>

From dalcinl at gmail.com  Fri Oct 19 10:59:18 2007
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 19 Oct 2007 11:59:18 -0300
Subject: [Numpy-discussion] NumPy 1.0.4 release
In-Reply-To: <47183C10.5050100@ar.media.kyoto-u.ac.jp>
References: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>
	<47182D70.7060304@ar.media.kyoto-u.ac.jp> <4718379F.806@enthought.com>
	<47183C10.5050100@ar.media.kyoto-u.ac.jp>
Message-ID: <e7ba66e40710190759s2127a7efq763deb6bafdece66@mail.gmail.com>

On 10/19/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> numpy.scons branch
>
> This is a much more massive change. Scons itself adds something like 350
> kb to a bzip tarball.

If numpy build system will not depend on scons (is this right?) then
.. Is it strictly needed to distribute scons with numpy?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594


From zyzhu2000 at gmail.com  Fri Oct 19 15:48:41 2007
From: zyzhu2000 at gmail.com (Geoffrey Zhu)
Date: Fri, 19 Oct 2007 14:48:41 -0500
Subject: [Numpy-discussion] Sum of the product of three or more arrays
In-Reply-To: <4717E756.3030904@gmail.com>
References: <fcedc2970710181600t2210c1e4qd3038c858612711a@mail.gmail.com>
	<4717E756.3030904@gmail.com>
Message-ID: <fcedc2970710191248p22f3aa20gee097fb41404d999@mail.gmail.com>

On 10/18/07, Robert Kern <robert.kern at gmail.com> wrote:
> Geoffrey Zhu wrote:
> > Hi All,
> >
> > Given three vectors of the same lengths, X, Y, and Z, I am looking for
> > an efficient way to calculate the following:
> >
> > sum(x[i]*y[i]*z[i], for i=1..n )
>
> (x*y*z).sum()
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless enigma
>  that is made terrible by our own mad attempt to interpret it as though it had
>  an underlying truth."
>  -- Umberto Eco
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>

Thanks Robert.

(x*y*z).sum() is probably faster than sum(x*y*z).


From zyzhu2000 at gmail.com  Fri Oct 19 23:52:12 2007
From: zyzhu2000 at gmail.com (Geoffrey Zhu)
Date: Fri, 19 Oct 2007 22:52:12 -0500
Subject: [Numpy-discussion] Bug in piecewise?
Message-ID: <fcedc2970710192052o52557545hc83a73a7b81907ea@mail.gmail.com>

Hi All,

If I execute the following code, I find that function f() sometimes is
called with an empty array. I am not sure why this is necessary. Is
this a bug?

def f(x):
   return x**2
return numpy.piecewise(u, abs(u)<1, [f, 0])

Thanks,
Geoffrey


From david at ar.media.kyoto-u.ac.jp  Sat Oct 20 05:16:10 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sat, 20 Oct 2007 18:16:10 +0900
Subject: [Numpy-discussion] NumPy 1.0.4 release
In-Reply-To: <e7ba66e40710190759s2127a7efq763deb6bafdece66@mail.gmail.com>
References: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>	<47182D70.7060304@ar.media.kyoto-u.ac.jp>
	<4718379F.806@enthought.com>	<47183C10.5050100@ar.media.kyoto-u.ac.jp>
	<e7ba66e40710190759s2127a7efq763deb6bafdece66@mail.gmail.com>
Message-ID: <4719C75A.8030705@ar.media.kyoto-u.ac.jp>

Lisandro Dalcin wrote:
> On 10/19/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>> numpy.scons branch
>>
>> This is a much more massive change. Scons itself adds something like 350
>> kb to a bzip tarball.
>
> If numpy build system will not depend on scons (is this right?) then
> .. Is it strictly needed to distribute scons with numpy?
Well, the choice is between requiring another dependency, with all the 
version problems, and including it in the sources; also, although numpy 
would not depend on scons in any way with the changes I suggested for 
1.0.4, it would have helped detecting problems on some platforms. And I 
hope that in the end, scons will be used for numpy (for 1.1 ?), once I 
finish the conversion.

I don't see situations where adding 350 Kb in the tarball can be an 
issue, so could you tell me what the problem would be ?

cheers,

David



From dalcinl at gmail.com  Sat Oct 20 17:53:49 2007
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 20 Oct 2007 18:53:49 -0300
Subject: [Numpy-discussion] NumPy 1.0.4 release
In-Reply-To: <4719C75A.8030705@ar.media.kyoto-u.ac.jp>
References: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>
	<47182D70.7060304@ar.media.kyoto-u.ac.jp> <4718379F.806@enthought.com>
	<47183C10.5050100@ar.media.kyoto-u.ac.jp>
	<e7ba66e40710190759s2127a7efq763deb6bafdece66@mail.gmail.com>
	<4719C75A.8030705@ar.media.kyoto-u.ac.jp>
Message-ID: <e7ba66e40710201453y1b1bad18h328d62850ba3bbcf@mail.gmail.com>

On 10/20/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
And I
> hope that in the end, scons will be used for numpy (for 1.1 ?), once I
> finish the conversion.
>
> I don't see situations where adding 350 Kb in the tarball can be an
> issue, so could you tell me what the problem would be ?

Then if numpy is going to use scons for build itself in the near
future, then there is no problem, and including scons in numpy tarball
is good from the user side.

Just a final observation. Try to make it possible to reuse a scons
other than that included in numpy tarball. Packagers would be happy
with this.

Regards,

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594


From david at ar.media.kyoto-u.ac.jp  Sun Oct 21 04:14:59 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Sun, 21 Oct 2007 17:14:59 +0900
Subject: [Numpy-discussion] NumPy 1.0.4 release
In-Reply-To: <e7ba66e40710201453y1b1bad18h328d62850ba3bbcf@mail.gmail.com>
References: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>	<47182D70.7060304@ar.media.kyoto-u.ac.jp>
	<4718379F.806@enthought.com>	<47183C10.5050100@ar.media.kyoto-u.ac.jp>	<e7ba66e40710190759s2127a7efq763deb6bafdece66@mail.gmail.com>	<4719C75A.8030705@ar.media.kyoto-u.ac.jp>
	<e7ba66e40710201453y1b1bad18h328d62850ba3bbcf@mail.gmail.com>
Message-ID: <471B0A83.6080907@ar.media.kyoto-u.ac.jp>

Lisandro Dalcin wrote:
> On 10/20/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> And I
>   
>> hope that in the end, scons will be used for numpy (for 1.1 ?), once I
>> finish the conversion.
>>
>> I don't see situations where adding 350 Kb in the tarball can be an
>> issue, so could you tell me what the problem would be ?
>>     
>
> Then if numpy is going to use scons for build itself in the near
> future, then there is no problem, and including scons in numpy tarball
> is good from the user side.
>   
Just as a precision: I am the only one so far working on this (using 
scons instead of distutils), and no decision has been made whether it 
will be used at all in the future.
> Just a final observation. Try to make it possible to reuse a scons
> other than that included in numpy tarball. Packagers would be happy
> with this.
>   
Yes, I though about that, it would not be too difficult to handle (and 
scons has a simple mechanism to ensure a given version), since scons is 
called externally (it could be a option given to the scons command 
inside distutils, for example).

cheers,

David


From matthieu.brucher at gmail.com  Tue Oct 23 02:30:48 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 23 Oct 2007 08:30:48 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710190328u7b2b950foebd04943fda15bc5@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
	<4713C111.5010609@enthought.com>
	<e76aa17f0710190327p18a55596q38fada88d23dddb@mail.gmail.com>
	<e76aa17f0710190328u7b2b950foebd04943fda15bc5@mail.gmail.com>
Message-ID: <e76aa17f0710222330i346423d1v5bb91ca69aa28363@mail.gmail.com>

The problem reappeared today, same error, always in the line :
xp = (x-self.loc) * (1/self.scale)

When multiplying an array with a float.

Matthieu

2007/10/19, Matthieu Brucher <matthieu.brucher at gmail.com>:
>
> Oups, sorry for the sending twice, we have some problem with our DNS in
> Strasbourg :(
>
> 2007/10/19, Matthieu Brucher <matthieu.brucher at gmail.com >:
> >
> > Here is an excerpt of the stack on the numpy svn of wednesday :
> >
> > #0  0x40000402 in __kernel_vsyscall ()
> > #1  0x00b8e382 in sem_post at GLIBC_2.0 () from /lib/libpthread.so.0
> > #2  0x080fe5b7 in PyThread_release_lock (lock=0x80d0fb2) at
> > Python/thread_pthread.h:374
> > #3  0x080d0fb2 in PyEval_SaveThread () at Python/ceval.c:299
> > #4  0x4064ec7a in dotblas_innerproduct (dummy=0x0, args=0x960522c) at
> > numpy/core/blasdot/_dotblas.c:835
> > #5  0x0811ecf1 in PyCFunction_Call (func=0x960522c, arg=0x80cfec7,
> > kw=0x942068c) at Objects/methodobject.c:73
> > #6  0x080cfec7 in call_function (pp_stack=0x0, oparg=1) at
> > Python/ceval.c:3564
> > #7  0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> > #8  0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> > #9  0x0811de5b in function_call (func=0x0, arg=0x960516c, kw=0x965b148)
> > at Objects/funcobject.c:517
> > #10 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960516c, kw=0x8061a66)
> > at Objects/abstract.c:1860
> > #11 0x08061a66 in instancemethod_call (func=0x95f16ac, arg=0x94f5824,
> > kw=0x805bdb6) at Objects/classobject.c:2497
> > #12 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac, kw=0x80a86e9)
> > at Objects/abstract.c:1860
> > #13 0x080a86e9 in slot_tp_call (self=0xbf975c78, args=0x0,
> > kwds=0x805bdb6) at Objects/typeobject.c:4633
> > #14 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac, kw=0x80d017b)
> > at Objects/abstract.c:1860
> > #15 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=154955456)
> > at Python/ceval.c:3775
> > #16 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> > Python/ceval.c:3587
> > #17 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> > #18 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> > #19 0x0811de5b in function_call (func=0x0, arg=0x960520c, kw=0x30) at
> > Objects/funcobject.c:517
> > #20 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960520c, kw=0x8061a66)
> > at Objects/abstract.c:1860
> > #21 0x08061a66 in instancemethod_call (func=0x95ecbac, arg=0x94f57d4,
> > kw=0x805bdb6) at Objects/classobject.c:2497
> > #22 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac, kw=0x80a86e9)
> > at Objects/abstract.c:1860
> > #23 0x080a86e9 in slot_tp_call (self=0xbf97609c, args=0x0,
> > kwds=0x805bdb6) at Objects/typeobject.c:4633
> > #24 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac, kw=0x80d017b)
> > at Objects/abstract.c:1860
> > #25 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=156957888)
> > at Python/ceval.c:3775
> > #26 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> > Python/ceval.c:3587
> > #27 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> > #28 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> > #29 0x0811de5b in function_call (func=0x0, arg=0x960518c, kw=0x9548a1)
> > at Objects/funcobject.c:517
> > #30 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960518c, kw=0x8061a66)
> > at Objects/abstract.c:1860
> > #31 0x08061a66 in instancemethod_call (func=0x960526c, arg=0x95fc874,
> > kw=0x805bdb6) at Objects/classobject.c:2497
> > #32 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c, kw=0x80a86e9)
> > at Objects/abstract.c:1860
> > #33 0x080a86e9 in slot_tp_call (self=0xbf9764c0, args=0x0,
> > kwds=0x805bdb6) at Objects/typeobject.c:4633
> > #34 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c, kw=0x80d017b)
> > at Objects/abstract.c:1860
> > #35 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=157640136)
> > at Python/ceval.c:3775
> > #36 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> > Python/ceval.c:3587
> > #37 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> > #38 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> > #39 0x0811de5b in function_call (func=0x0, arg=0x95f260c, kw=0x0) at
> > Objects/funcobject.c:517
> > #40 0x0805bdb6 in PyObject_Call (func=0x9609934, arg=0x95f260c,
> > kw=0x8061a66) at Objects/abstract.c:1860
> >
> > Seems that
> >
> > 2007/10/15, Travis E. Oliphant < oliphant at enthought.com >:
> > >
> > > Matthieu Brucher wrote:
> > > >
> > > >     The problem is that there is a data-type reference counting
> > > error some
> > > >     where that is attempting to deallocate the built-in data-type
> > > 'l'
> > > >
> > > >
> > > >
> > > > That's what I supposed, but I couldn't find the reason why it wanted
> > > > to do this
> > > >
> > > >
> > > >     It's not really a Python error but a logging.  The code won't
> > > let you
> > > >     deallocate the built-ins, but it will tell you that something
> > > >     tried to.
> > > >
> > > >     Reference counting on data-types is easy to get wrong
> > > >     (particularly with
> > > >     Pyrex extension modules) because most calls consume a reference
> > > to the
> > > >     data-type (if they return an object that  contains a reference
> > > to the
> > > >     data-type).
> > > >
> > > >     It is a bug, and it would be nice to figure it out, but that
> > > would
> > > >     require the code that caused it.
> > > >
> > > >
> > > > I've updated my numpy version to the latest svn, the behaviour seems
> > >
> > > > to be different (more warnings), I'll try to give more information
> > > > about the error, but giving the whole code will not be simple (it
> > > uses
> > > > a big data file that seems to trigger the error as with other data
> > > > files, the error didn't show up :()
> > > >
> > >
> > > There are two types of errors that can occur with reference counting
> > > on
> > > data-types.
> > >
> > > 1) There are too many DECREF's --- this gets us to the error quickly
> > > and
> > > is usually easy to reproduce
> > > 2) There are too many INCREF's (the reference count keeps going up
> > > until
> > > the internal counter wraps around to 0 and deallocation is attempted)
> > > --- this error is harder to reproduce and usually takes a while before
> > >
> > > it happens in the code.
> > >
> > >
> > > -Travis
> > >
> > > _______________________________________________
> > > Numpy-discussion mailing list
> > > Numpy-discussion at scipy.org
> > > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> > >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071023/4069ab53/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Tue Oct 23 06:48:52 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 23 Oct 2007 19:48:52 +0900
Subject: [Numpy-discussion] How to write unit tests in distutils ?
Message-ID: <471DD194.7020607@ar.media.kyoto-u.ac.jp>

Hi,

    I would like to implement some unit tests for the code I am 
producing for scons support. As the numpy unit test is part of 
distutils, there is a bootstrap problem, and I am not sure I understand 
how current unit tests work in distutils: can anyone enlighten me on 
this one ?

    cheers,

    David



From cimrman3 at ntc.zcu.cz  Tue Oct 23 09:54:49 2007
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Tue, 23 Oct 2007 15:54:49 +0200
Subject: [Numpy-discussion] ANN: SFE-00.31.06 release
Message-ID: <471DFD29.5040007@ntc.zcu.cz>

I am happy to announce the version 00.31.06 of SFE, featuring acoustic
band gaps computation, rigid body motion constraints, new solver classes
and reorganization, and regular bug fixes and updates, see
http://ui505p06-mbs.ntc.zcu.cz/sfe.

SFE is a finite element analysis software written almost entirely in Python.

This version is released under BSD license.

best wishes,
r.


From cookedm at physics.mcmaster.ca  Tue Oct 23 12:40:43 2007
From: cookedm at physics.mcmaster.ca (David M. Cooke)
Date: Tue, 23 Oct 2007 12:40:43 -0400
Subject: [Numpy-discussion] How to write unit tests in distutils ?
In-Reply-To: <471DD194.7020607@ar.media.kyoto-u.ac.jp> (David Cournapeau's
	message of "Tue\, 23 Oct 2007 19\:48\:52 +0900")
References: <471DD194.7020607@ar.media.kyoto-u.ac.jp>
Message-ID: <m2myu94wdw.fsf@saturn.dmc>

David Cournapeau <david at ar.media.kyoto-u.ac.jp> writes:

> Hi,
>
>     I would like to implement some unit tests for the code I am 
> producing for scons support. As the numpy unit test is part of 
> distutils, there is a bootstrap problem, and I am not sure I understand 
> how current unit tests work in distutils: can anyone enlighten me on 
> this one ?

You'd probably have better luck running them separately from the numpy
unittests. e.g, 'import numpy.distutils; numpy.distutils.test()'.
Numpy will have to be installed to do so (as opposed to running from the
source directory). Installing numpy before running the tests is good
practice, anyways.

-- 
|>|\/|<
/------------------------------------------------------------------\
|David M. Cooke              http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca


From dalcinl at gmail.com  Tue Oct 23 16:17:54 2007
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 23 Oct 2007 17:17:54 -0300
Subject: [Numpy-discussion] a simple question about f2py
Message-ID: <e7ba66e40710231317v55bdbef7gbc3c210fd1299e54@mail.gmail.com>

Im trying to use f2py to wrap some fortran functions wich receive as
argument PETSc objects. The usual way to implement this with
PETSc+fortran is to do something like this

soubrutine matvec(A,x,y)
 Mat A
 Vec x
 Vec y
end soubrutine

The 'Mat' and 'Vec' types are actually integers of appropriate kind,
PETSc uses a #define to
define them. I there any way I can teach/hack f2py to translate
Mat/Vec/etc. to a normal fortran type?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594


From millman at berkeley.edu  Wed Oct 24 01:26:38 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 23 Oct 2007 22:26:38 -0700
Subject: [Numpy-discussion] anyone compiling on IRIX 6.5
Message-ID: <c7009a550710232226y3343805ds54f835f8a98575aa@mail.gmail.com>

Hello,

I am hoping to close a few of the remaining tickets for the upcoming
NumPy 1.0.4 release.  Is anyone using NumPy on IRIX?  Or have access
to IRIX?  If so, could you please take a look at this ticket:
http://projects.scipy.org/scipy/numpy/ticket/417

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From millman at berkeley.edu  Wed Oct 24 01:37:52 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Tue, 23 Oct 2007 22:37:52 -0700
Subject: [Numpy-discussion] Fwd: Support universal gfortran compiler on OS X
In-Reply-To: <c7009a550710190149k2a098aa1t5b194176ec531b0b@mail.gmail.com>
References: <c7009a550710190149k2a098aa1t5b194176ec531b0b@mail.gmail.com>
Message-ID: <c7009a550710232237j66f1ff1ek3e1cd438368ee146@mail.gmail.com>

Hello,

If someone could confirm whether we can close this ticket, I would
appreciate it.

Thanks,

---------- Forwarded message ----------
From: Jarrod Millman <millman at berkeley.edu>
Date: Oct 19, 2007 1:49 AM
Subject: Support universal gfortran compiler on OS X
To: Discussion of Numerical Python <numpy-discussion at scipy.org>


Hello,

Could someone confirm whether this ticket can be closed:
http://projects.scipy.org/scipy/numpy/ticket/571

Thanks,

--
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From fdu.xiaojf at gmail.com  Wed Oct 24 02:53:50 2007
From: fdu.xiaojf at gmail.com (fdu.xiaojf at gmail.com)
Date: Wed, 24 Oct 2007 14:53:50 +0800
Subject: [Numpy-discussion] anyone compiling on IRIX 6.5
In-Reply-To: <c7009a550710232226y3343805ds54f835f8a98575aa@mail.gmail.com>
References: <c7009a550710232226y3343805ds54f835f8a98575aa@mail.gmail.com>
Message-ID: <471EEBFE.5090102@gmail.com>

Jarrod Millman wrote:
> Hello,
> 
> I am hoping to close a few of the remaining tickets for the upcoming
> NumPy 1.0.4 release.  Is anyone using NumPy on IRIX?  Or have access
> to IRIX?  If so, could you please take a look at this ticket:
> http://projects.scipy.org/scipy/numpy/ticket/417
> 
> Thanks,
> 

IRIX is a very annoying OS for me. There are numerous errors and warnings 
whenever I tried to compile. Now I always try Linux first if possible when I 
have to compile and use some programs, and IRIX is my last choice to try.

I do have access to IRIX, but I'm definitely not an expert about it. Howerer, 
I'm willing to try this job if there are no volunteers.

Regards,


From david at ar.media.kyoto-u.ac.jp  Wed Oct 24 03:01:55 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 24 Oct 2007 16:01:55 +0900
Subject: [Numpy-discussion] anyone compiling on IRIX 6.5
In-Reply-To: <471EEBFE.5090102@gmail.com>
References: <c7009a550710232226y3343805ds54f835f8a98575aa@mail.gmail.com>
	<471EEBFE.5090102@gmail.com>
Message-ID: <471EEDE3.5080901@ar.media.kyoto-u.ac.jp>

fdu.xiaojf at gmail.com wrote:
> Jarrod Millman wrote:
>   
>> Hello,
>>
>> I am hoping to close a few of the remaining tickets for the upcoming
>> NumPy 1.0.4 release.  Is anyone using NumPy on IRIX?  Or have access
>> to IRIX?  If so, could you please take a look at this ticket:
>> http://projects.scipy.org/scipy/numpy/ticket/417
>>
>> Thanks,
>>
>>     
>
> IRIX is a very annoying OS for me. There are numerous errors and warnings 
> whenever I tried to compile. Now I always try Linux first if possible when I 
> have to compile and use some programs, and IRIX is my last choice to try.
>
> I do have access to IRIX, but I'm definitely not an expert about it. Howerer, 
> I'm willing to try this job if there are no volunteers.
>   
If you are willing to test, could you try the numpy.scons branch on it ? 
I don't have access to IRIX myself, so I cannot test if it works. I 
would hope that it would work out of the box (it did on solaris, 
platform where the trunk does not build right now).

cheers,

David


From fdu.xiaojf at gmail.com  Wed Oct 24 03:46:25 2007
From: fdu.xiaojf at gmail.com (fdu.xiaojf at gmail.com)
Date: Wed, 24 Oct 2007 15:46:25 +0800
Subject: [Numpy-discussion] anyone compiling on IRIX 6.5
In-Reply-To: <471EEDE3.5080901@ar.media.kyoto-u.ac.jp>
References: <c7009a550710232226y3343805ds54f835f8a98575aa@mail.gmail.com>	<471EEBFE.5090102@gmail.com>
	<471EEDE3.5080901@ar.media.kyoto-u.ac.jp>
Message-ID: <471EF851.4010105@gmail.com>

Hi David,
David Cournapeau wrote:

> fdu.xiaojf at gmail.com wrote:
>> Jarrod Millman wrote:
>>   
>>> Hello,
>>>
>>> I am hoping to close a few of the remaining tickets for the upcoming
>>> NumPy 1.0.4 release.  Is anyone using NumPy on IRIX?  Or have access
>>> to IRIX?  If so, could you please take a look at this ticket:
>>> http://projects.scipy.org/scipy/numpy/ticket/417
>>>
>>> Thanks,
>>>
>>>     
>> IRIX is a very annoying OS for me. There are numerous errors and warnings 
>> whenever I tried to compile. Now I always try Linux first if possible when I 
>> have to compile and use some programs, and IRIX is my last choice to try.
>>
>> I do have access to IRIX, but I'm definitely not an expert about it. Howerer, 
>> I'm willing to try this job if there are no volunteers.
>>   
> If you are willing to test, could you try the numpy.scons branch on it ? 
> I don't have access to IRIX myself, so I cannot test if it works. I 
> would hope that it would work out of the box (it did on solaris, 
> platform where the trunk does not build right now).
> 
> cheers,
> 
> David

I have tried to compile numpy.scons on IRIX with "python setup.py install". It 
compiled with warnings, but not errors. Attached is the compile log file.

But It seems the numpy just doesn't work. The following is what I have tried:

"""
Python 2.4.4 (#2, May 17 2004, 22:47:37) [C] on irix6
Type "help", "copyright", "credits" or "license" for more information.
 >>> from numpy import *
Running from numpy source directory.
 >>> a = arange(1,100,0.1)
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
NameError: name 'arange' is not defined
 >>> a = array(range(10))
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
NameError: name 'array' is not defined
 >>> import numpy
 >>> numpy.test()
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
AttributeError: 'module' object has no attribute 'test'
 >>> numpy.array
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
AttributeError: 'module' object has no attribute 'array'
 >>> """
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compile.log.gz
Type: application/x-gzip
Size: 8861 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071024/b539e01f/attachment.bin>

From david at ar.media.kyoto-u.ac.jp  Wed Oct 24 04:11:59 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 24 Oct 2007 17:11:59 +0900
Subject: [Numpy-discussion] anyone compiling on IRIX 6.5
In-Reply-To: <471EF851.4010105@gmail.com>
References: <c7009a550710232226y3343805ds54f835f8a98575aa@mail.gmail.com>	<471EEBFE.5090102@gmail.com>	<471EEDE3.5080901@ar.media.kyoto-u.ac.jp>
	<471EF851.4010105@gmail.com>
Message-ID: <471EFE4F.90507@ar.media.kyoto-u.ac.jp>

fdu.xiaojf at gmail.com wrote:
> Hi David,
> David Cournapeau wrote:
>
>> fdu.xiaojf at gmail.com wrote:
>>> Jarrod Millman wrote:
>>>  
>>>> Hello,
>>>>
>>>> I am hoping to close a few of the remaining tickets for the upcoming
>>>> NumPy 1.0.4 release.  Is anyone using NumPy on IRIX?  Or have access
>>>> to IRIX?  If so, could you please take a look at this ticket:
>>>> http://projects.scipy.org/scipy/numpy/ticket/417
>>>>
>>>> Thanks,
>>>>
>>>>     
>>> IRIX is a very annoying OS for me. There are numerous errors and 
>>> warnings whenever I tried to compile. Now I always try Linux first 
>>> if possible when I have to compile and use some programs, and IRIX 
>>> is my last choice to try.
>>>
>>> I do have access to IRIX, but I'm definitely not an expert about it. 
>>> Howerer, I'm willing to try this job if there are no volunteers.
>>>   
>> If you are willing to test, could you try the numpy.scons branch on 
>> it ? I don't have access to IRIX myself, so I cannot test if it 
>> works. I would hope that it would work out of the box (it did on 
>> solaris, platform where the trunk does not build right now).
>>
>> cheers,
>>
>> David
>
> I have tried to compile numpy.scons on IRIX with "python setup.py 
> install". It compiled with warnings, but not errors. Attached is the 
> compile log file.
>
> But It seems the numpy just doesn't work. The following is what I have 
> tried:
>
> """
> Python 2.4.4 (#2, May 17 2004, 22:47:37) [C] on irix6
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from numpy import *
> Running from numpy source directory.
Here is the problem: this means numpy will NOT work. Two possibilities:
    - you really are in numpy source directory (i.e the directory where 
the setup.py file is): then launch python from another directory, and 
try importing again.
     - you are not in numpy source directory: there is a problem 
somewhere in the build system I should correct.

cheers,

David


From fdu.xiaojf at gmail.com  Wed Oct 24 04:36:22 2007
From: fdu.xiaojf at gmail.com (fdu.xiaojf at gmail.com)
Date: Wed, 24 Oct 2007 16:36:22 +0800
Subject: [Numpy-discussion] anyone compiling on IRIX 6.5
In-Reply-To: <471EFE4F.90507@ar.media.kyoto-u.ac.jp>
References: <c7009a550710232226y3343805ds54f835f8a98575aa@mail.gmail.com>	<471EEBFE.5090102@gmail.com>	<471EEDE3.5080901@ar.media.kyoto-u.ac.jp>	<471EF851.4010105@gmail.com>
	<471EFE4F.90507@ar.media.kyoto-u.ac.jp>
Message-ID: <471F0406.5090103@gmail.com>

David Cournapeau wrote:
 >> Python 2.4.4 (#2, May 17 2004, 22:47:37) [C] on irix6
 >> Type "help", "copyright", "credits" or "license" for more information.
 >>>>> from numpy import *
 >> Running from numpy source directory.
 > Here is the problem: this means numpy will NOT work. Two possibilities:
 >     - you really are in numpy source directory (i.e the directory where
 > the setup.py file is): then launch python from another directory, and
 > try importing again.

Oh, yes, I was in the numpy source directory.

 >      - you are not in numpy source directory: there is a problem
 > somewhere in the build system I should correct.
 >
 > cheers,
 >
 > David


I changed directory, and do the following test:

"""
Python 2.4.4 (#2, May 17 2004, 22:47:37) [C] on irix6
Type "help", "copyright", "credits" or "license" for more information.
 >>> import numpy
Warning: Error importing pyext, error was ctypes is not available.
 >>> numpy.test()
Numpy is installed in /disk2/jfxiao/local/lib/python2.4/site-packages/numpy
Numpy version 1.0.4.dev4276
Python version 2.4.4 (#2, May 17 2004, 22:47:37) [C]
   Found 10/10 tests for numpy.core.defmatrix
   Found 36/36 tests for numpy.core.ma
   Found 221/221 tests for numpy.core.multiarray
   Found 62/62 tests for numpy.core.numeric
   Found 31/31 tests for numpy.core.numerictypes
   Found 12/12 tests for numpy.core.records
   Found 6/6 tests for numpy.core.scalarmath
   Found 14/14 tests for numpy.core.umath
   Found 4/4 tests for numpy.ctypeslib
   Found 5/5 tests for numpy.distutils.misc_util
   Found 1/1 tests for numpy.fft.fftpack
   Found 3/3 tests for numpy.fft.helper
   Found 9/9 tests for numpy.lib.arraysetops
   Found 46/46 tests for numpy.lib.function_base
   Found 5/5 tests for numpy.lib.getlimits
   Found 4/4 tests for numpy.lib.index_tricks
   Found 3/3 tests for numpy.lib.polynomial
   Found 49/49 tests for numpy.lib.shape_base
   Found 14/14 tests for numpy.lib.twodim_base
   Found 43/43 tests for numpy.lib.type_check
   Found 1/1 tests for numpy.lib.ufunclike
   Found 40/40 tests for numpy.linalg
   Found 2/2 tests for numpy.random
   Found 1/1 tests for numpy.scons_fake.pyext.spam
   Found 0/0 tests for __main__
........................................................................................................................................................................................................................................................................................................................................................................................................F.....................................................................................................................................................................................................................................................................................................sh: 
dir:  not found
.
======================================================================
FAIL: test_divideerr (numpy.core.tests.test_numeric.test_seterr)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/disk2/jfxiao/local/lib/python2.4/site-packages/numpy/core/tests/test_numeric.py", 
line 196, in test_divideerr
     self.fail()
AssertionError

----------------------------------------------------------------------
Ran 687 tests in 4.659s

FAILED (failures=1)
<unittest._TextTestResult run=687 errors=0 failures=1>
 >>> a = numpy.array(range(10))
 >>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
 >>> b = a*a
 >>> b
array([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])
 >>>
"""

Regards,



From david at ar.media.kyoto-u.ac.jp  Wed Oct 24 04:34:31 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Wed, 24 Oct 2007 17:34:31 +0900
Subject: [Numpy-discussion] anyone compiling on IRIX 6.5
In-Reply-To: <471F0406.5090103@gmail.com>
References: <c7009a550710232226y3343805ds54f835f8a98575aa@mail.gmail.com>	<471EEBFE.5090102@gmail.com>	<471EEDE3.5080901@ar.media.kyoto-u.ac.jp>	<471EF851.4010105@gmail.com>	<471EFE4F.90507@ar.media.kyoto-u.ac.jp>
	<471F0406.5090103@gmail.com>
Message-ID: <471F0397.9010103@ar.media.kyoto-u.ac.jp>

fdu.xiaojf at gmail.com wrote:
> David Cournapeau wrote:
>  >> Python 2.4.4 (#2, May 17 2004, 22:47:37) [C] on irix6
>  >> Type "help", "copyright", "credits" or "license" for more information.
>  >>>>> from numpy import *
>  >> Running from numpy source directory.
>  > Here is the problem: this means numpy will NOT work. Two possibilities:
>  >     - you really are in numpy source directory (i.e the directory where
>  > the setup.py file is): then launch python from another directory, and
>  > try importing again.
>
> Oh, yes, I was in the numpy source directory.
>
>  >      - you are not in numpy source directory: there is a problem
>  > somewhere in the build system I should correct.
>  >
>  > cheers,
>  >
>  > David
>
>
> I changed directory, and do the following test:
>
> """
> Python 2.4.4 (#2, May 17 2004, 22:47:37) [C] on irix6
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import numpy
> Warning: Error importing pyext, error was ctypes is not available.
>  >>> numpy.test()
> Numpy is installed in /disk2/jfxiao/local/lib/python2.4/site-packages/numpy
> Numpy version 1.0.4.dev4276
> Python version 2.4.4 (#2, May 17 2004, 22:47:37) [C]
>    Found 10/10 tests for numpy.core.defmatrix
>    Found 36/36 tests for numpy.core.ma
>    Found 221/221 tests for numpy.core.multiarray
>    Found 62/62 tests for numpy.core.numeric
>    Found 31/31 tests for numpy.core.numerictypes
>    Found 12/12 tests for numpy.core.records
>    Found 6/6 tests for numpy.core.scalarmath
>    Found 14/14 tests for numpy.core.umath
>    Found 4/4 tests for numpy.ctypeslib
>    Found 5/5 tests for numpy.distutils.misc_util
>    Found 1/1 tests for numpy.fft.fftpack
>    Found 3/3 tests for numpy.fft.helper
>    Found 9/9 tests for numpy.lib.arraysetops
>    Found 46/46 tests for numpy.lib.function_base
>    Found 5/5 tests for numpy.lib.getlimits
>    Found 4/4 tests for numpy.lib.index_tricks
>    Found 3/3 tests for numpy.lib.polynomial
>    Found 49/49 tests for numpy.lib.shape_base
>    Found 14/14 tests for numpy.lib.twodim_base
>    Found 43/43 tests for numpy.lib.type_check
>    Found 1/1 tests for numpy.lib.ufunclike
>    Found 40/40 tests for numpy.linalg
>    Found 2/2 tests for numpy.random
>    Found 1/1 tests for numpy.scons_fake.pyext.spam
>    Found 0/0 tests for __main__
> ........................................................................................................................................................................................................................................................................................................................................................................................................F.....................................................................................................................................................................................................................................................................................................sh: 
> dir:  not found
> .
> ======================================================================
> FAIL: test_divideerr (numpy.core.tests.test_numeric.test_seterr)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File 
> "/disk2/jfxiao/local/lib/python2.4/site-packages/numpy/core/tests/test_numeric.py", 
> line 196, in test_divideerr
>      self.fail()
> AssertionError
>
> ----------------------------------------------------------------------
> Ran 687 tests in 4.659s
>
> FAILED (failures=1)
> <unittest._TextTestResult run=687 errors=0 failures=1>
>  >>> a = numpy.array(range(10))
>  >>> a
> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>  >>> b = a*a
>  >>> b
> array([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])
>  >>>
> """
>   
Ok. You can safely ignore the error importing pyext, as well as dir not 
found. The divideerr, I don't know what it is, I will check this. I am 
happy that IRIX works without touching it, though :)

cheers,

David


From ptvirtan at elisanet.fi  Wed Oct 24 10:23:06 2007
From: ptvirtan at elisanet.fi (Pauli Virtanen)
Date: Wed, 24 Oct 2007 17:23:06 +0300 (EEST)
Subject: [Numpy-discussion] a simple question about f2py
Message-ID: <23561802.18151193235787307.JavaMail.ptvirtan@elisanet.fi>

Lisandro Dalcin <dalcinl at gmail.com> kirjoitti: 
> Im trying to use f2py to wrap some fortran functions wich receive as
> argument PETSc objects. The usual way to implement this with
> PETSc+fortran is to do something like this
> 
> soubrutine matvec(A,x,y)
>  Mat A
>  Vec x
>  Vec y
> end soubrutine
> 
> The 'Mat' and 'Vec' types are actually integers of appropriate kind,
> PETSc uses a #define to
> define them. I there any way I can teach/hack f2py to translate
> Mat/Vec/etc. to a normal fortran type?

You can tell f2py to use a different type for the argument as follows:

    subroutine foo(a)
        !f2py integer, dimension(10), intent(inout) :: a
        Mat a
    end subroutine foo

This way, f2py will treat the parameter as a chunk of memory of size 10*sizeof(integer), and disregard any following definitions for it. Using a hack like this, it's also possible to pass derived type object pointers,  "type(Typename), pointer", from the Python side to the Fortran side, as opaque handles.

-- 
Pauli Virtanen


From barrywark at gmail.com  Wed Oct 24 12:52:05 2007
From: barrywark at gmail.com (Barry Wark)
Date: Wed, 24 Oct 2007 09:52:05 -0700
Subject: [Numpy-discussion] Fwd: Support universal gfortran compiler on
	OS X
In-Reply-To: <c7009a550710232237j66f1ff1ek3e1cd438368ee146@mail.gmail.com>
References: <c7009a550710190149k2a098aa1t5b194176ec531b0b@mail.gmail.com>
	<c7009a550710232237j66f1ff1ek3e1cd438368ee146@mail.gmail.com>
Message-ID: <cd7634ce0710240952i71338fdfm5e7ab5df648b43cf@mail.gmail.com>

The MacOSX/x86 buildslave at buildbot.scipy.org is running universal
gfortran. The output of gfortran -v -arch XX is:

siegfried:~ barry$ gfortran -v -arch i386
Using built-in specs.
Target: i686-apple-darwin8
Configured with: /Builds/unix/gcc/gcc-4.2/configure
--prefix=/usr/local --mandir=/share/man
--program-transform-name=/^[cg][^.-]*$/s/$/-4.2/
--build=i686-apple-darwin8 --host=i686-apple-darwin8
--target=i686-apple-darwin8 --enable-languages=fortran
Thread model: posix
gcc version 4.2.1

siegfried:~ barry$ gfortran -v -arch ppc
Using built-in specs.
Target: powerpc-apple-darwin8
Configured with: /Builds/unix/gcc/gcc-4.2/configure
--prefix=/usr/local --mandir=/share/man
--program-transform-name=/^[cg][^.-]*$/s/$/-4.2/
--build=i686-apple-darwin8 --host=i686-apple-darwin8
--target=powerpc-apple-darwin8 --enable-languages=fortran
Thread model: posix
gcc version 4.2.1


I can confirm that numpy builds properly with this compiler. I can
also confirm that the universal build works on PPC and Intel. I can't
confirm that the patch for universal gfortran support has not caused a
regression in the non-universal gfortran support on OS X.


barry

On 10/23/07, Jarrod Millman <millman at berkeley.edu> wrote:
> Hello,
>
> If someone could confirm whether we can close this ticket, I would
> appreciate it.
>
> Thanks,
>
> ---------- Forwarded message ----------
> From: Jarrod Millman <millman at berkeley.edu>
> Date: Oct 19, 2007 1:49 AM
> Subject: Support universal gfortran compiler on OS X
> To: Discussion of Numerical Python <numpy-discussion at scipy.org>
>
>
> Hello,
>
> Could someone confirm whether this ticket can be closed:
> http://projects.scipy.org/scipy/numpy/ticket/571
>
> Thanks,
>
> --
> Jarrod Millman
> Computational Infrastructure for Research Labs
> 10 Giannini Hall, UC Berkeley
> phone: 510.643.4014
> http://cirl.berkeley.edu/
>
>
> --
> Jarrod Millman
> Computational Infrastructure for Research Labs
> 10 Giannini Hall, UC Berkeley
> phone: 510.643.4014
> http://cirl.berkeley.edu/
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


From dalcinl at gmail.com  Wed Oct 24 17:12:53 2007
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 24 Oct 2007 18:12:53 -0300
Subject: [Numpy-discussion] a simple question about f2py
In-Reply-To: <23561802.18151193235787307.JavaMail.ptvirtan@elisanet.fi>
References: <23561802.18151193235787307.JavaMail.ptvirtan@elisanet.fi>
Message-ID: <e7ba66e40710241412k25ba8740tfe03bc04c5d677ef@mail.gmail.com>

On 10/24/07, Pauli Virtanen <ptvirtan at elisanet.fi> wrote:
>Using a hack like this, it's also possible to pass derived type
object >pointers,  "type(Typename), pointer", from the Python side to
the >Fortran side, as opaque handles.

Could you please send/point me an example of how I can actually pass
Fortran pointers between Python and Fortran? I cannot get that
working...

I'm trying to allocate arrays in Fortran, get a opaque handle to it in
Python, and finally pass-back the pointer to Fortran for
deallocation... Is this possible?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594


From dalcinl at gmail.com  Wed Oct 24 20:14:18 2007
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 24 Oct 2007 21:14:18 -0300
Subject: [Numpy-discussion] a simple question about f2py
In-Reply-To: <23561802.18151193235787307.JavaMail.ptvirtan@elisanet.fi>
References: <23561802.18151193235787307.JavaMail.ptvirtan@elisanet.fi>
Message-ID: <e7ba66e40710241714o10e219bbh6c3dc019841ed169@mail.gmail.com>

Pauli, I finally understood your idea!!!! What a good hack!!!

You have to pass an integer array with enough space in order Fortran
con store there some extra metadata, no just the buffer pointer, but
also dimensions and some other runtime stuff.

Many, many, many thanks....


On 10/24/07, Pauli Virtanen <ptvirtan at elisanet.fi> wrote:
>Using a hack like this, it's also possible to pass derived type
object >pointers,  "type(Typename), pointer", from the Python side to
the >Fortran side, as opaque handles.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594


From david at ar.media.kyoto-u.ac.jp  Thu Oct 25 00:50:34 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Thu, 25 Oct 2007 13:50:34 +0900
Subject: [Numpy-discussion] How to write unit tests in distutils ?
In-Reply-To: <m2myu94wdw.fsf@saturn.dmc>
References: <471DD194.7020607@ar.media.kyoto-u.ac.jp>
	<m2myu94wdw.fsf@saturn.dmc>
Message-ID: <4720209A.8050503@ar.media.kyoto-u.ac.jp>

David M. Cooke wrote:
> David Cournapeau <david at ar.media.kyoto-u.ac.jp> writes:
>
>   
>> Hi,
>>
>>     I would like to implement some unit tests for the code I am 
>> producing for scons support. As the numpy unit test is part of 
>> distutils, there is a bootstrap problem, and I am not sure I understand 
>> how current unit tests work in distutils: can anyone enlighten me on 
>> this one ?
>>     
>
> You'd probably have better luck running them separately from the numpy
> unittests. e.g, 'import numpy.distutils; numpy.distutils.test()'.
> Numpy will have to be installed to do so (as opposed to running from the
> source directory). Installing numpy before running the tests is good
> practice, anyways.
>
>   
Ok, I will do that, then. Thanks,

David


From o.kranz at gmx.de  Thu Oct 25 06:35:33 2007
From: o.kranz at gmx.de (Oliver Kranz)
Date: Thu, 25 Oct 2007 12:35:33 +0200
Subject: [Numpy-discussion] C-API for non-contiguous arrays
Message-ID: <47207175.3020108@gmx.de>

Hi,

I am working on a Python extension module using of the NumPy C-API. The 
extension module is an interface to an image processing and analysis 
library written in C++. The C++ functions are exported with 
boos::python. Currently I am implementing the support of 
three-dimensional data sets which can consume a huge amount of memory. 
The 3D data is stored in a numpy.ndarray. This array is passed to C++ 
functions which do the calculations.

In general, multi-dimensional arrays can be organized in memory in four 
different ways:
1. C order contiguous
2. Fortran order contiguous
3. C order non-contiguous
4. Fortran order non-contiguous

Am I right that the NumPy C-API can only distinguish between three ways 
the array is organized in memory? These are:
1. C order contiguous e.g. with PyArray_ISCONTIGUOUS(arr)
2. Fortran order contiguous e.g. with PyArray_ISFORTRAN(arr)
3. non-contiguous e.g. with !PyArray_ISCONTIGUOUS(arr) &&
!PyArray_ISFORTRAN(arr)

So there is no way to find out if a non-contiguous array has C order or 
Fortran order. The same holds for Python code e.g. by use of the flags.

a.flags.contiguous
a.flags.fortran

This is very important for me because I just want to avoid to copy every 
non-contiguous array into a contiguous array. This would be very 
inefficient. But I can't find an other solution than copying the array.

Also the iterator provided by the C-API only loops over the array in C 
order. Even if the array is in Fortran non-contiguous order.

Or are there just no Fortran order non-contiguous arrays? I think I can 
construct one.

a = numpy.ndarray((3,4,5), order="F")
b = a[:,1:2,:]

Now, I think b's elements are organized in memory in Fortran 
non-contiguous order. But the flags only tell me that it is 
non-contiguous and not if it is in Fortran order or in C order. And if b 
would be passed to a C++ function it would not be possible to find out 
with the C-API if it is in Fortran order or in C order, too.

Any ideas? Or do I always have to create contiguous arrays?

Cheers,
Oliver



From tim.hochberg at ieee.org  Thu Oct 25 07:18:13 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Thu, 25 Oct 2007 04:18:13 -0700
Subject: [Numpy-discussion] C-API for non-contiguous arrays
In-Reply-To: <47207175.3020108@gmx.de>
References: <47207175.3020108@gmx.de>
Message-ID: <e4412d6b0710250418w2c86c067sa9489cefed2644b@mail.gmail.com>

On 10/25/07, Oliver Kranz <o.kranz at gmx.de> wrote:
>
> Hi,
>
> I am working on a Python extension module using of the NumPy C-API. The
> extension module is an interface to an image processing and analysis
> library written in C++. The C++ functions are exported with
> boos::python. Currently I am implementing the support of
> three-dimensional data sets which can consume a huge amount of memory.
> The 3D data is stored in a numpy.ndarray. This array is passed to C++
> functions which do the calculations.
>
> In general, multi-dimensional arrays can be organized in memory in four
> different ways:
> 1. C order contiguous
> 2. Fortran order contiguous
> 3. C order non-contiguous
> 4. Fortran order non-contiguous


I believe that this is incorrect. Consider the following:

>>> import numpy as np
>>> a  = np.arange(27).reshape(3,3,3)
>>> a.strides
(36, 12, 4)
>>> a.transpose(2,1,0).strides
(4, 12, 36)
>>> a.transpose(0,2,1).strides
(36, 4, 12)

 I believe that the last transpose doesn't fit any of these four categories
and is simply discontiguous.

Am I right that the NumPy C-API can only distinguish between three ways
> the array is organized in memory? These are:
> 1. C order contiguous e.g. with PyArray_ISCONTIGUOUS(arr)
> 2. Fortran order contiguous e.g. with PyArray_ISFORTRAN(arr)
> 3. non-contiguous e.g. with !PyArray_ISCONTIGUOUS(arr) &&
> !PyArray_ISFORTRAN(arr)
>
> So there is no way to find out if a non-contiguous array has C order or
> Fortran order. The same holds for Python code e.g. by use of the flags.
>
> a.flags.contiguous
> a.flags.fortran
>
> This is very important for me because I just want to avoid to copy every
> non-contiguous array into a contiguous array. This would be very
> inefficient. But I can't find an other solution than copying the array.
>
> Also the iterator provided by the C-API only loops over the array in C
> order. Even if the array is in Fortran non-contiguous order.
>
> Or are there just no Fortran order non-contiguous arrays? I think I can
> construct one.
>
> a = numpy.ndarray((3,4,5), order="F")
> b = a[:,1:2,:]
>
> Now, I think b's elements are organized in memory in Fortran
> non-contiguous order. But the flags only tell me that it is
> non-contiguous and not if it is in Fortran order or in C order. And if b
> would be passed to a C++ function it would not be possible to find out
> with the C-API if it is in Fortran order or in C order, too.
>
> Any ideas? Or do I always have to create contiguous arrays?



By Fortran and C-Order discontiguous, do you simply mean that the strides
are in increasing and decreasing order respectively? If so, you could check
for that without too much trouble.



-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071025/e5e20e1f/attachment.html>

From bernhard.voigt at gmail.com  Thu Oct 25 08:25:30 2007
From: bernhard.voigt at gmail.com (Bernhard Voigt)
Date: Thu, 25 Oct 2007 14:25:30 +0200
Subject: [Numpy-discussion] how to select indices from boolean array
Message-ID: <21a270aa0710250525v15f0098dkebd3ae3ab787dd10@mail.gmail.com>

Dear list,

is nonzero the appropriate function to get the indexes of a boolean array?

foo = numpy.random.random(10000)
cut = foo > .5
indexes = cut.nonzero()[0]

Another question: Why is the the type returned by nonzero a tuple and not an
ndarray of the same shape as the input (or self if used on an object)?

Thanks! Bernhard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071025/b2482810/attachment.html>

From ptvirtan at elisanet.fi  Thu Oct 25 08:44:30 2007
From: ptvirtan at elisanet.fi (Pauli Virtanen)
Date: Thu, 25 Oct 2007 15:44:30 +0300 (EEST)
Subject: [Numpy-discussion] how to select indices from boolean array
Message-ID: <2167039.21341193316271619.JavaMail.ptvirtan@elisanet.fi>

wBernhard Voigt <bernhard.voigt at gmail.com> kirjoitti: 
> is nonzero the appropriate function to get the indexes of a boolean array?
> 
> foo = numpy.random.random(10000)
> cut = foo > .5
> indexes = cut.nonzero()[0]

I think writing

    indices = where(foo > 5)[0]

might be more clear, although it does the same thing as .nonzero().

> Another question: Why is the the type returned by nonzero a tuple and not an
> ndarray of the same shape as the input (or self if used on an object)?

Because indexing with booleans in numpy works differently from indexing with numbers, especially for multidimensional arrays. You want to have

   ( foo[foo > 5] == foo[where(foo > 5)] ).all()

so where/nonzero returns a tuple that contains 1D-arrays, one per dimension, which together enumerate the relevant entries. (IIRC, in Matlab, find returns indices in the flattened array.)

-- 
Pauli Virtanen


-- 




From o.kranz at gmx.de  Thu Oct 25 10:23:34 2007
From: o.kranz at gmx.de (Oliver Kranz)
Date: Thu, 25 Oct 2007 16:23:34 +0200
Subject: [Numpy-discussion] C-API for non-contiguous arrays
In-Reply-To: <e4412d6b0710250418w2c86c067sa9489cefed2644b@mail.gmail.com>
References: <47207175.3020108@gmx.de>
	<e4412d6b0710250418w2c86c067sa9489cefed2644b@mail.gmail.com>
Message-ID: <4720A6E6.5080100@gmx.de>

Timothy Hochberg wrote:
> 
> 
> On 10/25/07, *Oliver Kranz* <o.kranz at gmx.de <mailto:o.kranz at gmx.de>> wrote:
> 
>     Hi,
> 
>     I am working on a Python extension module using of the NumPy C-API. The
>     extension module is an interface to an image processing and analysis
>     library written in C++. The C++ functions are exported with
>     boos::python. Currently I am implementing the support of
>     three-dimensional data sets which can consume a huge amount of memory.
>     The 3D data is stored in a numpy.ndarray. This array is passed to C++
>     functions which do the calculations.
> 
>     In general, multi-dimensional arrays can be organized in memory in four
>     different ways:
>     1. C order contiguous
>     2. Fortran order contiguous
>     3. C order non-contiguous
>     4. Fortran order non-contiguous
> 
> 
> I believe that this is incorrect. Consider the following:
> 
>  >>> import numpy as np
>  >>> a  = np.arange(27).reshape(3,3,3)
>  >>> a.strides
> (36, 12, 4)
>  >>> a.transpose(2,1,0).strides
> (4, 12, 36)
>  >>> a.transpose(0,2,1).strides
> (36, 4, 12)
> 
>  I believe that the last transpose doesn't fit any of these four 
> categories and is simply discontiguous.

Yes, you are right. I did not consider this case.

>     Am I right that the NumPy C-API can only distinguish between three ways
>     the array is organized in memory? These are:
>     1. C order contiguous e.g. with PyArray_ISCONTIGUOUS(arr)
>     2. Fortran order contiguous e.g. with PyArray_ISFORTRAN(arr)
>     3. non-contiguous e.g. with !PyArray_ISCONTIGUOUS(arr) &&
>     !PyArray_ISFORTRAN(arr)
> 
>     So there is no way to find out if a non-contiguous array has C order or
>     Fortran order. The same holds for Python code e.g. by use of the flags.
> 
>     a.flags.contiguous
>     a.flags.fortran
> 
>     This is very important for me because I just want to avoid to copy
>     every
>     non-contiguous array into a contiguous array. This would be very
>     inefficient. But I can't find an other solution than copying the array.
> 
>     Also the iterator provided by the C-API only loops over the array in C
>     order. Even if the array is in Fortran non-contiguous order.
> 
>     Or are there just no Fortran order non-contiguous arrays? I think I can
>     construct one.
> 
>     a = numpy.ndarray((3,4,5), order="F")
>     b = a[:,1:2,:]
> 
>     Now, I think b's elements are organized in memory in Fortran
>     non-contiguous order. But the flags only tell me that it is
>     non-contiguous and not if it is in Fortran order or in C order. And if b
>     would be passed to a C++ function it would not be possible to find out
>     with the C-API if it is in Fortran order or in C order, too.
> 
>     Any ideas? Or do I always have to create contiguous arrays?
> 
> 
> 
> By Fortran and C-Order discontiguous, do you simply mean that the 
> strides are in increasing and decreasing order respectively? If so, you 
> could check for that without too much trouble.

Since I want to support all the different contiguous and non-contiguous 
arrays the best solution for me is always checking the strides if the 
array is not in C order contiguous.

Thanks,
Oliver


From jdh2358 at gmail.com  Thu Oct 25 14:00:29 2007
From: jdh2358 at gmail.com (John Hunter)
Date: Thu, 25 Oct 2007 13:00:29 -0500
Subject: [Numpy-discussion] convolution and wiener khinchin
Message-ID: <88e473830710251100v5bec9276ode04965a9260969b@mail.gmail.com>

I am working on an example to illustrate convolution in the temporal
and spectral domains, using the property that a convolution in the
time domain is a multiplication in the fourier domain.  I am using
numpy.fft and numpy.convolve to compute the solution two ways, and
comparing them.  I am getting an error for small times in my fourier
solution.  At first I thought this was because of edge affects, but I
see it even when I apply a windowing function.

Can anyone advise me about what I am doing wrong here?

"""
In signal processing, the output of a linear system to an arbitrary
input is given by the convolution of the impulse response function (the
system response to a Dirac-delta impulse) and the input signal.

Mathematically:

  y(t) = \int_0^\t x(\tau)r(t-\tau)d\tau


where x(t) is the input signal at time t, y(t) is the output, and r(t)
is the impulse response function.

In this exercise, we will compute investigate the convolution of a
white noise process with a double exponential impulse response
function, and compute the results 3 ways:

  * using numpy.convolve

  * in Fourier space using the property that a convolution in the
    temporal domain is a multiplication in the fourier domain
"""

import numpy as npy
import matplotlib.mlab as mlab
from pylab import figure, show

# build the time, input, output and response arrays
dt = 0.01
t = npy.arange(0.0, 10.0, dt)        # the time vector from 0..5
Nt = len(t)

def impulse_response(t):
    'double exponential response function'
    return (npy.exp(-t) - npy.exp(-5*t))*dt

win = npy.hanning(Nt)
x = npy.random.randn(Nt)            # gaussian white noise
x = win*x
r = impulse_response(t)*dt          # evaluate the impulse function
r = win*r
y = npy.convolve(x, r, mode='full')      # convultion of x with r
y = y[:Nt]

# plot t vs x, t vs y and t vs r in three subplots
fig = figure()
ax1 = fig.add_subplot(311)
ax1.plot(t, x)
ax1.set_ylabel('input x')

ax2 = fig.add_subplot(312)
ax2.plot(t, y, label='convolve')
ax2.set_ylabel('output y')


ax3 = fig.add_subplot(313)
ax3.plot(t, r)
ax3.set_ylabel('input response')
ax3.set_xlabel('time (s)')


# compute y via numerical integration of the convolution equation
F = npy.fft.fft(r)
X = npy.fft.fft(x)
Y = F*X
yi = npy.fft.ifft(Y).real
ax2.plot(t, yi, label='fft')
ax2.legend(loc='best')

show()
-------------- next part --------------
A non-text attachment was scrubbed...
Name: convolution_demo.py
Type: text/x-python
Size: 1815 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071025/34876435/attachment.py>

From stefan at sun.ac.za  Thu Oct 25 14:23:29 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Thu, 25 Oct 2007 20:23:29 +0200
Subject: [Numpy-discussion] convolution and wiener khinchin
In-Reply-To: <88e473830710251100v5bec9276ode04965a9260969b@mail.gmail.com>
References: <88e473830710251100v5bec9276ode04965a9260969b@mail.gmail.com>
Message-ID: <20071025182329.GA23232@mentat.za.net>

Hi John

The signals should be zero-padded, otherwise you get circular convolution:

F = npy.fft.fft(r,len(r)+len(x)-1)
X = npy.fft.fft(x,len(r)+len(x)-1)
Y = F*X
yi = npy.fft.ifft(Y)[:len(x)].real

Also take a look at fftconv.

Regards
St?fan

On Thu, Oct 25, 2007 at 01:00:29PM -0500, John Hunter wrote:
> I am working on an example to illustrate convolution in the temporal
> and spectral domains, using the property that a convolution in the
> time domain is a multiplication in the fourier domain.  I am using
> numpy.fft and numpy.convolve to compute the solution two ways, and
> comparing them.  I am getting an error for small times in my fourier
> solution.  At first I thought this was because of edge affects, but I
> see it even when I apply a windowing function.
> 
> Can anyone advise me about what I am doing wrong here?
> 
> """
> In signal processing, the output of a linear system to an arbitrary
> input is given by the convolution of the impulse response function (the
> system response to a Dirac-delta impulse) and the input signal.
> 
> Mathematically:
> 
>   y(t) = \int_0^\t x(\tau)r(t-\tau)d\tau
> 
> 
> where x(t) is the input signal at time t, y(t) is the output, and r(t)
> is the impulse response function.
> 
> In this exercise, we will compute investigate the convolution of a
> white noise process with a double exponential impulse response
> function, and compute the results 3 ways:
> 
>   * using numpy.convolve
> 
>   * in Fourier space using the property that a convolution in the
>     temporal domain is a multiplication in the fourier domain
> """
> 
> import numpy as npy
> import matplotlib.mlab as mlab
> from pylab import figure, show
> 
> # build the time, input, output and response arrays
> dt = 0.01
> t = npy.arange(0.0, 10.0, dt)        # the time vector from 0..5
> Nt = len(t)
> 
> def impulse_response(t):
>     'double exponential response function'
>     return (npy.exp(-t) - npy.exp(-5*t))*dt
> 
> win = npy.hanning(Nt)
> x = npy.random.randn(Nt)            # gaussian white noise
> x = win*x
> r = impulse_response(t)*dt          # evaluate the impulse function
> r = win*r
> y = npy.convolve(x, r, mode='full')      # convultion of x with r
> y = y[:Nt]
> 
> # plot t vs x, t vs y and t vs r in three subplots
> fig = figure()
> ax1 = fig.add_subplot(311)
> ax1.plot(t, x)
> ax1.set_ylabel('input x')
> 
> ax2 = fig.add_subplot(312)
> ax2.plot(t, y, label='convolve')
> ax2.set_ylabel('output y')
> 
> 
> ax3 = fig.add_subplot(313)
> ax3.plot(t, r)
> ax3.set_ylabel('input response')
> ax3.set_xlabel('time (s)')
> 
> 
> # compute y via numerical integration of the convolution equation
> F = npy.fft.fft(r)
> X = npy.fft.fft(x)
> Y = F*X
> yi = npy.fft.ifft(Y).real
> ax2.plot(t, yi, label='fft')
> ax2.legend(loc='best')
> 
> show()


From grh at mur.at  Thu Oct 25 16:25:38 2007
From: grh at mur.at (Georg Holzmann)
Date: Thu, 25 Oct 2007 22:25:38 +0200
Subject: [Numpy-discussion] fortran array storage question
Message-ID: <4720FBC2.6030709@mur.at>

Hallo!

I have the following problem: I get a data array in column major storage 
order and want to use it as numpy array without copying data.

Therefore I do the following (2D example):

   obj = PyArray_FromDimsAndData(2, dim0, PyArray_DOUBLE, (char*)data);
   PyArrayObject *tmp = (PyArrayObject*)obj;
   tmp->flags = NPY_FARRAY;

In python I can see that the flag F_CONTIGUOUS is true, but the python 
numpy array is still displayed in the wrong (C-style) order, same as if 
I don't set the NPY_FARRAY flag.

Do I have to consider something more - or are there any other problems ? 
(sorry - I'm new to numpy ;)

Thanks for any hint,
LG
Georg


From myeates at jpl.nasa.gov  Thu Oct 25 19:16:06 2007
From: myeates at jpl.nasa.gov (Mathew Yeates)
Date: Thu, 25 Oct 2007 16:16:06 -0700
Subject: [Numpy-discussion] vectorizing loops
Message-ID: <472123B6.1050609@jpl.nasa.gov>

Anybody know of any tricks for handling something like

z[0]=1.0
for i in range(100):
    out[i]=func1(z[i])
    z[i+1]=func2(out[i])

??
   



From david at ar.media.kyoto-u.ac.jp  Thu Oct 25 22:01:00 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 26 Oct 2007 11:01:00 +0900
Subject: [Numpy-discussion] C-API for non-contiguous arrays
In-Reply-To: <47207175.3020108@gmx.de>
References: <47207175.3020108@gmx.de>
Message-ID: <47214A5C.40206@ar.media.kyoto-u.ac.jp>

Oliver Kranz wrote:
> Hi,
>
> I am working on a Python extension module using of the NumPy C-API. The 
> extension module is an interface to an image processing and analysis 
> library written in C++. The C++ functions are exported with 
> boos::python. Currently I am implementing the support of 
> three-dimensional data sets which can consume a huge amount of memory. 
> The 3D data is stored in a numpy.ndarray. This array is passed to C++ 
> functions which do the calculations.
>
> In general, multi-dimensional arrays can be organized in memory in four 
> different ways:
> 1. C order contiguous
> 2. Fortran order contiguous
> 3. C order non-contiguous
> 4. Fortran order non-contiguous
>
> Am I right that the NumPy C-API can only distinguish between three ways 
> the array is organized in memory? These are:
> 1. C order contiguous e.g. with PyArray_ISCONTIGUOUS(arr)
> 2. Fortran order contiguous e.g. with PyArray_ISFORTRAN(arr)
> 3. non-contiguous e.g. with !PyArray_ISCONTIGUOUS(arr) &&
> !PyArray_ISFORTRAN(arr)
>
> So there is no way to find out if a non-contiguous array has C order or 
> Fortran order. The same holds for Python code e.g. by use of the flags.
>
> a.flags.contiguous
> a.flags.fortran
>
> This is very important for me because I just want to avoid to copy every 
> non-contiguous array into a contiguous array. This would be very 
> inefficient. But I can't find an other solution than copying the array.
It is inefficient depending on what you mean by inefficient. 
Memory-wise, copying is obviously inefficient. But speed-wise, copying 
the array into a contiguous array in C order is faster in most if not 
all cases, because of memory access times.

You may want to read the following article from Ulrich Drepper on memory 
and cache:

http://lwn.net/Articles/252125/

cheers,

David


From gael.varoquaux at normalesup.org  Fri Oct 26 02:36:19 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 26 Oct 2007 08:36:19 +0200
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <472123B6.1050609@jpl.nasa.gov>
References: <472123B6.1050609@jpl.nasa.gov>
Message-ID: <20071026063619.GC9025@clipper.ens.fr>

On Thu, Oct 25, 2007 at 04:16:06PM -0700, Mathew Yeates wrote:
> Anybody know of any tricks for handling something like

> z[0]=1.0
> for i in range(100):
>     out[i]=func1(z[i])
>     z[i+1]=func2(out[i])

Something like:

z[0] = 1.
out = func1(z)
z[1:] = func2(out[:-1])

HTH,

Ga?l


From david at ar.media.kyoto-u.ac.jp  Fri Oct 26 02:44:39 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 26 Oct 2007 15:44:39 +0900
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <20071026063619.GC9025@clipper.ens.fr>
References: <472123B6.1050609@jpl.nasa.gov>
	<20071026063619.GC9025@clipper.ens.fr>
Message-ID: <47218CD7.2000501@ar.media.kyoto-u.ac.jp>

Gael Varoquaux wrote:
> On Thu, Oct 25, 2007 at 04:16:06PM -0700, Mathew Yeates wrote:
>> Anybody know of any tricks for handling something like
>
>> z[0]=1.0
>> for i in range(100):
>>     out[i]=func1(z[i])
>>     z[i+1]=func2(out[i])
>
> Something like:
>
> z[0] = 1.
> out = func1(z)
> z[1:] = func2(out[:-1])
This only works if func1 has no side effect on its argument. The problem 
boils down to whether the above algorithm is recursive or not (does 
z[i+1] needs z[i]). If func1 has no side effect, then your solution is 
fine (but then writing the thing as out[i] = func1(z[i]); z[i+1] = 
func2(z[i]) is not really intuitive). If func1 has side effect, then you 
cannot vectorize easily.

cheers,

David

P.S: IMHO, this is one of the main limitation of numpy (or any language 
using arrays for speed; and this is really difficult to optimize: you 
need compilation, JIT or similar to solve those efficiently).


From robert.kern at gmail.com  Fri Oct 26 02:56:26 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 26 Oct 2007 01:56:26 -0500
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <20071026063619.GC9025@clipper.ens.fr>
References: <472123B6.1050609@jpl.nasa.gov>
	<20071026063619.GC9025@clipper.ens.fr>
Message-ID: <47218F9A.5070300@gmail.com>

Gael Varoquaux wrote:
> On Thu, Oct 25, 2007 at 04:16:06PM -0700, Mathew Yeates wrote:
>> Anybody know of any tricks for handling something like
> 
>> z[0]=1.0
>> for i in range(100):
>>     out[i]=func1(z[i])
>>     z[i+1]=func2(out[i])
> 
> Something like:
> 
> z[0] = 1.
> out = func1(z)
> z[1:] = func2(out[:-1])

No, that doesn't work. The way you have it, for each i>0,

  z[i] = func2(func1(0))

What Matthew wants is this

  z[0] = 1.0
  z[1] = func2(func1(1.0))
  z[2] = func2(func1(func2(func1(1.0))))
  ...

I'm afraid that there is no fast, elegant way to do this. If you could turn
func2(func1(x)) into a binary ufunc f(x, y) with an ignored y, then you could
call f.accumulate(z). However, if that ufunc is not implemented in C, there's
not much point.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From gael.varoquaux at normalesup.org  Fri Oct 26 03:31:44 2007
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 26 Oct 2007 09:31:44 +0200
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <47218F9A.5070300@gmail.com>
References: <472123B6.1050609@jpl.nasa.gov>
	<20071026063619.GC9025@clipper.ens.fr> <47218F9A.5070300@gmail.com>
Message-ID: <20071026073144.GE9025@clipper.ens.fr>

On Fri, Oct 26, 2007 at 01:56:26AM -0500, Robert Kern wrote:
> Gael Varoquaux wrote:
> > On Thu, Oct 25, 2007 at 04:16:06PM -0700, Mathew Yeates wrote:
> >> Anybody know of any tricks for handling something like

> >> z[0]=1.0
> >> for i in range(100):
> >>     out[i]=func1(z[i])
> >>     z[i+1]=func2(out[i])

> > Something like:

> > z[0] = 1.
> > out = func1(z)
> > z[1:] = func2(out[:-1])

> No, that doesn't work. The way you have it, for each i>0,

>   z[i] = func2(func1(0))

> What Matthew wants is this

>   z[0] = 1.0
>   z[1] = func2(func1(1.0))
>   z[2] = func2(func1(func2(func1(1.0))))

Yes, obviously. Sorry for being dense.

I can't see a fast way of doing this appart in Python.

Ga?l


From vputz at nyx.net  Fri Oct 26 04:31:00 2007
From: vputz at nyx.net (Victor Putz)
Date: Fri, 26 Oct 2007 02:31:00 -0600 (MDT)
Subject: [Numpy-discussion] Best way to handle "wrapped" array accesses?
Message-ID: <Pine.GSO.4.64.0710260224001.8689@nyx3.nyx.net>


I've seen a few references on this, but hadn't found a proper solution... 
I'm doing Lattice-Boltzmann simulations with periodic boundary conditions,
which always necessarily involve either padding the edges and doing 
additional steps, or making a "wrapped" array (for example, if I have an 
array that's N rows by M columns, and cells are moving right, I then make 
a "wrapped" array a la "D = hstack( A[:,0:-1], A[:,-1:] )" (creating a 
temporary) and adding them.

I'd like to avoid the temporary and still make the operation as fast as 
possible.  A really elegant solution would be a "wrapped array" subclass 
that would automatically handle the indexing (C = A[:,1:M+1] + B), but I'm 
not certain how friendly the array class is with subclassing.

What's the best way to handle this?  Am I pretty much stuck with 
temporaries or hacked functions ("add_wrapped(A, offset, B, C)")?

(It's probably a moot point in the long term, as once things get big I 
will have to use the cluster, which is probably not SciPy-friendly and 
it'll likely be a bit before I can convince the sys admin to let me play 
outside the "installed software" boundaries.  But now I just want to 
know).

Thanks,
-->VPutz



From robert.crida at ska.ac.za  Fri Oct 26 05:26:12 2007
From: robert.crida at ska.ac.za (Robert Crida)
Date: Fri, 26 Oct 2007 11:26:12 +0200
Subject: [Numpy-discussion] Memory leak in ndarray
Message-ID: <c0935a560710260226i6c12d955g3c087ac4638264d4@mail.gmail.com>

Hi all

I seem to have tracked down a memory leak in the string conversion mechanism
of numpy. It is demonstrated using the following code:

import numpy as np

a = np.array([1.0, 2.0, 3.0])
while True:
    b = str(a)

What happens above is that is repeatedly converted to a string. The process
size grow quite rapidly.

Has anyone else come across this? Where do I look to try to correct it?

Thanks
Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071026/22e136bc/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Fri Oct 26 05:22:52 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 26 Oct 2007 18:22:52 +0900
Subject: [Numpy-discussion] Memory leak in ndarray
In-Reply-To: <c0935a560710260226i6c12d955g3c087ac4638264d4@mail.gmail.com>
References: <c0935a560710260226i6c12d955g3c087ac4638264d4@mail.gmail.com>
Message-ID: <4721B1EC.1060906@ar.media.kyoto-u.ac.jp>

Robert Crida wrote:
> Hi all
>
> I seem to have tracked down a memory leak in the string conversion 
> mechanism of numpy. It is demonstrated using the following code:
>
> import numpy as np
>
> a = np.array([1.0, 2.0, 3.0])
> while True:
>     b = str(a)
>
> What happens above is that is repeatedly converted to a string. The 
> process size grow quite rapidly.
>
> Has anyone else come across this? Where do I look to try to correct it?
Hi Robert,

I cannot reproduce this on my machine. Could you give more details 
(which numpy version, etc...) ?

cheers,

David


From robert.crida at ska.ac.za  Fri Oct 26 05:39:51 2007
From: robert.crida at ska.ac.za (Robert Crida)
Date: Fri, 26 Oct 2007 11:39:51 +0200
Subject: [Numpy-discussion] Memory leak in ndarray, more info
Message-ID: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>

Hi all

I recently posted about a memory leak in numpy and failed to mention the
version. The leak manifests itself in numpy-1.0.3.1 but is not present in
numpy-1.0.2

The following code reproduces the bug:

import numpy as np

a = np.array([1.0, 2.0, 3.0])
while True:
    b = str(a)

What happens above is that is repeatedly converted to a string. The process
size grow quite rapidly.

Has anyone else come across this? Where do I look to try to correct it?

Thanks
Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071026/3f02303a/attachment.html>

From matthieu.brucher at gmail.com  Fri Oct 26 06:46:20 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 26 Oct 2007 12:46:20 +0200
Subject: [Numpy-discussion] Memory leak in ndarray, more info
In-Reply-To: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>
References: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>
Message-ID: <e76aa17f0710260346h118db4aayd80253dd4d596035@mail.gmail.com>

Which version Python are you using ?

Matthieu

2007/10/26, Robert Crida <robert.crida at ska.ac.za>:
>
> Hi all
>
> I recently posted about a memory leak in numpy and failed to mention the
> version. The leak manifests itself in numpy-1.0.3.1 but is not present in
> numpy-1.0.2
>
> The following code reproduces the bug:
>
> import numpy as np
>
> a = np.array([1.0, 2.0, 3.0])
> while True:
>     b = str(a)
>
> What happens above is that is repeatedly converted to a string. The
> process size grow quite rapidly.
>
> Has anyone else come across this? Where do I look to try to correct it?
>
> Thanks
>  Robert
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>


-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071026/73bb1154/attachment.html>

From matthew.brett at gmail.com  Fri Oct 26 06:52:40 2007
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 26 Oct 2007 11:52:40 +0100
Subject: [Numpy-discussion] Memory leak in ndarray
In-Reply-To: <c0935a560710260226i6c12d955g3c087ac4638264d4@mail.gmail.com>
References: <c0935a560710260226i6c12d955g3c087ac4638264d4@mail.gmail.com>
Message-ID: <1e2af89e0710260352w28d574b8s458313a2170db310@mail.gmail.com>

Hi,

> I seem to have tracked down a memory leak in the string conversion mechanism
> of numpy. It is demonstrated using the following code:
>
> import numpy as np
>
> a = np.array([1.0, 2.0, 3.0])
> while True:
>     b = str(a)

Would you not expect python rather than numpy to be dealing with the
memory here though?

Matthew


From robert.crida at ska.ac.za  Fri Oct 26 07:14:12 2007
From: robert.crida at ska.ac.za (Robert Crida)
Date: Fri, 26 Oct 2007 13:14:12 +0200
Subject: [Numpy-discussion] Memory leak in ndarray
In-Reply-To: <1e2af89e0710260352w28d574b8s458313a2170db310@mail.gmail.com>
References: <c0935a560710260226i6c12d955g3c087ac4638264d4@mail.gmail.com>
	<1e2af89e0710260352w28d574b8s458313a2170db310@mail.gmail.com>
Message-ID: <c0935a560710260414i7fbfbf30r57ff8a251abee6dc@mail.gmail.com>

Hi

I don't think it is a python issue because if you change the line b = str(a)
to just read
str(a)
then the problem still occurs.

Also, if you change a to be a list instead of ndarray then the problem does
not occur.

Cheers
Robert


On 10/26/07, Matthew Brett <matthew.brett at gmail.com> wrote:
>
> Hi,
>
> > I seem to have tracked down a memory leak in the string conversion
> mechanism
> > of numpy. It is demonstrated using the following code:
> >
> > import numpy as np
> >
> > a = np.array([1.0, 2.0, 3.0])
> > while True:
> >     b = str(a)
>
> Would you not expect python rather than numpy to be dealing with the
> memory here though?
>
> Matthew
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071026/25c86052/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Fri Oct 26 07:15:05 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 26 Oct 2007 20:15:05 +0900
Subject: [Numpy-discussion] Memory leak in ndarray
In-Reply-To: <c0935a560710260414i7fbfbf30r57ff8a251abee6dc@mail.gmail.com>
References: <c0935a560710260226i6c12d955g3c087ac4638264d4@mail.gmail.com>	<1e2af89e0710260352w28d574b8s458313a2170db310@mail.gmail.com>
	<c0935a560710260414i7fbfbf30r57ff8a251abee6dc@mail.gmail.com>
Message-ID: <4721CC39.5090705@ar.media.kyoto-u.ac.jp>

Robert Crida wrote:
> Hi
>
> I don't think it is a python issue because if you change the line b = 
> str(a) to just read
> str(a)
> then the problem still occurs.
>
> Also, if you change a to be a list instead of ndarray then the problem 
> does not occur. 
How do you know there is a memory leak ?

cheers,

David


From haase at msg.ucsf.edu  Fri Oct 26 08:06:11 2007
From: haase at msg.ucsf.edu (Sebastian Haase)
Date: Fri, 26 Oct 2007 14:06:11 +0200
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <47218CD7.2000501@ar.media.kyoto-u.ac.jp>
References: <472123B6.1050609@jpl.nasa.gov>
	<20071026063619.GC9025@clipper.ens.fr>
	<47218CD7.2000501@ar.media.kyoto-u.ac.jp>
Message-ID: <bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>

On 10/26/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> P.S: IMHO, this is one of the main limitation of numpy (or any language
> using arrays for speed; and this is really difficult to optimize: you
> need compilation, JIT or similar to solve those efficiently).

This is where the scipy - sandbox numexpr  project comes in
- if I'm not misaken ....

http://www.scipy.org/SciPyPackages/NumExpr

Description
The scipy.sandbox.numexpr package supplies routines for the fast
evaluation of array expressions elementwise by using a vector-based
virtual machine. It's comparable to scipy.weave.blitz (in Weave), but
doesn't require a separate compile step of C or C++ code.


I hope that more noise around this  will result  in more interest and
subsequentially result in more support.
I think numexpr  might be one of the most powerful ideas in numpy /
scipy  "recently".
Did you know about numexpr - David ?

Cheers,
Sebastian Haase


From david at ar.media.kyoto-u.ac.jp  Fri Oct 26 08:20:34 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 26 Oct 2007 21:20:34 +0900
Subject: [Numpy-discussion] [Announce] numpy.scons , ALPHA version
Message-ID: <4721DB92.4000503@ar.media.kyoto-u.ac.jp>

Hi there,

    I've finally managed to implement most of the things I wanted for 
numpy.scons, hence a first alpha. Compared to the 2d milestone from a 
few days ago, a few optimized libraries are supported (ATLAS, Sun 
sunperf, Apple Accelerate and vecLib, Intel MKL).

Who
===

Outside people interested in numpy.scons, people who have trouble 
building numpy may want to try this. In particular, using MKL or sunperf 
should be easier (for sunperf, it should work out of the box if sun 
compilers are used). Also, compiling with intel compilers on linux 
should be easier.

DO NOT USE THIS FOR PRODUCTION USE ! I am interested in success (that is 
build and all the test pass) as well as failure.

Numpy developers: I would be interested in comments about the code:
    - the package are built with the setupscons.py/SConstruct
    - most of the new code is in numpy/distutils/scons

How to test
===========

Grab the sources:

svn co http://svn.scipy.org/svn/numpy/branches/numpy.scons

And then just do: python setup.py scons; python setup.py install. If you 
have multiple CPU/Cores, you may want to use the parallel build options:

python setup.py scons --jobs=N with N an integer (number of CPU/CORES + 
1 is a rule of thumb).

For non default compilers, use the --compiler and --fcompiler options 
(distutils names).

For the MKL, you should add a mkl option in the site.cfg. For ATLAS or 
sunperf, if it is not in standard places, it won't work (yet).

What works:
===========

The tested platforms have been tested:
    - linux (ubuntu x86) with gcc, g77 and ATLAS.
    - linux (x86 and x64) with gcc, without BLAS or LAPACK.
    - linux (ubuntu x86) with gcc, g77, and mkl (9.1.023).
    - linux (ubuntu x86) with intel compilers.
    - windows (32 bits) with mingw.
    - windows (32 bits) with VS 2003.
    - solaris (solaris studio express 12) with sunperf on x86.
    - Mac Os X Intel with Accelerate framework.
    - People reported basic success on IRIX, too.

What does NOT work
==================

One important target missing is windows 64, but this should not be too 
difficult to solve.

There are still many corner cases not yet solved (in particular some 
windows things, most libraries path cannot yet be overriden in 
site.cfg); also, I do not attempt to be 100 % backward compatible with 
the current build system (that for a given environment, you won't 
necessarily have the same build options). Some things are not yet 
supported either (overriding libraries with env variables, for example, 
is not supported).

There is no real doc yet (I will do this once the basic public API is 
stabilized).

Details
=======

Library checkers
----------------

The main difference compared to the second milestone is the support for 
libraries checkers (think BLAS, Atlas, etc...). There is now support to 
check for different libraries, with different compilers. This 
effectively replaces numpy.distutils.system_info. The way I've 
implemented those checks are fundamentally different compared to the 
distutils way: instead of looking for files, I look for capabilities at 
runtime.

For example, for mkl, instead of looking for libmkl.so, etc... I try to 
compile, link and run a small mkl program: if any of the step failed, 
then the mkl is not used. This means that I can detect config errors 
much sooner; also, since I use rpath, there is no need to set things 
like LD_LIBRARY_PATH and so on. If the path is set right in site.cfg, 
then everything is taken care of.

Also, as before, a general library checker for new libraries is 
available: look at basic examples in numpy/scons_fake/. In particular, 
there is a NumpyCheckLib which can be used to check a library with some 
symbols, whose paths can be overriden with a site.cfg.

Fortran support
---------------

The other big work has been on fortran support. Here too, the way I 
implemented it is different than numpy.distutils. Instead of harcoding 
flags depending on versions, I use code snipets, to get link options, 
mangling, and so on. For example, if you call the following in a Sconscript:

config.CheckF77Mangling()

This will try to get the mangling of the F77 compiler, and if 
sucessfull, will set a python function mangle, which returns the fortran 
name from the 'common' name. For example, for g77, mangle('dgemm') will 
returns dgemm_. Since the mangling is found at runtime, this can be used 
for not yet known compilers.

Next
====

This was quite a lot of work, and I wanted to get something working as 
fast as possible. As such, the code is not always clean. Once I am 
satisfied with the global 'feel' of the new build, I will sanitize the 
API and implement the doc. Still, I believe the code to be much easier, 
simpler and robust than distutils (everything, from fortran support to 
libraries checkers is less than 2000 lines of python code, including 
tests; of course, this is largely thanks to scons).

The thing I am the less sure about is how to combine checkers in 
meta-checkers: for example, I have checker for mkl, ATLAS, sunperf, and 
we sometimes want a BLAS checker, which would check all those as long as 
nothing is found. API-wise, I am not yet satisfied with the current way.

I also want to play nice with f2py and scipy (totally untested for now).


From grh at mur.at  Fri Oct 26 08:30:57 2007
From: grh at mur.at (Georg Holzmann)
Date: Fri, 26 Oct 2007 14:30:57 +0200
Subject: [Numpy-discussion] fortran array storage question
In-Reply-To: <4720FBC2.6030709@mur.at>
References: <4720FBC2.6030709@mur.at>
Message-ID: <4721DE01.2010505@mur.at>

Hallo!

I found now a way to get the data:

> Therefore I do the following (2D example):
> 
>    obj = PyArray_FromDimsAndData(2, dim0, PyArray_DOUBLE, (char*)data);
>    PyArrayObject *tmp = (PyArrayObject*)obj;
>    tmp->flags = NPY_FARRAY;

if in that example I also change the strides:

   int s = tmp->strides[1];
   tmp->strides[0] = s;
   tmp->strides[1] = s * dim0[0];

Then I get in python the fortran-style array in right order.

However, I don't know if this is the usual way or if it has a 
performance overhead ...

Thanks,
LG
Georg


From david at ar.media.kyoto-u.ac.jp  Fri Oct 26 08:25:44 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Fri, 26 Oct 2007 21:25:44 +0900
Subject: [Numpy-discussion] fortran array storage question
In-Reply-To: <4721DE01.2010505@mur.at>
References: <4720FBC2.6030709@mur.at> <4721DE01.2010505@mur.at>
Message-ID: <4721DCC8.2010705@ar.media.kyoto-u.ac.jp>

Georg Holzmann wrote:
> Hallo!
>
> I found now a way to get the data:
>
>   
>> Therefore I do the following (2D example):
>>
>>    obj = PyArray_FromDimsAndData(2, dim0, PyArray_DOUBLE, (char*)data);
>>    PyArrayObject *tmp = (PyArrayObject*)obj;
>>    tmp->flags = NPY_FARRAY;
>>     
>
> if in that example I also change the strides:
>
>    int s = tmp->strides[1];
>    tmp->strides[0] = s;
>    tmp->strides[1] = s * dim0[0];
>
> Then I get in python the fortran-style array in right order.
>
> However, I don't know if this is the usual way or if it has a 
> performance overhead ...
>
>   
This depends on what you are trying to do, but generally, I find that if 
you can afford it memory-wise, it is much faster to just get a C 
contiguous array if you treat your C array element per element. If you 
don't access element per element, then it can become much more 
difficult, of course (specially if you have to use several times the 
same parts of the memory).

cheers,

David


From grh at mur.at  Fri Oct 26 08:41:39 2007
From: grh at mur.at (Georg Holzmann)
Date: Fri, 26 Oct 2007 14:41:39 +0200
Subject: [Numpy-discussion] fortran array storage question
In-Reply-To: <4721DCC8.2010705@ar.media.kyoto-u.ac.jp>
References: <4720FBC2.6030709@mur.at> <4721DE01.2010505@mur.at>
	<4721DCC8.2010705@ar.media.kyoto-u.ac.jp>
Message-ID: <4721E083.5070205@mur.at>

Hallo!

> This depends on what you are trying to do, but generally, I find that if 
> you can afford it memory-wise, it is much faster to just get a C 
> contiguous array if you treat your C array element per element. If you 

Yes, but the problem is that this data is very big (up to my memory 
limit) and in python I only want to read it for debugging purpose.

And if I want to make a copy to a C contiguous array I can always do it 
like:
    my_cstyle_array = my_method().copy()


Thanks,
LG
Georg


From robert.crida at ska.ac.za  Fri Oct 26 09:27:56 2007
From: robert.crida at ska.ac.za (Robert Crida)
Date: Fri, 26 Oct 2007 15:27:56 +0200
Subject: [Numpy-discussion] Memory leak in ndarray
In-Reply-To: <4721CC39.5090705@ar.media.kyoto-u.ac.jp>
References: <c0935a560710260226i6c12d955g3c087ac4638264d4@mail.gmail.com>
	<1e2af89e0710260352w28d574b8s458313a2170db310@mail.gmail.com>
	<c0935a560710260414i7fbfbf30r57ff8a251abee6dc@mail.gmail.com>
	<4721CC39.5090705@ar.media.kyoto-u.ac.jp>
Message-ID: <c0935a560710260627t79681103lfc09d91d3fe83be5@mail.gmail.com>

Hi again

I watch the VmSize of the process using eg top or ps

If a is a list then it remains constant. If a is an ndarray as shown in the
example, then the VmSize grows quite rapidly.

Cheers
Robert

On 10/26/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>
> Robert Crida wrote:
> > Hi
> >
> > I don't think it is a python issue because if you change the line b =
> > str(a) to just read
> > str(a)
> > then the problem still occurs.
> >
> > Also, if you change a to be a list instead of ndarray then the problem
> > does not occur.
> How do you know there is a memory leak ?
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071026/aee55619/attachment.html>

From tim.hochberg at ieee.org  Fri Oct 26 09:30:24 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Fri, 26 Oct 2007 06:30:24 -0700
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>
References: <472123B6.1050609@jpl.nasa.gov>
	<20071026063619.GC9025@clipper.ens.fr>
	<47218CD7.2000501@ar.media.kyoto-u.ac.jp>
	<bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>
Message-ID: <e4412d6b0710260630m167f36a3he2fc1686fb3b39df@mail.gmail.com>

On 10/26/07, Sebastian Haase <haase at msg.ucsf.edu> wrote:
>
> On 10/26/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> > P.S: IMHO, this is one of the main limitation of numpy (or any language
> > using arrays for speed; and this is really difficult to optimize: you
> > need compilation, JIT or similar to solve those efficiently).
>
> This is where the scipy - sandbox numexpr  project comes in
> - if I'm not misaken ....
>
> http://www.scipy.org/SciPyPackages/NumExpr
>
> Description
> The scipy.sandbox.numexpr package supplies routines for the fast
> evaluation of array expressions elementwise by using a vector-based
> virtual machine. It's comparable to scipy.weave.blitz (in Weave), but
> doesn't require a separate compile step of C or C++ code.
>
>
> I hope that more noise around this  will result  in more interest and
> subsequentially result in more support.
> I think numexpr  might be one of the most powerful ideas in numpy /
> scipy  "recently".
> Did you know about numexpr - David ?


Sadly, I don't think numexpr will help here; It basically handles the same
cases as numpy; only faster because it can avoid a lot of temporaries. I
think he's going to need Psyco,  Pyrex, Weave or similar.




-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071026/a40f086e/attachment.html>

From robince at gmail.com  Fri Oct 26 09:33:42 2007
From: robince at gmail.com (Robin)
Date: Fri, 26 Oct 2007 14:33:42 +0100
Subject: [Numpy-discussion] Memory leak in ndarray
In-Reply-To: <c0935a560710260627t79681103lfc09d91d3fe83be5@mail.gmail.com>
References: <c0935a560710260226i6c12d955g3c087ac4638264d4@mail.gmail.com>
	<1e2af89e0710260352w28d574b8s458313a2170db310@mail.gmail.com>
	<c0935a560710260414i7fbfbf30r57ff8a251abee6dc@mail.gmail.com>
	<4721CC39.5090705@ar.media.kyoto-u.ac.jp>
	<c0935a560710260627t79681103lfc09d91d3fe83be5@mail.gmail.com>
Message-ID: <b14b627d0710260633u1de1fdaajb55804f4a6de8faf@mail.gmail.com>

I can confirm the same behaviour with numpy '1.0.4.dev4271' on OS X
10.4with python
2.5.1 (installer from python.org).

For me the memory used by the python process grows at about 1MB/sec. The
memory isn't released when the loop is canceled.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071026/058fe6f1/attachment.html>

From cournape at gmail.com  Fri Oct 26 09:48:57 2007
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 26 Oct 2007 22:48:57 +0900
Subject: [Numpy-discussion] Memory leak in ndarray
In-Reply-To: <c0935a560710260627t79681103lfc09d91d3fe83be5@mail.gmail.com>
References: <c0935a560710260226i6c12d955g3c087ac4638264d4@mail.gmail.com>
	<1e2af89e0710260352w28d574b8s458313a2170db310@mail.gmail.com>
	<c0935a560710260414i7fbfbf30r57ff8a251abee6dc@mail.gmail.com>
	<4721CC39.5090705@ar.media.kyoto-u.ac.jp>
	<c0935a560710260627t79681103lfc09d91d3fe83be5@mail.gmail.com>
Message-ID: <5b8d13220710260648w35833c44m8bff6c65310f37a6@mail.gmail.com>

On 10/26/07, Robert Crida <robert.crida at ska.ac.za> wrote:
> Hi again
>
> I watch the VmSize of the process using eg top or ps
>
> If a is a list then it remains constant. If a is an ndarray as shown in the
> example, then the VmSize grows quite rapidly.
>
Actually, I did a typo while copying your example. I can confirm the
memory leak (happen in PyObject_Malloc).

cheers,

David


From cournape at gmail.com  Fri Oct 26 10:05:08 2007
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 26 Oct 2007 23:05:08 +0900
Subject: [Numpy-discussion] Memory leak in ndarray
In-Reply-To: <5b8d13220710260648w35833c44m8bff6c65310f37a6@mail.gmail.com>
References: <c0935a560710260226i6c12d955g3c087ac4638264d4@mail.gmail.com>
	<1e2af89e0710260352w28d574b8s458313a2170db310@mail.gmail.com>
	<c0935a560710260414i7fbfbf30r57ff8a251abee6dc@mail.gmail.com>
	<4721CC39.5090705@ar.media.kyoto-u.ac.jp>
	<c0935a560710260627t79681103lfc09d91d3fe83be5@mail.gmail.com>
	<5b8d13220710260648w35833c44m8bff6c65310f37a6@mail.gmail.com>
Message-ID: <5b8d13220710260705k9e8210eo1f77e605ccb1d7d9@mail.gmail.com>

On 10/26/07, David Cournapeau <cournape at gmail.com> wrote:
> On 10/26/07, Robert Crida <robert.crida at ska.ac.za> wrote:
> > Hi again
> >
> > I watch the VmSize of the process using eg top or ps
> >
> > If a is a list then it remains constant. If a is an ndarray as shown in the
> > example, then the VmSize grows quite rapidly.
> >
> Actually, I did a typo while copying your example. I can confirm the
> memory leak (happen in PyObject_Malloc).
>
The problem *may* be within the ufunc machinery. I am still
investigating, but since we can reproduce the problem with such a
simple code, it should not be difficult to track down the problem with
high level tools such as valgrind.

David


From matthieu.brucher at gmail.com  Fri Oct 26 10:37:02 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Fri, 26 Oct 2007 16:37:02 +0200
Subject: [Numpy-discussion] [Announce] numpy.scons , ALPHA version
In-Reply-To: <4721DB92.4000503@ar.media.kyoto-u.ac.jp>
References: <4721DB92.4000503@ar.media.kyoto-u.ac.jp>
Message-ID: <e76aa17f0710260737m75778defi29f706d87501a5e0@mail.gmail.com>

>
> What does NOT work
> ==================
>
> One important target missing is windows 64, but this should not be too
> difficult to solve.
>
> There are still many corner cases not yet solved (in particular some
> windows things, most libraries path cannot yet be overriden in
> site.cfg); also, I do not attempt to be 100 % backward compatible with
> the current build system (that for a given environment, you won't
> necessarily have the same build options). Some things are not yet
> supported either (overriding libraries with env variables, for example,
> is not supported).


I would add that MSVC 2005 (windows32) does not work either as it needs a
modification in the SharedLibrary builder (which I can provide if you can
check the version of MSVC).
This would enable people that cannot compile extensions with distutils to
compile them with a compiler they can freely download on the net.

-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071026/d2a4b2ac/attachment.html>

From peridot.faceted at gmail.com  Fri Oct 26 11:27:31 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 26 Oct 2007 11:27:31 -0400
Subject: [Numpy-discussion] fortran array storage question
In-Reply-To: <4721DE01.2010505@mur.at>
References: <4720FBC2.6030709@mur.at> <4721DE01.2010505@mur.at>
Message-ID: <ce557a360710260827t6db077x66e18d0b46509068@mail.gmail.com>

On 26/10/2007, Georg Holzmann <grh at mur.at> wrote:

> if in that example I also change the strides:
>
>    int s = tmp->strides[1];
>    tmp->strides[0] = s;
>    tmp->strides[1] = s * dim0[0];
>
> Then I get in python the fortran-style array in right order.

This is the usual way. More or less, at least. numpy is designed from
the start to handle arrays with arbitrary striding; this is how slices
are implemented, for example. There will be no major performance hit
from numpy code itself. The actual organization of data in memory will
of course affect the speed at which your code runs. The flags, as you
discovered, are just a performance optimization, so that code that
needs arrays organized as C- or FORTRAN-standard doesn't need to check
the strides every time.

I don't think numpy's loops - for example in ones((100,100))+eye(100)
- are smart about doing operations in an order that makes
cache-coherent use of memory. The important exception is the loops
that use ATLAS, which I think is mostly the dot() function.

Anne


From o.kranz at gmx.de  Fri Oct 26 11:38:47 2007
From: o.kranz at gmx.de (Oliver Kranz)
Date: Fri, 26 Oct 2007 17:38:47 +0200
Subject: [Numpy-discussion] C-API for non-contiguous arrays
In-Reply-To: <47214A5C.40206@ar.media.kyoto-u.ac.jp>
References: <47207175.3020108@gmx.de> <47214A5C.40206@ar.media.kyoto-u.ac.jp>
Message-ID: <47220A07.8030207@gmx.de>

David Cournapeau wrote:
> Oliver Kranz wrote:
>> Hi,
>>
>> I am working on a Python extension module using of the NumPy C-API. The 
>> extension module is an interface to an image processing and analysis 
>> library written in C++. The C++ functions are exported with 
>> boos::python. Currently I am implementing the support of 
>> three-dimensional data sets which can consume a huge amount of memory. 
>> The 3D data is stored in a numpy.ndarray. This array is passed to C++ 
>> functions which do the calculations.
>>
>> In general, multi-dimensional arrays can be organized in memory in four 
>> different ways:
>> 1. C order contiguous
>> 2. Fortran order contiguous
>> 3. C order non-contiguous
>> 4. Fortran order non-contiguous
>>
>> Am I right that the NumPy C-API can only distinguish between three ways 
>> the array is organized in memory? These are:
>> 1. C order contiguous e.g. with PyArray_ISCONTIGUOUS(arr)
>> 2. Fortran order contiguous e.g. with PyArray_ISFORTRAN(arr)
>> 3. non-contiguous e.g. with !PyArray_ISCONTIGUOUS(arr) &&
>> !PyArray_ISFORTRAN(arr)
>>
>> So there is no way to find out if a non-contiguous array has C order or 
>> Fortran order. The same holds for Python code e.g. by use of the flags.
>>
>> a.flags.contiguous
>> a.flags.fortran
>>
>> This is very important for me because I just want to avoid to copy every 
>> non-contiguous array into a contiguous array. This would be very 
>> inefficient. But I can't find an other solution than copying the array.
> It is inefficient depending on what you mean by inefficient. 
> Memory-wise, copying is obviously inefficient. But speed-wise, copying 
> the array into a contiguous array in C order is faster in most if not 
> all cases, because of memory access times.
> 
> You may want to read the following article from Ulrich Drepper on memory 
> and cache:
> 
> http://lwn.net/Articles/252125/

That's an interesting note. We already thought about this. At the 
moment, we decided to consequently avoid copying in our apecial case. 
It's not unusal to work with data sets consuming about 1 GB of memory. 
In the case of arrays not being in contiguous C order we have to live 
with the inefficiency in speed.

Cheers,
Oliver


From oliphant at enthought.com  Fri Oct 26 11:38:59 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 26 Oct 2007 10:38:59 -0500
Subject: [Numpy-discussion] fortran array storage question
In-Reply-To: <ce557a360710260827t6db077x66e18d0b46509068@mail.gmail.com>
References: <4720FBC2.6030709@mur.at> <4721DE01.2010505@mur.at>
	<ce557a360710260827t6db077x66e18d0b46509068@mail.gmail.com>
Message-ID: <47220A13.3010905@enthought.com>

Anne Archibald wrote:
> On 26/10/2007, Georg Holzmann <grh at mur.at> wrote:
>
>   
>> if in that example I also change the strides:
>>
>>    int s = tmp->strides[1];
>>    tmp->strides[0] = s;
>>    tmp->strides[1] = s * dim0[0];
>>
>> Then I get in python the fortran-style array in right order.
>>     
>
> This is the usual way. More or less, at least. numpy is designed from
> the start to handle arrays with arbitrary striding; this is how slices
> are implemented, for example. There will be no major performance hit
> from numpy code itself. The actual organization of data in memory will
> of course affect the speed at which your code runs. The flags, as you
> discovered, are just a performance optimization, so that code that
> needs arrays organized as C- or FORTRAN-standard doesn't need to check
> the strides every time.
>
> I don't think numpy's loops - for example in ones((100,100))+eye(100)
> - are smart about doing operations in an order that makes
> cache-coherent use of memory. The important exception is the loops
> that use ATLAS, which I think is mostly the dot() function.
>
>   
There is an optimization where-in the inner-loops are done over the 
dimension with the smallest stride. 

What other cache-coherent optimizations do you recommend?

-Travis



From dmitrey.kroshko at scipy.org  Fri Oct 26 14:30:55 2007
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Fri, 26 Oct 2007 21:30:55 +0300
Subject: [Numpy-discussion] Request for advice: project to get NumPy
 working in IronPython
In-Reply-To: <4713C334.2000407@enthought.com>
References: <4713AFAF.10201@resolversystems.com>
	<4713C334.2000407@enthought.com>
Message-ID: <4722325F.7070302@scipy.org>

Travis E. Oliphant wrote:
> Giles Thomas wrote:
>   
>> Hi,
>>
>> At Resolver Systems, we have a product that is written in IronPython - 
>> the .NET Python implementation - and allows users to use that language 
>> to script a spreadsheet-like interface.  Because they're using 
>> IronPython, they can access their existing .NET objects and libraries, 
>> which has worked out really well for us and for them.  But there's an 
>> increasing number of users who would like access to CPython C 
>> extensions - in particular, NumPy.
>>     
> An IronPython compatible version of NumPy would be great.    Of course 
> it could be done by using C# to write NumPy, but I'm not sure that this 
> would really be any less work than creating a "glue" layer that allowed 
> most (or all) C-Python extensions to work with IronPython.
>
>   
So can anyone inform will IronPython have bridge to NumPy or something else?
And will it be available in nearest future or some weeks/months/years 
are required?
(I'm interested in scikits.openopt being available for IronPython as 
well, and it requires numpy)
And what about pylab for IronPython? Is it work already now, or will be 
available in nearest future, or the situation is undefined?

Regards, D.


From charlesr.harris at gmail.com  Fri Oct 26 14:37:51 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 26 Oct 2007 12:37:51 -0600
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <e4412d6b0710260630m167f36a3he2fc1686fb3b39df@mail.gmail.com>
References: <472123B6.1050609@jpl.nasa.gov>
	<20071026063619.GC9025@clipper.ens.fr>
	<47218CD7.2000501@ar.media.kyoto-u.ac.jp>
	<bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>
	<e4412d6b0710260630m167f36a3he2fc1686fb3b39df@mail.gmail.com>
Message-ID: <e06186140710261137w18ff8525w1de6255c7735cbb4@mail.gmail.com>

Hi,

On 10/26/07, Timothy Hochberg <tim.hochberg at ieee.org> wrote:
>
>
> On 10/26/07, Sebastian Haase <haase at msg.ucsf.edu> wrote:
> > On 10/26/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
> > > P.S: IMHO, this is one of the main limitation of numpy (or any language
> > > using arrays for speed; and this is really difficult to optimize: you
> > > need compilation, JIT or similar to solve those efficiently).
> >
> > This is where the scipy - sandbox numexpr  project comes in
> > - if I'm not misaken ....
> >
> > http://www.scipy.org/SciPyPackages/NumExpr
> >
> > Description
> > The scipy.sandbox.numexpr package supplies routines for the fast
> > evaluation of array expressions elementwise by using a vector-based
> > virtual machine. It's comparable to scipy.weave.blitz (in Weave), but
> > doesn't require a separate compile step of C or C++ code.
> >
> >
> > I hope that more noise around this  will result  in more interest and
> > subsequentially result in more support.
> > I think numexpr  might be one of the most powerful ideas in numpy /
> > scipy  "recently".
> > Did you know about numexpr - David ?
>
> Sadly, I don't think numexpr will help here; It basically handles the same
> cases as numpy; only faster because it can avoid a lot of temporaries. I
> think he's going to need Psyco,  Pyrex, Weave or similar.
>
>
>
This problem of efficient recursion turns up often. The only versions
we support now are the reduce and accumulate methods for ufuncs. I
wonder if we can think of some way of offering support that will speed
it up? The problem space is big enough that there is probably no
method that will solve all problems, but maybe we can toss around some
thoughts. For 2-D sorts of things recursion based on functions on
"half squares" preceding the current value could be useful. A reverse
recursion would also be useful in evaluating a lot of series. The
posted problem is sort of a matrix recursion, though not necessarily
linear. Some sort of accumulate method using a vector of function
calls would probably do the trick. Hmmm... I think that would not be
too hard to do, although the calls to python functions would probably
keep the speedup down to a factor of 2-3x if implemented in C.

In a sense, I am thinking of accumulate and reduce type methods
attached to vectors of functions with more than 2 arguments, iterated
multidimensional maps, if you will.

Chuck


From charlesr.harris at gmail.com  Fri Oct 26 14:48:43 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 26 Oct 2007 12:48:43 -0600
Subject: [Numpy-discussion] Request for advice: project to get NumPy
	working in IronPython
In-Reply-To: <4722325F.7070302@scipy.org>
References: <4713AFAF.10201@resolversystems.com>
	<4713C334.2000407@enthought.com> <4722325F.7070302@scipy.org>
Message-ID: <e06186140710261148rcbe9ad4r9e2a92ac4dfe27ce@mail.gmail.com>

On 10/26/07, dmitrey <dmitrey.kroshko at scipy.org> wrote:
> Travis E. Oliphant wrote:
> > Giles Thomas wrote:
> >
> >> Hi,
> >>
> >> At Resolver Systems, we have a product that is written in IronPython -
> >> the .NET Python implementation - and allows users to use that language
> >> to script a spreadsheet-like interface.  Because they're using
> >> IronPython, they can access their existing .NET objects and libraries,
> >> which has worked out really well for us and for them.  But there's an
> >> increasing number of users who would like access to CPython C
> >> extensions - in particular, NumPy.
> >>
> > An IronPython compatible version of NumPy would be great.    Of course
> > it could be done by using C# to write NumPy, but I'm not sure that this
> > would really be any less work than creating a "glue" layer that allowed
> > most (or all) C-Python extensions to work with IronPython.
> >
> >
> So can anyone inform will IronPython have bridge to NumPy or something else?
> And will it be available in nearest future or some weeks/months/years
> are required?
> (I'm interested in scikits.openopt being available for IronPython as
> well, and it requires numpy)
> And what about pylab for IronPython? Is it work already now, or will be
> available in nearest future, or the situation is undefined?
>

Jim Hugunin, who did the original Numeric, started the IronPython
project and was later hired by Microsoft to bring it to completion. He
mentioned at one of the SciPy conferences that if he had it to do over
again, he would do Numeric in C#. But that is not really an option for
us ;)

Chuck


From tim.hochberg at ieee.org  Fri Oct 26 14:55:52 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Fri, 26 Oct 2007 11:55:52 -0700
Subject: [Numpy-discussion] Request for advice: project to get NumPy
	working in IronPython
In-Reply-To: <e06186140710261148rcbe9ad4r9e2a92ac4dfe27ce@mail.gmail.com>
References: <4713AFAF.10201@resolversystems.com>
	<4713C334.2000407@enthought.com> <4722325F.7070302@scipy.org>
	<e06186140710261148rcbe9ad4r9e2a92ac4dfe27ce@mail.gmail.com>
Message-ID: <e4412d6b0710261155m353d3624l820623f3833422fa@mail.gmail.com>

On 10/26/07, Charles R Harris <charlesr.harris at gmail.com> wrote:
>
> On 10/26/07, dmitrey <dmitrey.kroshko at scipy.org> wrote:
> > Travis E. Oliphant wrote:
> > > Giles Thomas wrote:
> > >
> > >> Hi,
> > >>
> > >> At Resolver Systems, we have a product that is written in IronPython
> -
> > >> the .NET Python implementation - and allows users to use that
> language
> > >> to script a spreadsheet-like interface.  Because they're using
> > >> IronPython, they can access their existing .NET objects and
> libraries,
> > >> which has worked out really well for us and for them.  But there's an
> > >> increasing number of users who would like access to CPython C
> > >> extensions - in particular, NumPy.
> > >>
> > > An IronPython compatible version of NumPy would be great.    Of course
> > > it could be done by using C# to write NumPy, but I'm not sure that
> this
> > > would really be any less work than creating a "glue" layer that
> allowed
> > > most (or all) C-Python extensions to work with IronPython.
> > >
> > >
> > So can anyone inform will IronPython have bridge to NumPy or something
> else?
> > And will it be available in nearest future or some weeks/months/years
> > are required?
> > (I'm interested in scikits.openopt being available for IronPython as
> > well, and it requires numpy)
> > And what about pylab for IronPython? Is it work already now, or will be
> > available in nearest future, or the situation is undefined?
> >
>
> Jim Hugunin, who did the original Numeric, started the IronPython
> project and was later hired by Microsoft to bring it to completion. He
> mentioned at one of the SciPy conferences that if he had it to do over
> again, he would do Numeric in C#. But that is not really an option for
> us ;)


FWIW, I once rewrote the core of Numeric in Java for use with Jython (then
JPyhton). I gave it up after I became disillusioned with Java, but the
code's still out there (http://jnumerical.sourceforge.net/) although I
suspect it is pretty dated at this point.





-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071026/5677bb88/attachment.html>

From oliphant at enthought.com  Fri Oct 26 15:42:08 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Fri, 26 Oct 2007 14:42:08 -0500
Subject: [Numpy-discussion] Request for advice: project to get NumPy
 working in IronPython
In-Reply-To: <4722325F.7070302@scipy.org>
References: <4713AFAF.10201@resolversystems.com>	<4713C334.2000407@enthought.com>
	<4722325F.7070302@scipy.org>
Message-ID: <47224310.1080109@enthought.com>


>> An IronPython compatible version of NumPy would be great.    Of course 
>> it could be done by using C# to write NumPy, but I'm not sure that this 
>> would really be any less work than creating a "glue" layer that allowed 
>> most (or all) C-Python extensions to work with IronPython.
>>     
I'm curious about why all the discussion is about putting Python and its 
extensions on top of C# and very little discussion about just using 
C#-based tools as an extension from CPython.

Python .NET is a great example of what I'm referring to.

The C# language and the CLR does solve some problems, but it does not 
solve all the problems related to scientific computing that it could.   
In particular, I wish it's cross-platform visibility where higher.  Mono 
is a great start, but there are a lot of C# libraries that just don't 
get made to work on Linux or Mac OS X.

The task of moving scipy to sit on top of the CLR seems rather large 
without some automatic tool to allow calling CPython extensions from the 
CLR that works in a cross-platform way.

I don't really see the benefit that the CLR offers (unless all the hype 
is just so you can write code that runs in a browser --- in which case, 
are you really going to run matrix inversion on the CLR in a browser?)

How does legacy code interact with the "magic" of the CLR?

What are people's opinions about the value of NumPy and SciPy on the CLR?

-Travis O.





From charlesr.harris at gmail.com  Fri Oct 26 15:53:26 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Fri, 26 Oct 2007 13:53:26 -0600
Subject: [Numpy-discussion] Request for advice: project to get NumPy
	working in IronPython
In-Reply-To: <47224310.1080109@enthought.com>
References: <4713AFAF.10201@resolversystems.com>
	<4713C334.2000407@enthought.com> <4722325F.7070302@scipy.org>
	<47224310.1080109@enthought.com>
Message-ID: <e06186140710261253l57300a92h894e21373b6fc2b9@mail.gmail.com>

On 10/26/07, Travis E. Oliphant <oliphant at enthought.com> wrote:
>
> >> An IronPython compatible version of NumPy would be great.    Of course
> >> it could be done by using C# to write NumPy, but I'm not sure that this
> >> would really be any less work than creating a "glue" layer that allowed
> >> most (or all) C-Python extensions to work with IronPython.
> >>
> I'm curious about why all the discussion is about putting Python and its
> extensions on top of C# and very little discussion about just using
> C#-based tools as an extension from CPython.
>
> Python .NET is a great example of what I'm referring to.
>
> The C# language and the CLR does solve some problems, but it does not
> solve all the problems related to scientific computing that it could.
> In particular, I wish it's cross-platform visibility where higher.  Mono
> is a great start, but there are a lot of C# libraries that just don't
> get made to work on Linux or Mac OS X.
>
> The task of moving scipy to sit on top of the CLR seems rather large
> without some automatic tool to allow calling CPython extensions from the
> CLR that works in a cross-platform way.
>
> I don't really see the benefit that the CLR offers (unless all the hype
> is just so you can write code that runs in a browser --- in which case,
> are you really going to run matrix inversion on the CLR in a browser?)
>
> How does legacy code interact with the "magic" of the CLR?
>
> What are people's opinions about the value of NumPy and SciPy on the CLR?
>

If we were all Microsoft, all the time, it might be worth it. I run
Linux myself and haven't installed Windows at home in years, so for me
it is a non-starter. Keep it in C  and target generic python.
Portability and compiler independence is the goal.

Chuck


From sransom at nrao.edu  Fri Oct 26 15:53:42 2007
From: sransom at nrao.edu (Scott Ransom)
Date: Fri, 26 Oct 2007 15:53:42 -0400
Subject: [Numpy-discussion] Request for advice: project to get NumPy
	working in IronPython
In-Reply-To: <47224310.1080109@enthought.com>
References: <4713AFAF.10201@resolversystems.com> <4722325F.7070302@scipy.org>
	<47224310.1080109@enthought.com>
Message-ID: <200710261553.42436.sransom@nrao.edu>


> What are people's opinions about the value of NumPy and SciPy on the
> CLR?

As someone who uses Numpy/Scipy almost exclusively on Linux workstations 
or on clusters (in coordination with lots of C code), I wouldn't value 
NumPy and SciPy on the CLR at all.

I am kind of curious, though, to see how many people _would_ think it 
would be usefull....

S


-- 
Scott M. Ransom            Address:  NRAO
Phone:  (434) 296-0320               520 Edgemont Rd.
email:  sransom at nrao.edu             Charlottesville, VA 22903 USA
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989


From dmitrey.kroshko at scipy.org  Fri Oct 26 16:35:55 2007
From: dmitrey.kroshko at scipy.org (dmitrey)
Date: Fri, 26 Oct 2007 23:35:55 +0300
Subject: [Numpy-discussion] Request for advice: project to get
 NumPy	working in IronPython
In-Reply-To: <200710261553.42436.sransom@nrao.edu>
References: <4713AFAF.10201@resolversystems.com>
	<4722325F.7070302@scipy.org>	<47224310.1080109@enthought.com>
	<200710261553.42436.sransom@nrao.edu>
Message-ID: <47224FAB.8020204@scipy.org>

Scott Ransom wrote:
>> What are people's opinions about the value of NumPy and SciPy on the
>> CLR?
>>     
>
> As someone who uses Numpy/Scipy almost exclusively on Linux workstations 
> or on clusters (in coordination with lots of C code), I wouldn't value 
> NumPy and SciPy on the CLR at all.
>
> I am kind of curious, though, to see how many people _would_ think it 
> would be usefull....
>
> S
>
>
>   
my 2 cents:
let me mention that 2/3 of scipy.org/.../openopt webpages visitors have 
Windows installed
I guess whole scipy.org visitors doesn't differ too much.
However, scipy.org website admins could see their own statistic and 
inform us more exactly.
Regards, D

-------------- next part --------------
A non-text attachment was scrubbed...
Name: os.png
Type: image/png
Size: 16171 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071026/342e74cc/attachment.png>

From fuzzyman at voidspace.org.uk  Fri Oct 26 16:50:23 2007
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 26 Oct 2007 21:50:23 +0100
Subject: [Numpy-discussion] [python] Re: Request for advice: project to
 get NumPy working in IronPython
In-Reply-To: <47224310.1080109@enthought.com>
References: <4713AFAF.10201@resolversystems.com>	<4713C334.2000407@enthought.com>	<4722325F.7070302@scipy.org>
	<47224310.1080109@enthought.com>
Message-ID: <4722530F.603@voidspace.org.uk>

Travis E. Oliphant wrote:
>>> An IronPython compatible version of NumPy would be great.    Of course 
>>> it could be done by using C# to write NumPy, but I'm not sure that this 
>>> would really be any less work than creating a "glue" layer that allowed 
>>> most (or all) C-Python extensions to work with IronPython.
>>>     
>>>       
> I'm curious about why all the discussion is about putting Python and its 
> extensions on top of C# and very little discussion about just using 
> C#-based tools as an extension from CPython.
>
> Python .NET is a great example of what I'm referring to.
>
> The C# language and the CLR does solve some problems, but it does not 
> solve all the problems related to scientific computing that it could.   
> In particular, I wish it's cross-platform visibility where higher.  Mono 
> is a great start, but there are a lot of C# libraries that just don't 
> get made to work on Linux or Mac OS X.
>
> The task of moving scipy to sit on top of the CLR seems rather large 
> without some automatic tool to allow calling CPython extensions from the 
> CLR that works in a cross-platform way.
>   

This is exactly what Resolver hopes to achieve in the long-run.

I already have a proof-of-concept hack that already allows you to use 
CPython extensions from IronPython, using the Python.NET work:

    http://www.voidspace.org.uk/ironpython/cpython_extensions.shtml

I had matplotlib (using numpy and Tkinter) working - but it is only a 
proof of concept. :-)

In the long run we would like to provide a mechanism that allows CPython 
extensions to be used seamlessly from IronPython. We're starting with 
Numpy (and not by reimplementing in C#) because we have customers who 
want to use it.

On the CLR subject, the CLR is a great runtime. Advantages include a 
powerful JIT, no GIL, an accepted (and widely used) technology in the 
corporate world, and a native GUI library for Windows (better than any 
I've seen for CPython).

Michael
http://www.manning.com/foord



> I don't really see the benefit that the CLR offers (unless all the hype 
> is just so you can write code that runs in a browser --- in which case, 
> are you really going to run matrix inversion on the CLR in a browser?)
>
> How does legacy code interact with the "magic" of the CLR?
>
> What are people's opinions about the value of NumPy and SciPy on the CLR?
>
> -Travis O.
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   



From kwmsmith at gmail.com  Fri Oct 26 17:12:40 2007
From: kwmsmith at gmail.com (Kurt Smith)
Date: Fri, 26 Oct 2007 16:12:40 -0500
Subject: [Numpy-discussion] Memory leak in ndarray, more info
In-Reply-To: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>
References: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>
Message-ID: <c266a7930710261412gcda4f64raea180cce32b20fd@mail.gmail.com>

On 10/26/07, Robert Crida <robert.crida at ska.ac.za> wrote:
> Hi all
>
> I recently posted about a memory leak in numpy and failed to mention the
> version. The leak manifests itself in numpy-1.0.3.1 but is not present in
> numpy-1.0.2
>
> The following code reproduces the bug:
>
> import numpy as np
>
> a = np.array([1.0, 2.0, 3.0])
> while True:
>     b = str(a)
>
> What happens above is that is repeatedly converted to a string. The process
> size grow quite rapidly.
>
> Has anyone else come across this? Where do I look to try to correct it?

I find the same leak:

Python version: 2.5.1

numpy version: 1.0.4.dev4081

Regards,

Kurt

>
> Thanks
>  Robert


From lists.steve at arachnedesign.net  Fri Oct 26 17:19:42 2007
From: lists.steve at arachnedesign.net (Steve Lianoglou)
Date: Fri, 26 Oct 2007 17:19:42 -0400
Subject: [Numpy-discussion] Request for advice: project to get NumPy
	working in IronPython
In-Reply-To: <47224310.1080109@enthought.com>
References: <4713AFAF.10201@resolversystems.com>	<4713C334.2000407@enthought.com>
	<4722325F.7070302@scipy.org> <47224310.1080109@enthought.com>
Message-ID: <0580F25A-B4D2-4AEE-A5CF-6D898D066C37@arachnedesign.net>

> What are people's opinions about the value of NumPy and SciPy on  
> the CLR?

If anything, wouldn't the "big win" (if it's a win at all) be to get  
NumPy/SciPy working on top of the JVM (as T. Hochber tried)? This way  
it's pretty much universally portable.

I know Jython isn't as up to speed as IronPython, but the folks in  
Jython land are doing a pretty good job of it these days.

Their next "target" CPython compatibility will definitely be good  
enough for NumPy since it will be at least CPythoon 2.3 ... maybe  
even 2.5.

-steve



From nwagner at iam.uni-stuttgart.de  Fri Oct 26 18:02:01 2007
From: nwagner at iam.uni-stuttgart.de (Nils Wagner)
Date: Sat, 27 Oct 2007 00:02:01 +0200
Subject: [Numpy-discussion] Bug in scalartypes.inc.src
Message-ID: <web-91206755@uni-stuttgart.de>

Hi all,

I cannot install numpy from recent svn

creating build/temp.linux-x86_64-2.5
creating build/temp.linux-x86_64-2.5/numpy
creating build/temp.linux-x86_64-2.5/numpy/core
creating build/temp.linux-x86_64-2.5/numpy/core/src
compile options: 
'-Ibuild/src.linux-x86_64-2.5/numpy/core/src 
-Inumpy/core/include 
-Ibuild/src.linux-x86_64-2.5/numpy/core -Inumpy/core/src 
-Inumpy/core/include -I/usr/include/python2.5 -c'
gcc: numpy/core/src/multiarraymodule.c
In file included from numpy/core/src/arrayobject.c:509,
                  from 
numpy/core/src/multiarraymodule.c:97:
numpy/core/src/scalartypes.inc.src: In function 
?gentype_reduce?:
numpy/core/src/scalartypes.inc.src:1312: error: label at 
end of compound statement
In file included from numpy/core/src/arrayobject.c:509,
                  from 
numpy/core/src/multiarraymodule.c:97:
numpy/core/src/scalartypes.inc.src: In function 
?gentype_reduce?:
numpy/core/src/scalartypes.inc.src:1312: error: label at 
end of compound statement
error: Command "gcc -pthread -fno-strict-aliasing -DNDEBUG 
-O2 -fmessage-length=0 -Wall -D_FORTIFY_SOURCE=2 -g -fPIC 
-Ibuild/src.linux-x86_64-2.5/numpy/core/src 
-Inumpy/core/include 
-Ibuild/src.linux-x86_64-2.5/numpy/core -Inumpy/core/src 
-Inumpy/core/include -I/usr/include/python2.5 -c 
numpy/core/src/multiarraymodule.c -o 
build/temp.linux-x86_64-2.5/numpy/core/src/multiarraymodule.o" 
failed with exit status 1

Nils


From peridot.faceted at gmail.com  Fri Oct 26 18:14:25 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Fri, 26 Oct 2007 18:14:25 -0400
Subject: [Numpy-discussion] fortran array storage question
In-Reply-To: <47220A13.3010905@enthought.com>
References: <4720FBC2.6030709@mur.at> <4721DE01.2010505@mur.at>
	<ce557a360710260827t6db077x66e18d0b46509068@mail.gmail.com>
	<47220A13.3010905@enthought.com>
Message-ID: <ce557a360710261514r36ae27c1xc37592f493b9bdd7@mail.gmail.com>

On 26/10/2007, Travis E. Oliphant <oliphant at enthought.com> wrote:

> There is an optimization where-in the inner-loops are done over the
> dimension with the smallest stride.
>
> What other cache-coherent optimizations do you recommend?

That sounds like a very good first step. I'm far from an expert on
this sort of thing, but here are a few ideas at random:
* internally flattening arrays when this doesn't affect the result
(e.g. ones((10,10))+ones((10,10)))
* prefetching memory: in a C application I recently wrote, explicitly
prefetching data for interpolation cut my runtime by 30%. This
includes telling the processor when you're done with data so it can be
purged from the cache.
* aligning (some) arrays to 8- 16- 32- or 64-byte boundaries so that
they divide nicely into cache lines
* using MMX/SSE instructions when available
* combining ufuncs so that computations can keep the CPU busy while it
waits for data to come in from main RAM (I realize that this is
properly the domain of numexpr)
* using ATLAS- or FFTW-style autotuning to determine the fastest ways
to structure computations (again more relevant for significant
expressions rather than simple ufuncs)
* reducing use of temporaries in the interest of reducing traffic to main memory
* openmp parallel operations when this actually speeds up calculation

I realize most of these are a lot of work, and some of them are
probably in numpy already. Moreover without using an expression parser
it's probably not feasible to implement others. But an array language
offers the possibility that a runtime can implement all sorts of
optimizations without effort on the user's part.

Anne


From karol.langner at kn.pl  Sun Oct 28 00:57:58 2007
From: karol.langner at kn.pl (Karol Langner)
Date: Sun, 28 Oct 2007 00:57:58 -0400
Subject: [Numpy-discussion] Memory leak in ndarray, more info
In-Reply-To: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>
References: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>
Message-ID: <200710280057.58646.karol.langner@kn.pl>

On Friday 26 October 2007 05:39, Robert Crida wrote:
> Hi all
>
> I recently posted about a memory leak in numpy and failed to mention the
> version. The leak manifests itself in numpy-1.0.3.1 but is not present in
> numpy-1.0.2
>
> The following code reproduces the bug:
>
> import numpy as np
>
> a = np.array([1.0, 2.0, 3.0])
> while True:
>     b = str(a)
>
> What happens above is that is repeatedly converted to a string. The process
> size grow quite rapidly.
>
> Has anyone else come across this? Where do I look to try to correct it?
>
> Thanks
> Robert

I can reproduce this leak:

Python 2.5 (r25:51908, Apr 30 2007, 15:03:13) 
[GCC 3.4.6 (Debian 3.4.6-5)] on linux2
>>> import numpy
>>> numpy.version.version
'1.0.4.dev4317'
>>> a = numpy.array([1.0])
>>> while True:
...     b = a.__str__()
... 

which causes the resident size of the process to grow by about 1MB/s as 
mentioned earlier. Interestingly, using non-float dtypes does not cause the 
loop to leak.

Karol

-- 
written by Karol Langner
Sun Oct 28 00:47:31 EDT 2007


From matthieu.brucher at gmail.com  Sun Oct 28 10:55:01 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Sun, 28 Oct 2007 15:55:01 +0100
Subject: [Numpy-discussion] Getting an item in an array with its coordinates
	given by another array
Message-ID: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>

Hi,

I'm trying to get an item in an array when I only have another array giving
the position.

For instance:

>>> c = numpy.arange(0., 3*4*5).reshape((3,4,5))
>>> c***(numpy.array((2,3), dtype=int))
[55, 56, 57, 58, 59]

I'm trying to figure what to put between c and ((2,3)). I supposed that the
take method would be what I wanted but it is not... The [] operator does not
work on array(but it seems to give the expected result with a tuple or a
list). Is there a method for this ?

Matthieu

-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071028/f9a4fea1/attachment.html>

From matthieu.brucher at gmail.com  Sun Oct 28 10:59:20 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Sun, 28 Oct 2007 15:59:20 +0100
Subject: [Numpy-discussion] Getting an item in an array with its
	coordinates given by another array
In-Reply-To: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
References: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
Message-ID: <e76aa17f0710280759y19bdcb8ftb866e212c2f87a1e@mail.gmail.com>

Little correction, only c[(2,3)] gives me what I expect, not c[[2,3]], which
is even stranger.

2007/10/28, Matthieu Brucher <matthieu.brucher at gmail.com>:
>
> Hi,
>
> I'm trying to get an item in an array when I only have another array
> giving the position.
>
> For instance:
>
> >>> c = numpy.arange(0., 3*4*5).reshape((3,4,5))
> >>> c***(numpy.array ((2,3), dtype=int))
> [55, 56, 57, 58, 59]
>
> I'm trying to figure what to put between c and ((2,3)). I supposed that
> the take method would be what I wanted but it is not... The [] operator does
> not work on array(but it seems to give the expected result with a tuple or a
> list). Is there a method for this ?
>
> Matthieu
>
> --
> French PhD student
> Website : http://miles.developpez.com/
> Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92




-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071028/51dc5f6b/attachment.html>

From srossross at gmail.com  Sun Oct 28 14:27:01 2007
From: srossross at gmail.com (Sean Ross-Ross)
Date: Sun, 28 Oct 2007 11:27:01 -0700
Subject: [Numpy-discussion] Array struct
Message-ID: <DE796B35-FDB0-45E0-9401-053FFD77EA54@gmail.com>

Hi, I am looking to create an __array_struct__ member for a c++  
vector<vector<cppObj>> i have wrapped in python, where cppObj  
contains an array of floats.

Currently I have created  an __array_struct__ for the cppObj and in  
python I iterate over the vector of vectors and put all of the data  
into a 1-d array.

Is it possible to do this with an __array_struct__ within c++ for the  
whole object at once? Perhaps creating a float **.

Thanks for your help
~Sean




From karol.langner at kn.pl  Sun Oct 28 20:28:53 2007
From: karol.langner at kn.pl (Karol Langner)
Date: Sun, 28 Oct 2007 20:28:53 -0400
Subject: [Numpy-discussion] Memory leak in ndarray, more info
In-Reply-To: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>
References: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>
Message-ID: <200710282028.53170.karol.langner@kn.pl>

On Friday 26 October 2007 05:39, Robert Crida wrote:
> I recently posted about a memory leak in numpy and failed to mention the
> version. The leak manifests itself in numpy-1.0.3.1 but is not present in
> numpy-1.0.2
>
> The following code reproduces the bug:
>
> import numpy as np
>
> a = np.array([1.0, 2.0, 3.0])
> while True:
>     b = str(a)
>
> What happens above is that is repeatedly converted to a string. The process
> size grow quite rapidly.
>
> Has anyone else come across this? Where do I look to try to correct it?

Debugging exposes numpy.core.umath.seterrobj as the source, so you can get 
even faster memory growth with this (around 30MB/s on my machine):

import numpy
pyvals = numpy.core.umath.geterrobj()
while True:
    numpy.core.umath.seterrobj(pyvals)

That is actually ufunc_seterr in trunk/numpy/core/src/ufuncobject.c... which 
calls ufunc_update_use_defaults as of r3040. A call to Py_DECREF(errobj) is 
missing there after calling PyUFunc_GetPyValues.

So using the following patch on the current svn revision of ufuncobject.c 
should fix this leak:

3206a3207
>     Py_DECREF(errobj);

Cheers,
Karol

-- 
written by Karol Langner
Sun Oct 28 20:00:47 EDT 2007


From matthieu.brucher at gmail.com  Sun Oct 28 14:46:30 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Sun, 28 Oct 2007 19:46:30 +0100
Subject: [Numpy-discussion] Array struct
In-Reply-To: <DE796B35-FDB0-45E0-9401-053FFD77EA54@gmail.com>
References: <DE796B35-FDB0-45E0-9401-053FFD77EA54@gmail.com>
Message-ID: <e76aa17f0710281146i188dae9bj91207acd18dc576@mail.gmail.com>

Hi,

I think you will have to use non-contiguous arrays, it should be possible.
But I can't help you as I never use them.

Matthieu


2007/10/28, Sean Ross-Ross <srossross at gmail.com>:
>
> Hi, I am looking to create an __array_struct__ member for a c++
> vector<vector<cppObj>> i have wrapped in python, where cppObj
> contains an array of floats.
>
> Currently I have created  an __array_struct__ for the cppObj and in
> python I iterate over the vector of vectors and put all of the data
> into a 1-d array.
>
> Is it possible to do this with an __array_struct__ within c++ for the
> whole object at once? Perhaps creating a float **.
>
> Thanks for your help
> ~Sean
>
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071028/27dab7dd/attachment.html>

From karol.langner at kn.pl  Sun Oct 28 23:31:34 2007
From: karol.langner at kn.pl (Karol Langner)
Date: Sun, 28 Oct 2007 23:31:34 -0400
Subject: [Numpy-discussion] Memory leak in ndarray, more info
In-Reply-To: <200710282028.53170.karol.langner@kn.pl>
References: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>
	<200710282028.53170.karol.langner@kn.pl>
Message-ID: <200710282331.34822.karol.langner@kn.pl>

I opened a ticket for this (#602). Hopefully someone will confirm that adding 
that Py_DECREF call fixes the leak and someone with write access patches it 
in svn.

- Karol

-- 
written by Karol Langner
Sun Oct 28 23:29:18 EDT 2007


From oliphant at enthought.com  Sun Oct 28 21:01:14 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Sun, 28 Oct 2007 20:01:14 -0500
Subject: [Numpy-discussion] Memory leak in ndarray, more info
In-Reply-To: <200710282331.34822.karol.langner@kn.pl>
References: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>	<200710282028.53170.karol.langner@kn.pl>
	<200710282331.34822.karol.langner@kn.pl>
Message-ID: <472530DA.30300@enthought.com>

Karol Langner wrote:
> I opened a ticket for this (#602). Hopefully someone will confirm that adding 
> that Py_DECREF call fixes the leak and someone with write access patches it 
> in svn.
>   

Thanks for looking into this and isolating the problem...

-Travis O.




From david at ar.media.kyoto-u.ac.jp  Sun Oct 28 22:18:44 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 29 Oct 2007 11:18:44 +0900
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>
References: <472123B6.1050609@jpl.nasa.gov>	<20071026063619.GC9025@clipper.ens.fr>	<47218CD7.2000501@ar.media.kyoto-u.ac.jp>
	<bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>
Message-ID: <47254304.5000805@ar.media.kyoto-u.ac.jp>

Sebastian Haase wrote:
> On 10/26/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>   
>> P.S: IMHO, this is one of the main limitation of numpy (or any language
>> using arrays for speed; and this is really difficult to optimize: you
>> need compilation, JIT or similar to solve those efficiently).
>>     
>
> This is where the scipy - sandbox numexpr  project comes in
> - if I'm not misaken ....
>
> http://www.scipy.org/SciPyPackages/NumExpr
>
> Description
> The scipy.sandbox.numexpr package supplies routines for the fast
> evaluation of array expressions elementwise by using a vector-based
> virtual machine. It's comparable to scipy.weave.blitz (in Weave), but
> doesn't require a separate compile step of C or C++ code.
>
>
> I hope that more noise around this  will result  in more interest and
> subsequentially result in more support.
> I think numexpr  might be one of the most powerful ideas in numpy /
> scipy  "recently".
> Did you know about numexpr - David ?
>
>   
I knew about it, but never really tried it. But numexpr still through 
python for function calls, right ?

The big problem of python for (recursive) numeric work is the function 
call cost. On my macbook (core 2 duo @ 2 ghz), which I already consider 
quite beefy, I cannot call more than 0.5-1 million functions every 
second (really simple ones, e.g. not member functions of some objects, 
which are more costly). That's why I don't see much hope outside JIT for 
those (incidently, the kind of things 'we' are doing seem like the most 
simple things to JIT).

cheers,

David


From david at ar.media.kyoto-u.ac.jp  Sun Oct 28 22:58:38 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 29 Oct 2007 11:58:38 +0900
Subject: [Numpy-discussion] Request for advice: project to get NumPy
 working in IronPython
In-Reply-To: <47224310.1080109@enthought.com>
References: <4713AFAF.10201@resolversystems.com>	<4713C334.2000407@enthought.com>	<4722325F.7070302@scipy.org>
	<47224310.1080109@enthought.com>
Message-ID: <47254C5E.5030103@ar.media.kyoto-u.ac.jp>

Travis E. Oliphant wrote:
>>> An IronPython compatible version of NumPy would be great.    Of course 
>>> it could be done by using C# to write NumPy, but I'm not sure that this 
>>> would really be any less work than creating a "glue" layer that allowed 
>>> most (or all) C-Python extensions to work with IronPython.
>>>     
> I'm curious about why all the discussion is about putting Python and its 
> extensions on top of C# and very little discussion about just using 
> C#-based tools as an extension from CPython.
>
> Python .NET is a great example of what I'm referring to.
>
> The C# language and the CLR does solve some problems, but it does not 
> solve all the problems related to scientific computing that it could.   
> In particular, I wish it's cross-platform visibility where higher.  Mono 
> is a great start, but there are a lot of C# libraries that just don't 
> get made to work on Linux or Mac OS X.
I think it is useful to separate C# and the CLR: you can do C# without 
the CLR (with mono at least), and you can do CLR without C#. What is C# 
? At the core, it is a language with a C-like syntax with automatic 
memory management, and a huge framework. C is still, by far, the best 
language for interoperability. None (and this includes C++) comes even 
close. For a library, C is the best language to use if you care about 
interoperability, so I don't see how it would be a bad choice.

So I don't see the point of C#, at least related to numpy.
>
> The task of moving scipy to sit on top of the CLR seems rather large 
> without some automatic tool to allow calling CPython extensions from the 
> CLR that works in a cross-platform way.
>
> I don't really see the benefit that the CLR offers (unless all the hype 
> is just so you can write code that runs in a browser --- in which case, 
> are you really going to run matrix inversion on the CLR in a browser?)
I don't think running in a browser is the only useful thing of the CRL; 
I can see why this would be useful for some, but at least, for me, this 
is not interesting (in my current, PhD student position at least :) ). 
In a windows centric world, running on the CLR means having access to 
the whole windows framework. C# is not really an interesting language, 
frankly, the only thing which really matters is that it does automatic 
memory management, and is endorsed by MS, meaning access to the windows 
ecosystem. This by itself is already so huge that any other things is a 
detail.

No, the important point is the following: once you have an 
implementation on the CLR, you can use it from any language. For a less 
hyped based discussion of this, you can read the B. Meyer article (B. 
Meyer is quite involved with his company in .net, but at least, this 
article focus on the technology):

http://archive.eiffel.com/doc/manuals/technology/bmarticles/sd/dotnet.html

As a concrete example, look at F#: if numpy can run on the CLR, you can 
use it from C#, IronPython and F#. F# is, basically, OCAML on the .NET. 
And as much as I don't have much interest into C#, I consider F# to be 
much more interesting (before considering python + numpy as my 
environment of choice for all my numeric-based projects, I considered 
OCAML).

http://www.ffconsultancy.com/products/fsharp_for_visualization/index.html

If numpy can be used from F#, this would be interesting. But is this at 
all possible ? For example, are the semantic compatible (indexing, 
etc...) ? I have no idea.

Implementation-wise, this is an interesting problem to solve.

cheers,

David


From ckkart at hoc.net  Sun Oct 28 23:21:34 2007
From: ckkart at hoc.net (Christian K.)
Date: Mon, 29 Oct 2007 03:21:34 +0000 (UTC)
Subject: [Numpy-discussion]
	=?utf-8?q?Getting_an_item_in_an_array_with_its?=
	=?utf-8?q?=09coordinates_given_by_another_array?=
References: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
	<e76aa17f0710280759y19bdcb8ftb866e212c2f87a1e@mail.gmail.com>
Message-ID: <loom.20071029T030633-253@post.gmane.org>

Matthieu Brucher <matthieu.brucher <at> gmail.com> writes:

> 
> 
> Little correction, only c[(2,3)] gives me what I expect, not c[[2,3]], which
> is even stranger.

c[(2,3)] is the same as c[2,3] and obviously works as you expected.
c[[2,3]] is refered to as 'advanced indexing' in the numpy book.
It will return elements 2 and 3 along the first dimension. To get what you
want, you need to put it like this:

c[[2],[3]]

In [118]: c = N.arange(0.,3*4*5).reshape((3,4,5))

In [119]: c[[2],[3]]

Out[119]: array([[ 55.,  56.,  57.,  58.,  59.]])

This does not work however using a ndarray holding the indices. 

In [120]: ind = N.array([[2],[3]])

In [121]: c[ind]
---------------------------------------------------------------------------
<type 'exceptions.IndexError'>            Traceback (most recent call last)

/media/hda6/home/ck/<ipython console> in <module>()

<type 'exceptions.IndexError'>: index (3) out of range (0<=index<=2) in
dimension 0

so you have to convert it to a list before:

In [122]: c[ind.tolist()]
Out[122]: array([[ 55.,  56.,  57.,  58.,  59.]])

Christian



From robert.crida at ska.ac.za  Mon Oct 29 01:43:20 2007
From: robert.crida at ska.ac.za (Robert Crida)
Date: Mon, 29 Oct 2007 07:43:20 +0200
Subject: [Numpy-discussion] Memory leak in ndarray, more info
In-Reply-To: <200710282331.34822.karol.langner@kn.pl>
References: <c0935a560710260239t5dda66e5od3fd80d5ae9c78df@mail.gmail.com>
	<200710282028.53170.karol.langner@kn.pl>
	<200710282331.34822.karol.langner@kn.pl>
Message-ID: <c0935a560710282243s1f734d48r40e25dda596eb345@mail.gmail.com>

Hi Karol

Thank you very much for the sleuth work here. We are in the midst of
software ATP so it helps a lot that I will be able to resolve this bug
properly.

Robert

On 10/29/07, Karol Langner <karol.langner at kn.pl> wrote:
>
> I opened a ticket for this (#602). Hopefully someone will confirm that
> adding
> that Py_DECREF call fixes the leak and someone with write access patches
> it
> in svn.
>
> - Karol
>
> --
> written by Karol Langner
> Sun Oct 28 23:29:18 EDT 2007
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071029/8b777968/attachment.html>

From matthieu.brucher at gmail.com  Mon Oct 29 02:46:38 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 29 Oct 2007 07:46:38 +0100
Subject: [Numpy-discussion] Getting an item in an array with its
	coordinates given by another array
In-Reply-To: <loom.20071029T030633-253@post.gmane.org>
References: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
	<e76aa17f0710280759y19bdcb8ftb866e212c2f87a1e@mail.gmail.com>
	<loom.20071029T030633-253@post.gmane.org>
Message-ID: <e76aa17f0710282346x143b35d2y7bd8ef120407d4f8@mail.gmail.com>

>
> > Little correction, only c[(2,3)] gives me what I expect, not c[[2,3]],
> which
> > is even stranger.
>
> c[(2,3)] is the same as c[2,3] and obviously works as you expected.



Well, this is not indicated in the documentation.


c[[2,3]] is refered to as 'advanced indexing' in the numpy book.
> It will return elements 2 and 3 along the first dimension. To get what you
> want, you need to put it like this:
>
> c[[2],[3]]
>
> In [118]: c = N.arange(0.,3*4*5).reshape((3,4,5))
>
> In [119]: c[[2],[3]]
>
> Out[119]: array([[ 55.,  56.,  57.,  58.,  59.]])



This is very strange to say the least, as tuple do not work in the same way.


This does not work however using a ndarray holding the indices.
>
> In [120]: ind = N.array([[2],[3]])
>
> In [121]: c[ind]
>
> ---------------------------------------------------------------------------
> <type 'exceptions.IndexError'>            Traceback (most recent call
> last)
>
> /media/hda6/home/ck/<ipython console> in <module>()
>
> <type 'exceptions.IndexError'>: index (3) out of range (0<=index<=2) in
> dimension 0
>
> so you have to convert it to a list before:
>
> In [122]: c[ind.tolist()]
> Out[122]: array([[ 55.,  56.,  57.,  58.,  59.]])


But if I have the coordinates of the points in an array, I have to reshape
it and then convert it into a list. Or convert it into a list and then
convert it to a tuple. I know that advanced indexing is useful, but here it
is not coherent. tuples and lists should have the same result on the array,
or at least it should be documented.
So there is not way to get a sub-array based on coordinates in an array ?

Matthieu

-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071029/46b0702a/attachment.html>

From millman at berkeley.edu  Mon Oct 29 03:22:55 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 29 Oct 2007 00:22:55 -0700
Subject: [Numpy-discussion] NumPy 1.0.4 release
In-Reply-To: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>
References: <c7009a550710182035x1c0370eaufa6161ad23a4824d@mail.gmail.com>
Message-ID: <c7009a550710290022y6b4cc3a0rba83790f492257ac@mail.gmail.com>

Hello,

I just got back from a camping trip and haven't had time to check the
status of the remaining open tickets; but it looks like there are a
few tickets that are ready to close.  So I am pushing the schedule
back a week (with the hope that we can close a few more tickets before
the 1.0.4 release):

Sunday (11/4)
  * Release Tagged.  At this point, I will be asking for help from
packagers and testers.

Wednesday (11/7)
   * Release Announced.

If you have time this week or weekend, please take a look at the open
tickets and see if you can help close one:
http://projects.scipy.org/scipy/numpy/milestone/1.0.4

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From matthieu.brucher at gmail.com  Mon Oct 29 03:56:01 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 29 Oct 2007 09:56:01 +0200
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710222330i346423d1v5bb91ca69aa28363@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
	<4713C111.5010609@enthought.com>
	<e76aa17f0710190327p18a55596q38fada88d23dddb@mail.gmail.com>
	<e76aa17f0710190328u7b2b950foebd04943fda15bc5@mail.gmail.com>
	<e76aa17f0710222330i346423d1v5bb91ca69aa28363@mail.gmail.com>
Message-ID: <e76aa17f0710290056w4d549746r70c38147db92c1bf@mail.gmail.com>

Does someone have a clue ? I'm starting getting this with all my datasets,
always at the same line !

Matthieu

2007/10/23, Matthieu Brucher <matthieu.brucher at gmail.com>:
>
> The problem reappeared today, same error, always in the line :
> xp = (x-self.loc) * (1/self.scale)
>
> When multiplying an array with a float.
>
> Matthieu
>
> 2007/10/19, Matthieu Brucher < matthieu.brucher at gmail.com>:
> >
> > Oups, sorry for the sending twice, we have some problem with our DNS in
> > Strasbourg :(
> >
> > 2007/10/19, Matthieu Brucher <matthieu.brucher at gmail.com >:
> > >
> > > Here is an excerpt of the stack on the numpy svn of wednesday :
> > >
> > > #0  0x40000402 in __kernel_vsyscall ()
> > > #1  0x00b8e382 in sem_post at GLIBC_2.0 () from /lib/libpthread.so.0
> > > #2  0x080fe5b7 in PyThread_release_lock (lock=0x80d0fb2) at
> > > Python/thread_pthread.h:374
> > > #3  0x080d0fb2 in PyEval_SaveThread () at Python/ceval.c:299
> > > #4  0x4064ec7a in dotblas_innerproduct (dummy=0x0, args=0x960522c) at
> > > numpy/core/blasdot/_dotblas.c:835
> > > #5  0x0811ecf1 in PyCFunction_Call (func=0x960522c, arg=0x80cfec7,
> > > kw=0x942068c) at Objects/methodobject.c:73
> > > #6  0x080cfec7 in call_function (pp_stack=0x0, oparg=1) at
> > > Python/ceval.c:3564
> > > #7  0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> > > #8  0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> > > #9  0x0811de5b in function_call (func=0x0, arg=0x960516c,
> > > kw=0x965b148) at Objects/funcobject.c:517
> > > #10 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960516c,
> > > kw=0x8061a66) at Objects/abstract.c:1860
> > > #11 0x08061a66 in instancemethod_call (func=0x95f16ac, arg=0x94f5824,
> > > kw=0x805bdb6) at Objects/classobject.c:2497
> > > #12 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac,
> > > kw=0x80a86e9) at Objects/abstract.c:1860
> > > #13 0x080a86e9 in slot_tp_call (self=0xbf975c78, args=0x0,
> > > kwds=0x805bdb6) at Objects/typeobject.c:4633
> > > #14 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac,
> > > kw=0x80d017b) at Objects/abstract.c:1860
> > > #15 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=154955456)
> > > at Python/ceval.c:3775
> > > #16 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> > > Python/ceval.c:3587
> > > #17 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> > > #18 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> > > #19 0x0811de5b in function_call (func=0x0, arg=0x960520c, kw=0x30) at
> > > Objects/funcobject.c:517
> > > #20 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960520c,
> > > kw=0x8061a66) at Objects/abstract.c:1860
> > > #21 0x08061a66 in instancemethod_call (func=0x95ecbac, arg=0x94f57d4,
> > > kw=0x805bdb6) at Objects/classobject.c:2497
> > > #22 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac,
> > > kw=0x80a86e9) at Objects/abstract.c:1860
> > > #23 0x080a86e9 in slot_tp_call (self=0xbf97609c, args=0x0,
> > > kwds=0x805bdb6) at Objects/typeobject.c:4633
> > > #24 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac,
> > > kw=0x80d017b) at Objects/abstract.c:1860
> > > #25 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=156957888)
> > > at Python/ceval.c:3775
> > > #26 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> > > Python/ceval.c:3587
> > > #27 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> > > #28 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> > > #29 0x0811de5b in function_call (func=0x0, arg=0x960518c, kw=0x9548a1)
> > > at Objects/funcobject.c:517
> > > #30 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960518c,
> > > kw=0x8061a66) at Objects/abstract.c:1860
> > > #31 0x08061a66 in instancemethod_call (func=0x960526c, arg=0x95fc874,
> > > kw=0x805bdb6) at Objects/classobject.c:2497
> > > #32 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c,
> > > kw=0x80a86e9) at Objects/abstract.c:1860
> > > #33 0x080a86e9 in slot_tp_call (self=0xbf9764c0, args=0x0,
> > > kwds=0x805bdb6) at Objects/typeobject.c:4633
> > > #34 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c,
> > > kw=0x80d017b) at Objects/abstract.c:1860
> > > #35 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=157640136)
> > > at Python/ceval.c:3775
> > > #36 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> > > Python/ceval.c:3587
> > > #37 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> > > #38 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> > > #39 0x0811de5b in function_call (func=0x0, arg=0x95f260c, kw=0x0) at
> > > Objects/funcobject.c:517
> > > #40 0x0805bdb6 in PyObject_Call (func=0x9609934, arg=0x95f260c,
> > > kw=0x8061a66) at Objects/abstract.c:1860
> > >
> > > Seems that
> > >
> > > 2007/10/15, Travis E. Oliphant < oliphant at enthought.com >:
> > > >
> > > > Matthieu Brucher wrote:
> > > > >
> > > > >     The problem is that there is a data-type reference counting
> > > > error some
> > > > >     where that is attempting to deallocate the built-in data-type
> > > > 'l'
> > > > >
> > > > >
> > > > >
> > > > > That's what I supposed, but I couldn't find the reason why it
> > > > wanted
> > > > > to do this
> > > > >
> > > > >
> > > > >     It's not really a Python error but a logging.  The code won't
> > > > let you
> > > > >     deallocate the built-ins, but it will tell you that something
> > > > >     tried to.
> > > > >
> > > > >     Reference counting on data-types is easy to get wrong
> > > > >     (particularly with
> > > > >     Pyrex extension modules) because most calls consume a
> > > > reference to the
> > > > >     data-type (if they return an object that  contains a reference
> > > > to the
> > > > >     data-type).
> > > > >
> > > > >     It is a bug, and it would be nice to figure it out, but that
> > > > would
> > > > >     require the code that caused it.
> > > > >
> > > > >
> > > > > I've updated my numpy version to the latest svn, the behaviour
> > > > seems
> > > > > to be different (more warnings), I'll try to give more information
> > > > > about the error, but giving the whole code will not be simple (it
> > > > uses
> > > > > a big data file that seems to trigger the error as with other data
> > > >
> > > > > files, the error didn't show up :()
> > > > >
> > > >
> > > > There are two types of errors that can occur with reference counting
> > > > on
> > > > data-types.
> > > >
> > > > 1) There are too many DECREF's --- this gets us to the error quickly
> > > > and
> > > > is usually easy to reproduce
> > > > 2) There are too many INCREF's (the reference count keeps going up
> > > > until
> > > > the internal counter wraps around to 0 and deallocation is
> > > > attempted)
> > > > --- this error is harder to reproduce and usually takes a while
> > > > before
> > > > it happens in the code.
> > > >
> > > >
> > > > -Travis
> > > >
> > > > _______________________________________________
> > > > Numpy-discussion mailing list
> > > > Numpy-discussion at scipy.org
> > > > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> > > >
> > >
> > >
> >
>


-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071029/b422cef7/attachment.html>

From david at ar.media.kyoto-u.ac.jp  Mon Oct 29 03:48:20 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Mon, 29 Oct 2007 16:48:20 +0900
Subject: [Numpy-discussion] [numpy.scons] Some really basic starting info
Message-ID: <47259044.2050403@ar.media.kyoto-u.ac.jp>

Hi,

    Since several people have been asking me how to use scons to build 
python modules, I have started a wiki page on numpy wiki:

http://scipy.org/scipy/numpy/wiki/NumpyScons

This is really preliminary, but it should be enough to get you started 
if you have simple needs (building a Ctypes extension, checking for the 
presence of some libraries, etc...).

cheers,

David


From bernhard.voigt at gmail.com  Mon Oct 29 04:39:12 2007
From: bernhard.voigt at gmail.com (bernhard.voigt at gmail.com)
Date: Mon, 29 Oct 2007 08:39:12 -0000
Subject: [Numpy-discussion] Getting an item in an array with its
	coordinates given by another array
In-Reply-To: <e76aa17f0710282346x143b35d2y7bd8ef120407d4f8@mail.gmail.com>
References: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
	<e76aa17f0710280759y19bdcb8ftb866e212c2f87a1e@mail.gmail.com>
	<loom.20071029T030633-253@post.gmane.org>
	<e76aa17f0710282346x143b35d2y7bd8ef120407d4f8@mail.gmail.com>
Message-ID: <1193647152.496209.310410@y42g2000hsy.googlegroups.com>

Take a look at numpy.ix_
http://www.scipy.org/Numpy_Example_List_With_Doc#head-603de8bdb62d0412798c45fe1db0648d913c8a9c

This method creates the index array for you. You only have to specify
the coordinates in each dimesion.

Bernhard

On Oct 29, 8:46 am, "Matthieu Brucher" <matthieu.bruc... at gmail.com>
wrote:
> > > Little correction, only c[(2,3)] gives me what I expect, not c[[2,3]],
> > which
> > > is even stranger.
>
> > c[(2,3)] is the same as c[2,3] and obviously works as you expected.
>
> Well, this is not indicated in the documentation.
>
> c[[2,3]] is refered to as 'advanced indexing' in the numpy book.
>
> > It will return elements 2 and 3 along the first dimension. To get what you
> > want, you need to put it like this:
>
> > c[[2],[3]]
>
> > In [118]: c = N.arange(0.,3*4*5).reshape((3,4,5))
>
> > In [119]: c[[2],[3]]
>
> > Out[119]: array([[ 55.,  56.,  57.,  58.,  59.]])
>
> This is very strange to say the least, as tuple do not work in the same way.
>
> This does not work however using a ndarray holding the indices.
>
>
>
>
>
> > In [120]: ind = N.array([[2],[3]])
>
> > In [121]: c[ind]
>
> > ---------------------------------------------------------------------------
> > <type 'exceptions.IndexError'>            Traceback (most recent call
> > last)
>
> > /media/hda6/home/ck/<ipython console> in <module>()
>
> > <type 'exceptions.IndexError'>: index (3) out of range (0<=index<=2) in
> > dimension 0
>
> > so you have to convert it to a list before:
>
> > In [122]: c[ind.tolist()]
> > Out[122]: array([[ 55.,  56.,  57.,  58.,  59.]])
>
> But if I have the coordinates of the points in an array, I have to reshape
> it and then convert it into a list. Or convert it into a list and then
> convert it to a tuple. I know that advanced indexing is useful, but here it
> is not coherent. tuples and lists should have the same result on the array,
> or at least it should be documented.
> So there is not way to get a sub-array based on coordinates in an array ?
>
> Matthieu
>
> --
> French PhD student
> Website :http://miles.developpez.com/
> Blogs :http://matt.eifelle.comandhttp://blog.developpez.com/?blog=92
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discuss... at scipy.orghttp://projects.scipy.org/mailman/listinfo/numpy-discussion



From matthieu.brucher at gmail.com  Mon Oct 29 05:31:23 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 29 Oct 2007 10:31:23 +0100
Subject: [Numpy-discussion] Error in deallocation ?
In-Reply-To: <e76aa17f0710190053v272ae632y8dec40eec05980a3@mail.gmail.com>
References: <e76aa17f0710150602v7ac6b902le8e8d0cffa841dc1@mail.gmail.com>
	<47137A2E.6030006@enthought.com>
	<e76aa17f0710150739x3a1de7a8m2ee4bb74cb4f714b@mail.gmail.com>
	<4713C111.5010609@enthought.com>
	<e76aa17f0710190053v272ae632y8dec40eec05980a3@mail.gmail.com>
Message-ID: <e76aa17f0710290231x2567dea2l444035434c76d75d@mail.gmail.com>

It seems this could be a deadlock somewhere ? The line 835 is where threads
are allowed.
I had another problem today with the same instruction ((x-self.loc) *
(1/self.scale)), the computation just hung. I'm trying to test without the
thread lock.

Matthieu

2007/10/19, Matthieu Brucher <matthieu.brucher at gmail.com>:
>
> Here is an excerpt of the stack on the numpy svn of wednesday :
>
> #0  0x40000402 in __kernel_vsyscall ()
> #1  0x00b8e382 in sem_post at GLIBC_2.0 () from /lib/libpthread.so.0
> #2  0x080fe5b7 in PyThread_release_lock (lock=0x80d0fb2) at
> Python/thread_pthread.h:374
> #3  0x080d0fb2 in PyEval_SaveThread () at Python/ceval.c:299
> #4  0x4064ec7a in dotblas_innerproduct (dummy=0x0, args=0x960522c) at
> numpy/core/blasdot/_dotblas.c:835
> #5  0x0811ecf1 in PyCFunction_Call (func=0x960522c, arg=0x80cfec7,
> kw=0x942068c) at Objects/methodobject.c:73
> #6  0x080cfec7 in call_function (pp_stack=0x0, oparg=1) at
> Python/ceval.c:3564
> #7  0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> #8  0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> #9  0x0811de5b in function_call (func=0x0, arg=0x960516c, kw=0x965b148) at
> Objects/funcobject.c:517
> #10 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960516c, kw=0x8061a66) at
> Objects/abstract.c:1860
> #11 0x08061a66 in instancemethod_call (func=0x95f16ac, arg=0x94f5824,
> kw=0x805bdb6) at Objects/classobject.c:2497
> #12 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac, kw=0x80a86e9) at
> Objects/abstract.c:1860
> #13 0x080a86e9 in slot_tp_call (self=0xbf975c78, args=0x0, kwds=0x805bdb6)
> at Objects/typeobject.c:4633
> #14 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95f16ac, kw=0x80d017b) at
> Objects/abstract.c:1860
> #15 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=154955456) at
> Python/ceval.c:3775
> #16 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> Python/ceval.c:3587
> #17 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> #18 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> #19 0x0811de5b in function_call (func=0x0, arg=0x960520c, kw=0x30) at
> Objects/funcobject.c:517
> #20 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960520c, kw=0x8061a66) at
> Objects/abstract.c:1860
> #21 0x08061a66 in instancemethod_call (func=0x95ecbac, arg=0x94f57d4,
> kw=0x805bdb6) at Objects/classobject.c:2497
> #22 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac, kw=0x80a86e9) at
> Objects/abstract.c:1860
> #23 0x080a86e9 in slot_tp_call (self=0xbf97609c, args=0x0, kwds=0x805bdb6)
> at Objects/typeobject.c:4633
> #24 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x95ecbac, kw=0x80d017b) at
> Objects/abstract.c:1860
> #25 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=156957888) at
> Python/ceval.c:3775
> #26 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> Python/ceval.c:3587
> #27 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> #28 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> #29 0x0811de5b in function_call (func=0x0, arg=0x960518c, kw=0x9548a1) at
> Objects/funcobject.c:517
> #30 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960518c, kw=0x8061a66) at
> Objects/abstract.c:1860
> #31 0x08061a66 in instancemethod_call (func=0x960526c, arg=0x95fc874,
> kw=0x805bdb6) at Objects/classobject.c:2497
> #32 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c, kw=0x80a86e9) at
> Objects/abstract.c:1860
> #33 0x080a86e9 in slot_tp_call (self=0xbf9764c0, args=0x0, kwds=0x805bdb6)
> at Objects/typeobject.c:4633
> #34 0x0805bdb6 in PyObject_Call (func=0x0, arg=0x960526c, kw=0x80d017b) at
> Objects/abstract.c:1860
> #35 0x080d017b in do_call (func=0x0, pp_stack=0x1, na=1, nk=157640136) at
> Python/ceval.c:3775
> #36 0x080cfd56 in call_function (pp_stack=0x0, oparg=1) at
> Python/ceval.c:3587
> #37 0x080cb19a in PyEval_EvalFrameEx. () at Python/ceval.c:2267
> #38 0x080ca848 in PyEval_EvalCodeEx. () at Python/ceval.c:2831
> #39 0x0811de5b in function_call (func=0x0, arg=0x95f260c, kw=0x0) at
> Objects/funcobject.c:517
> #40 0x0805bdb6 in PyObject_Call (func=0x9609934, arg=0x95f260c,
> kw=0x8061a66) at Objects/abstract.c:1860
>
> Seems that the bug could be somewhere in the handling of the dot blas
> module ?
>
> Matthieu
>
> 2007/10/15, Travis E. Oliphant <oliphant at enthought.com >:
> >
> > Matthieu Brucher wrote:
> > >
> > >     The problem is that there is a data-type reference counting error
> > some
> > >     where that is attempting to deallocate the built-in data-type 'l'
> > >
> > >
> > >
> > > That's what I supposed, but I couldn't find the reason why it wanted
> > > to do this
> > >
> > >
> > >     It's not really a Python error but a logging.  The code won't let
> > you
> > >     deallocate the built-ins, but it will tell you that something
> > >     tried to.
> > >
> > >     Reference counting on data-types is easy to get wrong
> > >     (particularly with
> > >     Pyrex extension modules) because most calls consume a reference to
> > the
> > >     data-type (if they return an object that  contains a reference to
> > the
> > >     data-type).
> > >
> > >     It is a bug, and it would be nice to figure it out, but that would
> > >     require the code that caused it.
> > >
> > >
> > > I've updated my numpy version to the latest svn, the behaviour seems
> > > to be different (more warnings), I'll try to give more information
> > > about the error, but giving the whole code will not be simple (it uses
> > > a big data file that seems to trigger the error as with other data
> > > files, the error didn't show up :()
> > >
> >
> > There are two types of errors that can occur with reference counting on
> > data-types.
> >
> > 1) There are too many DECREF's --- this gets us to the error quickly and
> >
> > is usually easy to reproduce
> > 2) There are too many INCREF's (the reference count keeps going up until
> > the internal counter wraps around to 0 and deallocation is attempted)
> > --- this error is harder to reproduce and usually takes a while before
> > it happens in the code.
> >
> >
> > -Travis
> >
> > _______________________________________________
> > Numpy-discussion mailing list
> > Numpy-discussion at scipy.org
> > http://projects.scipy.org/mailman/listinfo/numpy-discussion
> >
>
>


-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071029/f10289a8/attachment.html>

From ckkart at hoc.net  Mon Oct 29 05:50:15 2007
From: ckkart at hoc.net (Christian K.)
Date: Mon, 29 Oct 2007 09:50:15 +0000 (UTC)
Subject: [Numpy-discussion]
	=?utf-8?q?Getting_an_item_in_an_array_with_its?=
	=?utf-8?q?=09coordinates_given_by_another_array?=
References: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
	<e76aa17f0710280759y19bdcb8ftb866e212c2f87a1e@mail.gmail.com>
	<loom.20071029T030633-253@post.gmane.org>
	<e76aa17f0710282346x143b35d2y7bd8ef120407d4f8@mail.gmail.com>
Message-ID: <loom.20071029T094150-315@post.gmane.org>

Matthieu Brucher <matthieu.brucher <at> gmail.com> writes:

> But if I have the coordinates of the points in an array, I have to reshape it
> and then convert it into a list. Or convert it into a list and then convert it
> to a tuple. I know that advanced indexing is useful, but here it is not
> coherent. tuples and lists should have the same result on the array, or at
> least it should be documented.

I can't give you the reasons why that behaviour was chosen. But it's simply
wrong that it's not documented. Before answering you I read the corresponding
chapter in the numpy book to be sure not to tell you nonsense. So go ahead and
do so, too.

> So there is not way to get a sub-array based on coordinates in an array?

It's just a guess, but probably you can't use an ndarray as index because
indices like this

[[2],[3,4]] 

aren't valid input for a ndarray.

Christian




From matthieu.brucher at gmail.com  Mon Oct 29 05:54:27 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 29 Oct 2007 10:54:27 +0100
Subject: [Numpy-discussion] Getting an item in an array with its
	coordinates given by another array
In-Reply-To: <loom.20071029T094150-315@post.gmane.org>
References: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
	<e76aa17f0710280759y19bdcb8ftb866e212c2f87a1e@mail.gmail.com>
	<loom.20071029T030633-253@post.gmane.org>
	<e76aa17f0710282346x143b35d2y7bd8ef120407d4f8@mail.gmail.com>
	<loom.20071029T094150-315@post.gmane.org>
Message-ID: <e76aa17f0710290254u28f6f195m650da4559adaa6d1@mail.gmail.com>

>
> > But if I have the coordinates of the points in an array, I have to
> reshape it
> > and then convert it into a list. Or convert it into a list and then
> convert it
> > to a tuple. I know that advanced indexing is useful, but here it is not
> > coherent. tuples and lists should have the same result on the array, or
> at
> > least it should be documented.
>
> I can't give you the reasons why that behaviour was chosen. But it's
> simply
> wrong that it's not documented. Before answering you I read the
> corresponding
> chapter in the numpy book to be sure not to tell you nonsense. So go ahead
> and
> do so, too.



Thank you for this answer, I suppose we'll have to wait the answer of a
numpy guru ;)

Matthieu
-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071029/7b402885/attachment.html>

From millman at berkeley.edu  Mon Oct 29 06:17:52 2007
From: millman at berkeley.edu (Jarrod Millman)
Date: Mon, 29 Oct 2007 03:17:52 -0700
Subject: [Numpy-discussion] numpy distutils patch
Message-ID: <c7009a550710290317o27f1f63y486fa5d30921a1fc@mail.gmail.com>

Hey,

I was looking at numpy/distutils/ccompiler.py and noticed that it has
a fix for distutils.util.split_quoted.

Here is the relevant code from split_quoted in numpy.distutils.ccompiler:
-----------------------------------
def split_quoted(s):

<snip>

           if _has_white_re.search(s[beg+1:end-1]):
                s = s[:beg] + s[beg+1:end-1] + s[end:]
                pos = m.end() - 2
            else:
                # Keeping quotes when a quoted word does not contain
                # white-space. XXX: send a patch to distutils
                pos = m.end()

<snip>
-----------------------------------

Here is the relevant code from split_quoted in distutils.util:
-----------------------------------
def split_quoted(s):

<snip>

            s = s[:beg] + s[beg+1:end-1] + s[end:]
            pos = m.end() - 2

<snip>
-----------------------------------

Does anyone know if a patch was ever submitted upstream?  If not, is
there any reason that a patch shouldn't be submitted?

Thanks,

-- 
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/


From tim.hochberg at ieee.org  Mon Oct 29 09:36:32 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Mon, 29 Oct 2007 06:36:32 -0700
Subject: [Numpy-discussion] Getting an item in an array with its
	coordinates given by another array
In-Reply-To: <e76aa17f0710282346x143b35d2y7bd8ef120407d4f8@mail.gmail.com>
References: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
	<e76aa17f0710280759y19bdcb8ftb866e212c2f87a1e@mail.gmail.com>
	<loom.20071029T030633-253@post.gmane.org>
	<e76aa17f0710282346x143b35d2y7bd8ef120407d4f8@mail.gmail.com>
Message-ID: <e4412d6b0710290636q5adf80a3mf8abe6c94c0b810d@mail.gmail.com>

On 10/28/07, Matthieu Brucher <matthieu.brucher at gmail.com> wrote:
>
> > Little correction, only c[(2,3)] gives me what I expect, not c[[2,3]],
> > which
> > > is even stranger.
> >
> > c[(2,3)] is the same as c[2,3] and obviously works as you expected.
>
>
>
> Well, this is not indicated in the documentation.
>

This is true at the Python level and is not related to numpy. "x=c[2,3]" is
equivalent to "x=c.__getitem__((2,3))".  Note that the index pair is passed
as a tuple. On the other hand, a single index is not passed as a tuple, but
is instead passed as is. For example: "x = c[a]" gets passed as
"x=c.__getitem__(a)". If 'a' happens to be '(2,3)' you get the behavior
above.

So, although lists and arrays can be used for "fancy-indexing", tuples
cannot be since you can't tell the difference between a tuple of indices and
multiple indices inside square brackets.


[SNIP]





-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071029/a546ce33/attachment.html>

From matthieu.brucher at gmail.com  Mon Oct 29 11:47:16 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 29 Oct 2007 16:47:16 +0100
Subject: [Numpy-discussion] More intel about the bug "Reference count error
	detected"
Message-ID: <e76aa17f0710290847y1dbcb96bg12de15121a65fc18@mail.gmail.com>

Hi,

Instead of using : xp = (x-self.loc) * (1/self.scale)
I've decided to use : xp = (x-self.loc) / self.scale

and by some miracle, the bug does not show up in almost all tests.
Before this bug, the program seems to be hung in a DOUBLE_substract
function() for quite some time (umathmodule.c file). Perhaps it is this
function that triggers to many references or dereferencing them too much ?
If someone wants to take a look, I may be able to wrap what is needed in a
single (but big) file.

Matthieu
-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071029/1cd28aa7/attachment.html>

From oliphant at enthought.com  Mon Oct 29 12:57:23 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Mon, 29 Oct 2007 11:57:23 -0500
Subject: [Numpy-discussion] Getting an item in an array with its
 coordinates given by another array
In-Reply-To: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
References: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
Message-ID: <472610F3.1030806@enthought.com>

Matthieu Brucher wrote:
> Hi,
>
> I'm trying to get an item in an array when I only have another array 
> giving the position.
>
> For instance:
>
> >>> c = numpy.arange(0., 3*4*5).reshape((3,4,5))
> >>> c***(numpy.array ((2,3), dtype=int))
> [55, 56, 57, 58, 59]
>
> I'm trying to figure what to put between c and ((2,3)). I supposed 
> that the take method would be what I wanted but it is not... The [] 
> operator does not work on array(but it seems to give the expected 
> result with a tuple or a list). Is there a method for this ?

What do you mean by "item" in an array?  Usually, when I refer to an 
"item" in the array, I'm talking about getting a particular element of 
the array. 

It would appear that you are trying to get a "sub-array" or "sub-space" 
of the array because you are using partial indexing. 

Getting a particular sub-array is the easiest thing to do with NumPy and 
only requires something like

c[2,3]

Presumably, you are trying to create the numbers on the fly and so want 
to know what to create.   The answer is that you should create a tuple 
of the numbers because as others have mentioned:

c[2,3] is equivalent to c[(2,3)] on the interpreter level (i.e. the 
numpy array cannot tell the difference between the two forms of syntax).

Almost all of the time a list is interpreted as "advanced indexing" 
(because there needed to be some way to distinguish the two).  I say 
"almost" because there is one exception to the rule that occurs if your 
list has less than or equal to the same number of elements as the number 
of dimensions of the array *and* contains slice objects, or Ellipsis, or 
newaxis.  In that case, the list will be interpreted as a tuple.

Thus, in your case you could do

c[[2,3,slice(None)]]  or
c[[2,3,Ellipsis]]

and get your desired result.   This (in my view) less than ideal 
behavior was created for backward compatibility to a pattern that had 
emerged for constructing indexing objects for multi-dimensional slicing.  

I'm pretty sure that numpy arrays of type integer are *always* 
interpreted as fancy indexing.

Let me know if there is more clarification needed.   The relevant code 
is in the function fancy_indexing_check in arrayobject.c of the source code.

Best regards,

-Travis O.


-Travis







>
> Matthieu
>
> -- 
> French PhD student
> Website : http://miles.developpez.com/
> Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From matthieu.brucher at gmail.com  Mon Oct 29 13:21:07 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 29 Oct 2007 18:21:07 +0100
Subject: [Numpy-discussion] Getting an item in an array with its
	coordinates given by another array
In-Reply-To: <472610F3.1030806@enthought.com>
References: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
	<472610F3.1030806@enthought.com>
Message-ID: <e76aa17f0710291021y582e8a38w1d1399ca2945d8bc@mail.gmail.com>

>
> What do you mean by "item" in an array?  Usually, when I refer to an
> "item" in the array, I'm talking about getting a particular element of
> the array.



In fact, I have a multi-dimensional array, and the last dimension is the
vector I want to get (I needed it for tests for my dense deformation field I
asked about on the scipy user list). So it's not a particular element, it's
a group of elements.


It would appear that you are trying to get a "sub-array" or "sub-space"
> of the array because you are using partial indexing.



Yes, that's what I really want to do in this case.


Let me know if there is more clarification needed.   The relevant code
> is in the function fancy_indexing_check in arrayobject.c of the source
> code.


Thank you, I have my explanation, but perhaps a totuple() method could be
added so that a double conversion is not needed anymore when an array is
used for indexing ?

Matthieu
-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071029/bfdd2261/attachment.html>

From Chris.Barker at noaa.gov  Mon Oct 29 12:44:19 2007
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Mon, 29 Oct 2007 09:44:19 -0700
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <47254304.5000805@ar.media.kyoto-u.ac.jp>
References: <472123B6.1050609@jpl.nasa.gov>
	<20071026063619.GC9025@clipper.ens.fr>
	<47218CD7.2000501@ar.media.kyoto-u.ac.jp>
	<bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>
	<47254304.5000805@ar.media.kyoto-u.ac.jp>
Message-ID: <47260DE3.5050701@noaa.gov>

 >> (incidently, the kind of things 'we' are doing seem like the most
> simple things to JIT).

Wouldn't a numpy-aware psyco be cool then?

Oh well, I'm not going to write it!

(though as I think about it, for the special case of a contiguous array, 
it would be awfully similar to an array.array --- hmmm.)

-CHB


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov


From tim.hochberg at ieee.org  Mon Oct 29 13:53:05 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Mon, 29 Oct 2007 10:53:05 -0700
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <47260DE3.5050701@noaa.gov>
References: <472123B6.1050609@jpl.nasa.gov>
	<20071026063619.GC9025@clipper.ens.fr>
	<47218CD7.2000501@ar.media.kyoto-u.ac.jp>
	<bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>
	<47254304.5000805@ar.media.kyoto-u.ac.jp> <47260DE3.5050701@noaa.gov>
Message-ID: <e4412d6b0710291053i6eca3dfdua4856eba7340de9c@mail.gmail.com>

On 10/29/07, Christopher Barker <Chris.Barker at noaa.gov> wrote:
>
> >> (incidently, the kind of things 'we' are doing seem like the most
> > simple things to JIT).
>
> Wouldn't a numpy-aware psyco be cool then?
>
> Oh well, I'm not going to write it!
>
> (though as I think about it, for the special case of a contiguous array,
> it would be awfully similar to an array.array --- hmmm.)


Psyco is aware of array.array and can operate on array.array's quite fast.
[In fact, somewhere I have a ndarray-like class based on Psyco that runs
pretty fast for integers, but not so fast for floats]. The problem that
Psyco has for numeric purposes is that it has no concept of floating point
numbers. It can "natively" store only a few different things: ints,
pointers, and arrays of ints or pointers. To put, for example, doubles, onto
the Psyco stack or into one of it's registers, you need to break the bits of
the double up, and stuff them into a couple of different int registers. Then
to operate on them you need to put them back together, since they may get
separated. All of this adds a fair amount of overhead.

I've been hoping that the PyPy jit will address this, but I haven't had time
to follow that project closely enough to see if that's on the agenda.



-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071029/7897dc35/attachment.html>

From oliphant at enthought.com  Mon Oct 29 14:20:48 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Mon, 29 Oct 2007 13:20:48 -0500
Subject: [Numpy-discussion] Getting an item in an array with
 its	coordinates given by another array
In-Reply-To: <e76aa17f0710291021y582e8a38w1d1399ca2945d8bc@mail.gmail.com>
References: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>	<472610F3.1030806@enthought.com>
	<e76aa17f0710291021y582e8a38w1d1399ca2945d8bc@mail.gmail.com>
Message-ID: <47262480.8050301@enthought.com>


>
> Thank you, I have my explanation, but perhaps a totuple() method could 
> be added so that a double conversion is not needed anymore when an 
> array is used for indexing ?
In this case the constructor

tuple(arr)

should work just fine. 

The tolist() method is really more useful for converting a 
multi-dimensional array to a list of lists (instead of a list of N-1 
dimensional arrays).


-Travis




From matthieu.brucher at gmail.com  Mon Oct 29 14:32:20 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Mon, 29 Oct 2007 19:32:20 +0100
Subject: [Numpy-discussion] Getting an item in an array with its
	coordinates given by another array
In-Reply-To: <47262480.8050301@enthought.com>
References: <e76aa17f0710280755s2e556f11w2d584ff50c33daa@mail.gmail.com>
	<472610F3.1030806@enthought.com>
	<e76aa17f0710291021y582e8a38w1d1399ca2945d8bc@mail.gmail.com>
	<47262480.8050301@enthought.com>
Message-ID: <e76aa17f0710291132i1a47537fp3c743a753f54a2d0@mail.gmail.com>

>
> In this case the constructor
>
> tuple(arr)
>
> should work just fine.
>

Sorry, I didn't know it could work (shame on me I should have tested).

-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071029/166e5eae/attachment.html>

From subscriber100 at rjs.org  Mon Oct 29 15:35:58 2007
From: subscriber100 at rjs.org (Ray S)
Date: Mon, 29 Oct 2007 11:35:58 -0800
Subject: [Numpy-discussion] numpy version and numpy.asarray behavior issue
Message-ID: <6.2.3.4.2.20071029112303.0591dfd8@blue-cove.com>

I have 2 PCs with 2 different installs:
ActivePython 2.4.3 Build 12 with numpy version '1.0b1'
and
Enthought 2.4.3 (1.0.0 #69)  with numpy version '0.9.7.2476'

The attached runs Ok on numpy v1.0, but on Enthought's,  print a1[0] 
gives:
IndexError: 0-d arrays can't be indexed.

It seems that the 0.9.7 numpy.asarray is not creating a true array 
from Dummy class in the code below. Enthought only comes with 
0.9.9.2706 (now).
When was the asarray behavior supported, or, is there some other 
issue I'm missing?
I'll use Activestate's distro if needed, but I'd like to keep 
Enthought for that one...



def fromaddress(address, dtype, shape, strides=None):
     """ Create a numpy array from an integer address, a dtype
     or dtype string, a shape tuple, and possibly strides.
     """
     import numpy
     # Make sure our dtype is a dtype, not just "f" or whatever.
     dtype = numpy.dtype(dtype)

     class Dummy(object):
         pass
     d = Dummy()
     d.__array_interface__ = dict(
         data = (address, False),
         typestr = dtype.str,
         descr = dtype.descr,
         shape = shape,
         strides = strides,
         version = 3,
     )
     return numpy.asarray(d)

Thanks,
Ray
-------------- next part --------------
""" nFromAddress.py
"""

def fromaddress(address, dtype, shape, strides=None):
    """ Create a numpy array from an integer address, a dtype 
    or dtype string, a shape tuple, and possibly strides.
    """
    import numpy
    # Make sure our dtype is a dtype, not just "f" or whatever.
    dtype = numpy.dtype(dtype)

    class Dummy(object):
        pass
    d = Dummy()
    d.__array_interface__ = dict(
        data = (address, False),
        typestr = dtype.str,
        descr = dtype.descr,
        shape = shape,
        strides = strides,
        version = 3,
    )
    return numpy.asarray(d)


##Numeric example, with address kludge
import Numeric, numpy, ctypes, string
a0 = Numeric.zeros((10000), Numeric.Int16)
nAddress = int(string.split(repr(a0.__copy__))[-1][:-1], 16)
tmp=(ctypes.c_long*1)(0)
ctypes.memmove(tmp, nAddress+8, 4)
nAddress = tmp[0]
a1 = fromaddress(nAddress, numpy.int16, (10000,))
a0[0] = 5
print a1[0]

## numpy example
a2 = numpy.zeros(10000, numpy.int16)
nAddress = a2.__array_interface__['data'][0]
nType = a2.__array_interface__['typestr']
nShape = a2.__array_interface__['shape']
a3 = fromaddress(nAddress, nType, nShape)
a2[0] = 5
print a3[0]

From subscriber100 at rjs.org  Mon Oct 29 16:54:49 2007
From: subscriber100 at rjs.org (Ray S)
Date: Mon, 29 Oct 2007 12:54:49 -0800
Subject: [Numpy-discussion] numpy version and numpy.asarray behavior issue
Message-ID: <6.2.3.4.2.20071029124527.059137b0@rjs.org>

I just installed 1.0.3.1 on top of Enthought's and asarray() works.

But...
Although the creation of an array from an address via a Dummy class 
is kosher in one process (as in the previous attachment), it fails 
across processes - the array is created, but gives a "Python has 
generated errors" window if the second process even attempts to read.

It can seem to work across processes with mmap.mmap() and tags (used 
in Windows)

def arrSharedMemory(shape, dtype, tag="PySharedMemory", access=None):
     ## Windows only !  share memory between different
     ## processes if same `tag` is used.
     itemsize = N.dtype(dtype).itemsize
     count = N.product(shape)
     size =  count * itemsize
     import mmap
     sharedmem = mmap.mmap(0, size, tag, access)
     a=N.frombuffer(sharedmem, dtype, count)
     a.shape = shape
     return a

I guess I'll use mmap unless someone can point out otherwise....

Thanks,
Ray
-------------- next part --------------
""" nFromAddress.py
"""

##Numeric example, with address kludge
import Numeric, numpy, ctypes, subprocess
from time import clock, sleep

"""a = Numeric.zeros((4), Numeric.Int16)
nAddress = int(repr(.__copy__).split()[-1][:-1], 16)
tmp=(ctypes.c_long*1)(0)
ctypes.memmove(tmp, nAddress+8, 4)
nAddress = tmp[0]
"""

a = numpy.zeros(4, numpy.int16)
nAddress = a.__array_interface__['data'][0]

print nAddress
pid = subprocess.Popen(
    [r'C:\python24\python.exe', 
     ['nFromAddress2.py '+str(nAddress)]
    ]).pid

while clock()<5:
    sleep(.1)
    if a[0]!=0: ## wait for a change...
        print a0[0]

-------------- next part --------------
""" nFromAddress.py
"""

import numpy
import time, sys
def fromaddress(address, dtype, shape, strides=None):
    """ Create a numpy array from an integer address, a dtype 
    or dtype string, a shape tuple, and possibly strides.
    """
    # Make sure our dtype is a dtype, not just "f" or whatever.
    dtype = numpy.dtype(dtype)

    class Dummy(object):
        pass
    d = Dummy()
    d.__array_interface__ = dict(
        data = (address, False),
        typestr = dtype.str,
        descr = dtype.descr,
        shape = shape,
        strides = strides,
        version = 3,
    )
    return numpy.asarray(d)


nAddress = sys.argv[1]
print 'recvd addr', nAddress
a3 = fromaddress(nAddress, numpy.int16, (4,))

## any of the following cause a Python/Windows error on access
print a3
#a3[0] = 5

From david at ar.media.kyoto-u.ac.jp  Mon Oct 29 23:20:12 2007
From: david at ar.media.kyoto-u.ac.jp (David Cournapeau)
Date: Tue, 30 Oct 2007 12:20:12 +0900
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <e4412d6b0710291053i6eca3dfdua4856eba7340de9c@mail.gmail.com>
References: <472123B6.1050609@jpl.nasa.gov>	<20071026063619.GC9025@clipper.ens.fr>	<47218CD7.2000501@ar.media.kyoto-u.ac.jp>	<bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>	<47254304.5000805@ar.media.kyoto-u.ac.jp>
	<47260DE3.5050701@noaa.gov>
	<e4412d6b0710291053i6eca3dfdua4856eba7340de9c@mail.gmail.com>
Message-ID: <4726A2EC.8070905@ar.media.kyoto-u.ac.jp>

Timothy Hochberg wrote:
>
>
> On 10/29/07, *Christopher Barker* <Chris.Barker at noaa.gov 
> <mailto:Chris.Barker at noaa.gov>> wrote:
>
>     >> (incidently, the kind of things 'we' are doing seem like the most
>     > simple things to JIT).
>
>     Wouldn't a numpy-aware psyco be cool then?
>
>     Oh well, I'm not going to write it!
>
>     (though as I think about it, for the special case of a contiguous
>     array,
>     it would be awfully similar to an array.array --- hmmm.)
>
>
> Psyco is aware of array.array and can operate on array.array's quite 
> fast. [In fact, somewhere I have a ndarray-like class based on Psyco 
> that runs pretty fast for integers, but not so fast for floats]. The 
> problem that Psyco has for numeric purposes is that it has no concept 
> of floating point numbers. It can "natively" store only a few 
> different things: ints, pointers, and arrays of ints or pointers. To 
> put, for example, doubles, onto the Psyco stack or into one of it's 
> registers, you need to break the bits of the double up, and stuff them 
> into a couple of different int registers. Then to operate on them you 
> need to put them back together, since they may get separated. All of 
> this adds a fair amount of overhead.
>
> I've been hoping that the PyPy jit will address this, but I haven't 
> had time to follow that project closely enough to see if that's on the 
> agenda.
My impression is that it is hard to follow pypy when you are not 'part 
of it', but just from the ML, and a couple of questions of mine, my 
understanding is that it is on the agenda. They have a JIT (maybe does 
not work on float, though), there is work on numpy arrays at the rpython 
level (rpython being the typed subset of python pypy is using at their 
core).

cheers,

David
>
>
>
> -- 
> .  __
> .   |-\
> .
> .  tim.hochberg at ieee.org <mailto:tim.hochberg at ieee.org>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>   



From matthieu.brucher at gmail.com  Tue Oct 30 04:46:45 2007
From: matthieu.brucher at gmail.com (Matthieu Brucher)
Date: Tue, 30 Oct 2007 09:46:45 +0100
Subject: [Numpy-discussion] More intel about the bug "Reference count
	error detected"
In-Reply-To: <e76aa17f0710290847y1dbcb96bg12de15121a65fc18@mail.gmail.com>
References: <e76aa17f0710290847y1dbcb96bg12de15121a65fc18@mail.gmail.com>
Message-ID: <e76aa17f0710300146w31cf3eceja9b651d9e8019008@mail.gmail.com>

Hi again,

I'm trying to debug what is going on, but I do not know much about the
universal functions. I'm trying to understand what PyUFunc_GenericFunction()
does and what the parameters should be.
Perhaps I should move to the numpy development list ?

Matthieu

2007/10/29, Matthieu Brucher <matthieu.brucher at gmail.com>:
>
> Hi,
>
> Instead of using : xp = (x-self.loc) * (1/self.scale)
> I've decided to use : xp = (x-self.loc) / self.scale
>
> and by some miracle, the bug does not show up in almost all tests.
> Before this bug, the program seems to be hung in a DOUBLE_substract
> function() for quite some time (umathmodule.c file). Perhaps it is this
> function that triggers to many references or dereferencing them too much ?
> If someone wants to take a look, I may be able to wrap what is needed in a
> single (but big) file.
>
> Matthieu
> --
> French PhD student
> Website : http://miles.developpez.com/
> Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92




-- 
French PhD student
Website : http://miles.developpez.com/
Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071030/f571a31d/attachment.html>

From charlesr.harris at gmail.com  Tue Oct 30 12:48:25 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Tue, 30 Oct 2007 10:48:25 -0600
Subject: [Numpy-discussion] MTL 4
Message-ID: <e06186140710300948j2d2de81fm72d39c777a2bfe3f@mail.gmail.com>

Just a note that an alpha version of the hitherto mostly invisible
MTL4 library has been released. It is template based and has automatic
bindings to blas. You can download it from
http://www.osl.iu.edu/research/mtl/mtl4/.

Chuck


From jdh2358 at gmail.com  Tue Oct 30 13:31:41 2007
From: jdh2358 at gmail.com (John Hunter)
Date: Tue, 30 Oct 2007 12:31:41 -0500
Subject: [Numpy-discussion] trailing max
Message-ID: <88e473830710301031x7dee0926v2dc0d720eb79ee2e@mail.gmail.com>

In financial time series, it is very common to keep track of things
like a trailing N day max, trailing N day average, etc.  Generally,
for a 1D array x, I'd like to be able to efficiently compute a new
len(x) vector where y[i] = func(x[i-N:])  and I need to be able to
handle edge effects (eg where i<N).  I'm thinking of writing some
extension code to handle this, but since this is a common task, I
thought I'd elicit some suggestions about how to best handle this.
One approach might to create a very large 2D array with each row the
appropriate masked (to handle edge effects) shifted array, and then
one could do sums and maxes over the columns.  For a length 1000
vector with a max lag of 250, you'd have 250K elements, which is
probably manageable, but I'm still not sure how to best create this
shifted array and if there are better approaches.

JDH


From pgmdevlist at gmail.com  Tue Oct 30 13:55:48 2007
From: pgmdevlist at gmail.com (Pierre GM)
Date: Tue, 30 Oct 2007 13:55:48 -0400
Subject: [Numpy-discussion] trailing max
In-Reply-To: <88e473830710301031x7dee0926v2dc0d720eb79ee2e@mail.gmail.com>
References: <88e473830710301031x7dee0926v2dc0d720eb79ee2e@mail.gmail.com>
Message-ID: <200710301355.49428.pgmdevlist@gmail.com>

On Tuesday 30 October 2007 13:31:41 John Hunter wrote:
> In financial time series, it is very common to keep track of things
> like a trailing N day max, trailing N day average, etc. 

John,
Have you ever tried the timeseries package in the scipy SVN ? We (Matt Knox 
and I) tried to address some of these issues for environmental and financial 
time series.
http://www.scipy.org/SciPyPackages/TimeSeries


From karol.langner at kn.pl  Tue Oct 30 20:09:08 2007
From: karol.langner at kn.pl (Karol Langner)
Date: Tue, 30 Oct 2007 20:09:08 -0400
Subject: [Numpy-discussion] trailing max
In-Reply-To: <200710301355.49428.pgmdevlist@gmail.com>
References: <88e473830710301031x7dee0926v2dc0d720eb79ee2e@mail.gmail.com>
	<200710301355.49428.pgmdevlist@gmail.com>
Message-ID: <200710302009.08877.karol.langner@kn.pl>

On Tuesday 30 October 2007 13:55, Pierre GM wrote:
> On Tuesday 30 October 2007 13:31:41 John Hunter wrote:
> > In financial time series, it is very common to keep track of things
> > like a trailing N day max, trailing N day average, etc.
>
> John,
> Have you ever tried the timeseries package in the scipy SVN ? We (Matt Knox
> and I) tried to address some of these issues for environmental and
> financial time series.
> http://www.scipy.org/SciPyPackages/TimeSeries

You might also want to look at ta-lib: http://ta-lib.org/

-- 
written by Karol Langner
Tue Oct 30 20:08:13 EDT 2007


From stefan at sun.ac.za  Tue Oct 30 17:05:50 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Tue, 30 Oct 2007 23:05:50 +0200
Subject: [Numpy-discussion] trailing max
In-Reply-To: <88e473830710301031x7dee0926v2dc0d720eb79ee2e@mail.gmail.com>
References: <88e473830710301031x7dee0926v2dc0d720eb79ee2e@mail.gmail.com>
Message-ID: <20071030210550.GB30790@mentat.za.net>

On Tue, Oct 30, 2007 at 12:31:41PM -0500, John Hunter wrote:
> In financial time series, it is very common to keep track of things
> like a trailing N day max, trailing N day average, etc.  Generally,
> for a 1D array x, I'd like to be able to efficiently compute a new
> len(x) vector where y[i] = func(x[i-N:])  and I need to be able to
> handle edge effects (eg where i<N).  I'm thinking of writing some
> extension code to handle this, but since this is a common task, I
> thought I'd elicit some suggestions about how to best handle this.
> One approach might to create a very large 2D array with each row the
> appropriate masked (to handle edge effects) shifted array, and then
> one could do sums and maxes over the columns.  For a length 1000
> vector with a max lag of 250, you'd have 250K elements, which is
> probably manageable, but I'm still not sure how to best create this
> shifted array and if there are better approaches.

A while ago Anne posted the attached 'segmentaxis'.  This is a very
useful piece of code!

Regards
St?fan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: segmentaxis.py
Type: text/x-python
Size: 5434 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071030/dc0212dd/attachment.py>

From mattknox_ca at hotmail.com  Tue Oct 30 18:51:53 2007
From: mattknox_ca at hotmail.com (Matt Knox)
Date: Tue, 30 Oct 2007 22:51:53 +0000 (UTC)
Subject: [Numpy-discussion] trailing max
References: <88e473830710301031x7dee0926v2dc0d720eb79ee2e@mail.gmail.com>
	<200710301355.49428.pgmdevlist@gmail.com>
Message-ID: <loom.20071030T224445-730@post.gmane.org>

Pierre GM <pgmdevlist <at> gmail.com> writes:

> 
> On Tuesday 30 October 2007 13:31:41 John Hunter wrote:
> > In financial time series, it is very common to keep track of things
> > like a trailing N day max, trailing N day average, etc. 
> 
> John,
> Have you ever tried the timeseries package in the scipy SVN ? We (Matt Knox 
> and I) tried to address some of these issues for environmental and financial 
> time series.
> http://www.scipy.org/SciPyPackages/TimeSeries
> 


I have just added a trailing max and trailing min function to the timeseries 
package to go along with the already existing trailing median function. These 
functions are implemented in C and attempt to be memory and computationally 
efficient, and should be much quicker than creating a 2D array as described in 
another post. Here is some example code...

>>> from timeseries.lib.moving_funcs import mov_max, mov_median, mov_min
>>> import numpy as np
>>> a = np.arange(10)
>>> print mov_max(a, 4)
[-- -- -- 3 4 5 6 7 8 9]
>>> print mov_median(a, 4)
[-- -- -- 1.5 2.5 3.5 4.5 5.5 6.5 7.5]
>>> print mov_min(a, 4)
[-- -- -- 0 1 2 3 4 5 6]


Note that the function returns a MaskedArray from the maskedarray package 
(currently in the sandbox), and maskedarray is a requirement for the 
timeseries package. You can pass TimeSeries objects, regular ndarray's, or 
MaskedArray's to these functions.

If you decide to give the timeseries package a shot, I'm happy to help with 
any questions/concerns you may have.

- Matt




From myeates at jpl.nasa.gov  Tue Oct 30 20:06:25 2007
From: myeates at jpl.nasa.gov (Mathew Yeates)
Date: Tue, 30 Oct 2007 17:06:25 -0700
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>
References: <472123B6.1050609@jpl.nasa.gov>	<20071026063619.GC9025@clipper.ens.fr>	<47218CD7.2000501@ar.media.kyoto-u.ac.jp>
	<bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>
Message-ID: <4727C701.2030104@jpl.nasa.gov>

Thanks all for your suggestions. I see it brought up a lot of 
interesting asides. But my loop was simple enough that I was able to 
figure it out.

I also took a look at NumExpr. While it wasn't something I needed for 
vectorizing, it still looks very interesting. What kinds of performance 
improvements would be expected using this?

Mathew

Sebastian Haase wrote:
> On 10/26/07, David Cournapeau <david at ar.media.kyoto-u.ac.jp> wrote:
>   
>> P.S: IMHO, this is one of the main limitation of numpy (or any language
>> using arrays for speed; and this is really difficult to optimize: you
>> need compilation, JIT or similar to solve those efficiently).
>>     
>
> This is where the scipy - sandbox numexpr  project comes in
> - if I'm not misaken ....
>
> http://www.scipy.org/SciPyPackages/NumExpr
>
> Description
> The scipy.sandbox.numexpr package supplies routines for the fast
> evaluation of array expressions elementwise by using a vector-based
> virtual machine. It's comparable to scipy.weave.blitz (in Weave), but
> doesn't require a separate compile step of C or C++ code.
>
>
> I hope that more noise around this  will result  in more interest and
> subsequentially result in more support.
> I think numexpr  might be one of the most powerful ideas in numpy /
> scipy  "recently".
> Did you know about numexpr - David ?
>
> Cheers,
> Sebastian Haase
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>   




From stefan at sun.ac.za  Wed Oct 31 01:55:32 2007
From: stefan at sun.ac.za (Stefan van der Walt)
Date: Wed, 31 Oct 2007 07:55:32 +0200
Subject: [Numpy-discussion] MTL 4
In-Reply-To: <e06186140710300948j2d2de81fm72d39c777a2bfe3f@mail.gmail.com>
References: <e06186140710300948j2d2de81fm72d39c777a2bfe3f@mail.gmail.com>
Message-ID: <20071031055532.GD30790@mentat.za.net>

On Tue, Oct 30, 2007 at 10:48:25AM -0600, Charles R Harris wrote:
> Just a note that an alpha version of the hitherto mostly invisible
> MTL4 library has been released. It is template based and has automatic
> bindings to blas. You can download it from
> http://www.osl.iu.edu/research/mtl/mtl4/.

I wonder why they are so paranoid with their licensing:

  5.  Products derived from this software may not be called "MTL", nor
      may "MTL" appear in their name, without prior written permission
      of Indiana University Advanced Research & Technology Institute.

No MTL4py, then :)

Cheers
St?fan


From aisaac at american.edu  Wed Oct 31 02:40:27 2007
From: aisaac at american.edu (Alan G Isaac)
Date: Wed, 31 Oct 2007 02:40:27 -0400
Subject: [Numpy-discussion] matrix problem: float to matrix power
Message-ID: <Mahogany-0.67.0-1280-20071031-024027.00@american.edu>

>>> 1.0**numpy.array([1,2,3])
array([ 1.,  1.,  1.])
>>> 1.0**numpy.mat([1,2,3])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for ** or pow(): 'float' and 'matrix'

Why the restriction for matrices?

Same question for matrices conformable for broadcasting:
why not produce the equivalent of the array result?

Cheers,
Alan Isaac





From charlesr.harris at gmail.com  Wed Oct 31 03:49:49 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 31 Oct 2007 01:49:49 -0600
Subject: [Numpy-discussion] matrix problem: float to matrix power
In-Reply-To: <Mahogany-0.67.0-1280-20071031-024027.00@american.edu>
References: <Mahogany-0.67.0-1280-20071031-024027.00@american.edu>
Message-ID: <e06186140710310049w7b9d1fa7x854b246efc1ea621@mail.gmail.com>

On 10/31/07, Alan G Isaac <aisaac at american.edu> wrote:
> >>> 1.0**numpy.array([1,2,3])
> array([ 1.,  1.,  1.])
> >>> 1.0**numpy.mat([1,2,3])
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: unsupported operand type(s) for ** or pow(): 'float' and 'matrix'
>
> Why the restriction for matrices?
>

Looks like it just got overlooked. It works fine for 2D arrays.

In [11]: 1.0**eye(3)
Out[11]:
array([[ 1.,  1.,  1.],
       [ 1.,  1.,  1.],
       [ 1.,  1.,  1.]])

Chuck


From faltet at carabos.com  Wed Oct 31 06:18:14 2007
From: faltet at carabos.com (Francesc Altet)
Date: Wed, 31 Oct 2007 11:18:14 +0100
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <4727C701.2030104@jpl.nasa.gov>
References: <472123B6.1050609@jpl.nasa.gov>
	<bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>
	<4727C701.2030104@jpl.nasa.gov>
Message-ID: <200710311118.15239.faltet@carabos.com>

A Wednesday 31 October 2007, Mathew Yeates escrigu?:
[...]
> I also took a look at NumExpr. While it wasn't something I needed for
> vectorizing, it still looks very interesting. What kinds of
> performance improvements would be expected using this?

Well, for some speedup figures you can always check the first (and very 
exciting) message (and many of the interesting followups) of the 
original author, David Cooke, in this very same list:

http://thread.gmane.org/gmane.comp.python.numeric.general/4081/focus=4138

This was back in March 2006.  Since then, David Cooke and Tim Hochberg 
(with some additions of Ivan Vilata and myself) have added many 
interesting functionality, like the support for logical and comparison 
operators, many functions, like 'where', 'sqrt' and all the 
trigonometric and inverse trigonometric, reductions ('sum' and 'prod') 
and also optimizations for some specific operations, like division 
(a/b) or exponentation (a**b) with floating points, among others.

I'm attaching a file (timing2-orig.out) with the results of the 
file 'timing.py' that comes with the numexpr distribution for more 
up-to-date figures (all the runs mentioned here have been executed on a 
Opteron at 2 GHz machine).  On it, you can see that the speed-ups for a 
variety of tests range from 1.1x up to 8.7x, being the mean speedup of 
aproximately 2x, which is pretty good.

Furthermore, and due to the specifics needs of Pytables (mainly, for 
doing 'in-core' selections in tables), Ivan and me have ended creating 
an enhanced version of numexpr that has improved support for more 
datatypes (long integers and strings, mainly), improvements for boolean 
operations and specific optimization for unidimensional arrays being 
strided and/or unaligned (these cases can be quite common in PyTables).  

You can see the result of these improvements in the 
attached 'boolean_timing-tables.out' file which is the output of the 
(also attached) benchmark 'boolean_timing.py' 
(see 'boolean_timing-orig.out' for the results using the original 
numexpr).  The main things to notice is that, for these kind 
of 'boolean' computations, the Pytables version of numexpr can be more 
than 3x faster than plain Numpy and up to 1.8x faster (2.0x for the 
unidimensional strided case) than the original numexpr.

Incidentally, all the improvements of the PyTables flavor of numexpr 
have been reported to the original authors, but, for the sake of 
keeping numexpr simple, they decided to implement only some of them.  
However, people is encouraged to try out the Pytables flavor from:

http://www.pytables.org/trac/browser/trunk/tables/numexpr/

Or, by installing PyTables 2.x series and importing numexpr as: 

"from tables import numexpr"

Cheers,

-- 
>0,0<   Francesc Altet ? ? http://www.carabos.com/
V   V   C?rabos Coop. V. ??Enjoy Data
 "-"
-------------- next part --------------
Expression: b*c+d*e
numpy: 0.0432945489883
Skipping weave timing
numexpr: 0.024453997612
Speed-up of numexpr over numpy: 1.77044872889

Expression: 2*a+3*b
numpy: 0.0386545658112
Skipping weave timing
numexpr: 0.019765496254
Speed-up of numexpr over numpy: 1.95565875577

Expression: 2*a + (cos(3)+5)*sinh(cos(b))
numpy: 0.271282076836
Skipping weave timing
numexpr: 0.244947910309
Speed-up of numexpr over numpy: 1.10750925163

Expression: 2*a + arctan2(a, b)
numpy: 0.136862039566
Skipping weave timing
numexpr: 0.120795488358
Speed-up of numexpr over numpy: 1.13300621925

Expression: a**2 + (b+1)**-2.5
numpy: 0.328014492989
Skipping weave timing
numexpr: 0.0521146059036
Speed-up of numexpr over numpy: 6.29409907839

Expression: (a+1)**50
numpy: 0.278326034546
Skipping weave timing
numexpr: 0.0682574510574
Speed-up of numexpr over numpy: 4.07759197324

Expression: sqrt(a**2 + b**2)
numpy: 0.0681229829788
Skipping weave timing
numexpr: 0.0376839637756
Speed-up of numexpr over numpy: 1.80774462539

Average = 1.64964169387
Expression: b*c+d*e
numpy: 0.0410040616989
Skipping weave timing
numexpr: 0.022455573082
Speed-up of numexpr over numpy: 1.82600824968

Expression: 2*a+3*b
numpy: 0.0371245145798
Skipping weave timing
numexpr: 0.0187830924988
Speed-up of numexpr over numpy: 1.97648574547

Expression: 2*a + (cos(3)+5)*sinh(cos(b))
numpy: 0.270690441132
Skipping weave timing
numexpr: 0.246992945671
Speed-up of numexpr over numpy: 1.09594401733

Expression: 2*a + arctan2(a, b)
numpy: 0.137767076492
Skipping weave timing
numexpr: 0.120716929436
Speed-up of numexpr over numpy: 1.14124072851

Expression: a**2 + (b+1)**-2.5
numpy: 0.331321954727
Skipping weave timing
numexpr: 0.0489059686661
Speed-up of numexpr over numpy: 6.77467318947

Expression: (a+1)**50
numpy: 0.27830851078
Skipping weave timing
numexpr: 0.0324734449387
Speed-up of numexpr over numpy: 8.57034143763

Expression: sqrt(a**2 + b**2)
numpy: 0.0671869516373
Skipping weave timing
numexpr: 0.0361994504929
Speed-up of numexpr over numpy: 1.8560213131

Average = 2.11279224374
Expression: b*c+d*e
numpy: 0.0418599843979
Skipping weave timing
numexpr: 0.0225175619125
Speed-up of numexpr over numpy: 1.85899275243

Expression: 2*a+3*b
numpy: 0.0368804931641
Skipping weave timing
numexpr: 0.0186915397644
Speed-up of numexpr over numpy: 1.97311155897

Expression: 2*a + (cos(3)+5)*sinh(cos(b))
numpy: 0.270949482918
Skipping weave timing
numexpr: 0.242916107178
Speed-up of numexpr over numpy: 1.11540352785

Expression: 2*a + arctan2(a, b)
numpy: 0.137181043625
Skipping weave timing
numexpr: 0.119929432869
Speed-up of numexpr over numpy: 1.14384801415

Expression: a**2 + (b+1)**-2.5
numpy: 0.328627943993
Skipping weave timing
numexpr: 0.0484969615936
Speed-up of numexpr over numpy: 6.77625841277

Expression: (a+1)**50
numpy: 0.278501987457
Skipping weave timing
numexpr: 0.0319650173187
Speed-up of numexpr over numpy: 8.7127119213

Expression: sqrt(a**2 + b**2)
numpy: 0.0678375959396
Skipping weave timing
numexpr: 0.0358920097351
Speed-up of numexpr over numpy: 1.89004729577

Average = 2.13367031666
Expression: b*c+d*e
numpy: 0.0413310527802
Skipping weave timing
numexpr: 0.0224200487137
Speed-up of numexpr over numpy: 1.84348630585

Expression: 2*a+3*b
numpy: 0.0371279716492
Skipping weave timing
numexpr: 0.0207829475403
Speed-up of numexpr over numpy: 1.78646323276

Expression: 2*a + (cos(3)+5)*sinh(cos(b))
numpy: 0.270565509796
Skipping weave timing
numexpr: 0.245254993439
Speed-up of numexpr over numpy: 1.1032008197

Expression: 2*a + arctan2(a, b)
numpy: 0.137845516205
Skipping weave timing
numexpr: 0.120264410973
Speed-up of numexpr over numpy: 1.146187098

Expression: a**2 + (b+1)**-2.5
numpy: 0.327829003334
Skipping weave timing
numexpr: 0.0490430593491
Speed-up of numexpr over numpy: 6.68451372498

Expression: (a+1)**50
numpy: 0.27824151516
Skipping weave timing
numexpr: 0.0320014953613
Speed-up of numexpr over numpy: 8.6946410478

Expression: sqrt(a**2 + b**2)
numpy: 0.0679360628128
Skipping weave timing
numexpr: 0.0359455347061
Speed-up of numexpr over numpy: 1.88997224184

Average = 2.10440586099
Expression: b*c+d*e
numpy: 0.0416020154953
Skipping weave timing
numexpr: 0.0223726034164
Speed-up of numexpr over numpy: 1.85950712668

Expression: 2*a+3*b
numpy: 0.0368609428406
Skipping weave timing
numexpr: 0.0182800292969
Speed-up of numexpr over numpy: 2.01645972454

Expression: 2*a + (cos(3)+5)*sinh(cos(b))
numpy: 0.271070599556
Skipping weave timing
numexpr: 0.246081471443
Speed-up of numexpr over numpy: 1.10154819039

Expression: 2*a + arctan2(a, b)
numpy: 0.136904478073
Skipping weave timing
numexpr: 0.119737029076
Speed-up of numexpr over numpy: 1.14337627324

Expression: a**2 + (b+1)**-2.5
numpy: 0.328311920166
Skipping weave timing
numexpr: 0.0483285188675
Speed-up of numexpr over numpy: 6.79333709908

Expression: (a+1)**50
numpy: 0.278018951416
Skipping weave timing
numexpr: 0.0320415496826
Speed-up of numexpr over numpy: 8.67682600155

Expression: sqrt(a**2 + b**2)
numpy: 0.0675255060196
Skipping weave timing
numexpr: 0.0356780290604
Speed-up of numexpr over numpy: 1.89263554624

Average = 2.13488090561
Expression: b*c+d*e
numpy: 0.0417284965515
Skipping weave timing
numexpr: 0.0224235057831
Speed-up of numexpr over numpy: 1.86092651859

Expression: 2*a+3*b
numpy: 0.0372844934464
Skipping weave timing
numexpr: 0.0183354616165
Speed-up of numexpr over numpy: 2.03346358145

Expression: 2*a + (cos(3)+5)*sinh(cos(b))
numpy: 0.271299004555
Skipping weave timing
numexpr: 0.242529511452
Speed-up of numexpr over numpy: 1.11862264898

Expression: 2*a + arctan2(a, b)
numpy: 0.137305498123
Skipping weave timing
numexpr: 0.11990404129
Speed-up of numexpr over numpy: 1.14512819289

Expression: a**2 + (b+1)**-2.5
numpy: 0.328257918358
Skipping weave timing
numexpr: 0.0484659671783
Speed-up of numexpr over numpy: 6.77295713815

Expression: (a+1)**50
numpy: 0.278239488602
Skipping weave timing
numexpr: 0.0320734977722
Speed-up of numexpr over numpy: 8.67505909638

Expression: sqrt(a**2 + b**2)
numpy: 0.0679830312729
Skipping weave timing
numexpr: 0.0358599424362
Speed-up of numexpr over numpy: 1.89579309542

Average = 2.13654093381
Expression: b*c+d*e
numpy: 0.0416275262833
Skipping weave timing
numexpr: 0.0223845243454
Speed-up of numexpr over numpy: 1.85965650379

Expression: 2*a+3*b
numpy: 0.0367059707642
Skipping weave timing
numexpr: 0.0184335708618
Speed-up of numexpr over numpy: 1.99125666098

Expression: 2*a + (cos(3)+5)*sinh(cos(b))
numpy: 0.27404999733
Skipping weave timing
numexpr: 0.245867967606
Speed-up of numexpr over numpy: 1.11462261635

Expression: 2*a + arctan2(a, b)
numpy: 0.137671470642
Skipping weave timing
numexpr: 0.120179057121
Speed-up of numexpr over numpy: 1.14555292694

Expression: a**2 + (b+1)**-2.5
numpy: 0.328163504601
Skipping weave timing
numexpr: 0.0487785339355
Speed-up of numexpr over numpy: 6.72762131462

Expression: (a+1)**50
numpy: 0.278432488441
Skipping weave timing
numexpr: 0.0322035551071
Speed-up of numexpr over numpy: 8.64601710946

Expression: sqrt(a**2 + b**2)
numpy: 0.0673484802246
Skipping weave timing
numexpr: 0.0359764099121
Speed-up of numexpr over numpy: 1.87201781359

Average = 2.12334044961
Expression: b*c+d*e
numpy: 0.0412619113922
Skipping weave timing
numexpr: 0.0225330591202
Speed-up of numexpr over numpy: 1.8311721978

Expression: 2*a+3*b
numpy: 0.0367000102997
Skipping weave timing
numexpr: 0.0186544656754
Speed-up of numexpr over numpy: 1.96735789373

Expression: 2*a + (cos(3)+5)*sinh(cos(b))
numpy: 0.269914507866
Skipping weave timing
numexpr: 0.242934584618
Speed-up of numexpr over numpy: 1.11105838755

Expression: 2*a + arctan2(a, b)
numpy: 0.137315988541
Skipping weave timing
numexpr: 0.119785428047
Speed-up of numexpr over numpy: 1.14634969194

Expression: a**2 + (b+1)**-2.5
numpy: 0.328106999397
Skipping weave timing
numexpr: 0.0484600067139
Speed-up of numexpr over numpy: 6.77067589641

Expression: (a+1)**50
numpy: 0.27797794342
Skipping weave timing
numexpr: 0.0319224596024
Speed-up of numexpr over numpy: 8.70791119742

Expression: sqrt(a**2 + b**2)
numpy: 0.0679694414139
Skipping weave timing
numexpr: 0.0358340740204
Speed-up of numexpr over numpy: 1.89678241372

Average = 2.13011887987
Expression: b*c+d*e
numpy: 0.0415569543839
Skipping weave timing
numexpr: 0.0224959850311
Speed-up of numexpr over numpy: 1.84730538922

Expression: 2*a+3*b
numpy: 0.0367209911346
Skipping weave timing
numexpr: 0.0182545185089
Speed-up of numexpr over numpy: 2.01161104944

Expression: 2*a + (cos(3)+5)*sinh(cos(b))
numpy: 0.270547986031
Skipping weave timing
numexpr: 0.243052005768
Speed-up of numexpr over numpy: 1.11312797101

Expression: 2*a + arctan2(a, b)
numpy: 0.137174010277
Skipping weave timing
numexpr: 0.119675517082
Speed-up of numexpr over numpy: 1.14621614864

Expression: a**2 + (b+1)**-2.5
numpy: 0.327585458755
Skipping weave timing
numexpr: 0.0484219789505
Speed-up of numexpr over numpy: 6.76522244352

Expression: (a+1)**50
numpy: 0.278404474258
Skipping weave timing
numexpr: 0.0317900180817
Speed-up of numexpr over numpy: 8.75760666582

Expression: sqrt(a**2 + b**2)
numpy: 0.0682315826416
Skipping weave timing
numexpr: 0.0355939865112
Speed-up of numexpr over numpy: 1.91694129625

Average = 2.14163917854
Expression: b*c+d*e
numpy: 0.0415704250336
Skipping weave timing
numexpr: 0.0228176116943
Speed-up of numexpr over numpy: 1.82185697567

Expression: 2*a+3*b
numpy: 0.0368599891663
Skipping weave timing
numexpr: 0.0182430744171
Speed-up of numexpr over numpy: 2.02049217821

Expression: 2*a + (cos(3)+5)*sinh(cos(b))
numpy: 0.269914984703
Skipping weave timing
numexpr: 0.246359467506
Speed-up of numexpr over numpy: 1.09561441837

Expression: 2*a + arctan2(a, b)
numpy: 0.137582421303
Skipping weave timing
numexpr: 0.120151996613
Speed-up of numexpr over numpy: 1.14506978812

Expression: a**2 + (b+1)**-2.5
numpy: 0.328064441681
Skipping weave timing
numexpr: 0.0486860275269
Speed-up of numexpr over numpy: 6.73836947366

Expression: (a+1)**50
numpy: 0.278306007385
Skipping weave timing
numexpr: 0.032343506813
Speed-up of numexpr over numpy: 8.60469487721

Expression: sqrt(a**2 + b**2)
numpy: 0.0673480033875
Skipping weave timing
numexpr: 0.0361359119415
Speed-up of numexpr over numpy: 1.863741629

Average = 2.11725812184
Averages: 18.15, 23.24, 23.47, 23.15, 23.48, 23.50, 23.36, 23.43, 23.56, 23.29
-------------- next part --------------
Python version:    2.5 (r25:51908, Nov  3 2006, 12:01:01) 
[GCC 4.0.2 20050901 (prerelease) (SUSE Linux)]
NumPy version:     1.0.3
******************* Expression: i2 > 0
numpy: 0.00318
numpy strided: 0.00695
numpy unaligned: 0.01249
numexpr: 0.02332 Speed-up of numexpr over numpy: 0.1364
numexpr strided: 0.02728 Speed-up of numexpr strided over numpy: 0.2548
numexpr unaligned: 0.02604 Speed-up of numexpr unaligned over numpy: 0.4796
******************* Expression: i2 < 0
numpy: 0.00309
numpy strided: 0.00698
numpy unaligned: 0.01153
numexpr: 0.02358 Speed-up of numexpr over numpy: 0.131
numexpr strided: 0.02708 Speed-up of numexpr strided over numpy: 0.2578
numexpr unaligned: 0.02599 Speed-up of numexpr unaligned over numpy: 0.4436
******************* Expression: i2 < f3
numpy: 0.00872
numpy strided: 0.01488
numpy unaligned: 0.02254
numexpr: 0.02757 Speed-up of numexpr over numpy: 0.3163
numexpr strided: 0.03473 Speed-up of numexpr strided over numpy: 0.4284
numexpr unaligned: 0.04398 Speed-up of numexpr unaligned over numpy: 0.5125
******************* Expression: i2-10 < f3
numpy: 0.01517
numpy strided: 0.02121
numpy unaligned: 0.0332
numexpr: 0.02946 Speed-up of numexpr over numpy: 0.5149
numexpr strided: 0.0375 Speed-up of numexpr strided over numpy: 0.5656
numexpr unaligned: 0.04521 Speed-up of numexpr unaligned over numpy: 0.7344
******************* Expression: i2*f3+f3*f3 > i2
numpy: 0.05012
numpy strided: 0.06811
numpy unaligned: 0.09033
numexpr: 0.0344 Speed-up of numexpr over numpy: 1.457
numexpr strided: 0.04196 Speed-up of numexpr strided over numpy: 1.6232
numexpr unaligned: 0.04948 Speed-up of numexpr unaligned over numpy: 1.8256
******************* Expression: 0.1*i2 > arctan2(i2, f3)
numpy: 0.13276
numpy strided: 0.14845
numpy unaligned: 0.15729
numexpr: 0.12835 Speed-up of numexpr over numpy: 1.0344
numexpr strided: 0.1368 Speed-up of numexpr strided over numpy: 1.0852
numexpr unaligned: 0.14563 Speed-up of numexpr unaligned over numpy: 1.0801
******************* Expression: i2%2 > 3
numpy: 0.03263
numpy strided: 0.03386
numpy unaligned: 0.04204
numexpr: 0.06113 Speed-up of numexpr over numpy: 0.5338
numexpr strided: 0.06615 Speed-up of numexpr strided over numpy: 0.5119
numexpr unaligned: 0.06473 Speed-up of numexpr unaligned over numpy: 0.6495
******************* Expression: i2%10 < 4
numpy: 0.0327
numpy strided: 0.03373
numpy unaligned: 0.04215
numexpr: 0.06154 Speed-up of numexpr over numpy: 0.5314
numexpr strided: 0.06539 Speed-up of numexpr strided over numpy: 0.5158
numexpr unaligned: 0.06492 Speed-up of numexpr unaligned over numpy: 0.6493
******************* Expression: i2**2 + (f3+1)**-2.5 < 3
numpy: 0.32493
numpy strided: 0.33087
numpy unaligned: 0.35109
numexpr: 0.05999 Speed-up of numexpr over numpy: 5.4164
numexpr strided: 0.06683 Speed-up of numexpr strided over numpy: 4.9509
numexpr unaligned: 0.07455 Speed-up of numexpr unaligned over numpy: 4.7095
******************* Expression: (f3+1)**50 > i2
numpy: 0.30474
numpy strided: 0.31431
numpy unaligned: 0.3192
numexpr: 0.05154 Speed-up of numexpr over numpy: 5.9127
numexpr strided: 0.06164 Speed-up of numexpr strided over numpy: 5.0991
numexpr unaligned: 0.06035 Speed-up of numexpr unaligned over numpy: 5.2891
******************* Expression: sqrt(i2**2 + f3**2) > 1
numpy: 0.06741
numpy strided: 0.07349
numpy unaligned: 0.09405
numexpr: 0.04676 Speed-up of numexpr over numpy: 1.4416
numexpr strided: 0.05361 Speed-up of numexpr strided over numpy: 1.3708
numexpr unaligned: 0.0615 Speed-up of numexpr unaligned over numpy: 1.5293
******************* Expression: (i2>2) | ((f3**2>3) & ~(i2*f3<2))
numpy: 0.05085
numpy strided: 0.06494
numpy unaligned: 0.08386
numexpr: 0.04527 Speed-up of numexpr over numpy: 1.1233
numexpr strided: 0.05323 Speed-up of numexpr strided over numpy: 1.22
numexpr unaligned: 0.06087 Speed-up of numexpr unaligned over numpy: 1.3777

*************** TOTALS **************************
numpy total: 1.0263
numpy strided total: 1.11778
numpy unaligned total: 1.25977
numexpr total: 0.59291
Speed-up of numexpr over numpy: 1.731
numexpr strided total: 0.6722
Speed-up of numexpr strided over numpy: 1.663
numexpr unaligned total: 0.72325
Speed-up of numexpr unaligned over numpy: 1.742
-------------- next part --------------
Python version:    2.5 (r25:51908, Nov  3 2006, 12:01:01) 
[GCC 4.0.2 20050901 (prerelease) (SUSE Linux)]
NumPy version:     1.0.3
******************* Expression: i2 > 0
numpy: 0.00315
numpy strided: 0.007
numpy unaligned: 0.01252
numexpr: 0.00385 Speed-up of numexpr over numpy: 0.8182
numexpr strided: 0.00738 Speed-up of numexpr strided over numpy: 0.9485
numexpr unaligned: 0.00886 Speed-up of numexpr unaligned over numpy: 1.4131
******************* Expression: i2 < 0
numpy: 0.00316
numpy strided: 0.00699
numpy unaligned: 0.01246
numexpr: 0.00377 Speed-up of numexpr over numpy: 0.8382
numexpr strided: 0.00742 Speed-up of numexpr strided over numpy: 0.942
numexpr unaligned: 0.00886 Speed-up of numexpr unaligned over numpy: 1.4063
******************* Expression: i2 < f3
numpy: 0.00876
numpy strided: 0.01584
numpy unaligned: 0.02571
numexpr: 0.00821 Speed-up of numexpr over numpy: 1.067
numexpr strided: 0.01 Speed-up of numexpr strided over numpy: 1.584
numexpr unaligned: 0.01348 Speed-up of numexpr unaligned over numpy: 1.9073
******************* Expression: i2-10 < f3
numpy: 0.01578
numpy strided: 0.02121
numpy unaligned: 0.03416
numexpr: 0.01004 Speed-up of numexpr over numpy: 1.5717
numexpr strided: 0.01181 Speed-up of numexpr strided over numpy: 1.7959
numexpr unaligned: 0.0153 Speed-up of numexpr unaligned over numpy: 2.2327
******************* Expression: i2*f3+f3*f3 > i2
numpy: 0.05065
numpy strided: 0.06868
numpy unaligned: 0.09667
numexpr: 0.01507 Speed-up of numexpr over numpy: 3.361
numexpr strided: 0.01651 Speed-up of numexpr strided over numpy: 4.1599
numexpr unaligned: 0.02024 Speed-up of numexpr unaligned over numpy: 4.7762
******************* Expression: 0.1*i2 > arctan2(i2, f3)
numpy: 0.13315
numpy strided: 0.14904
numpy unaligned: 0.1584
numexpr: 0.10897 Speed-up of numexpr over numpy: 1.2219
numexpr strided: 0.11178 Speed-up of numexpr strided over numpy: 1.3333
numexpr unaligned: 0.1155 Speed-up of numexpr unaligned over numpy: 1.3714
******************* Expression: i2%2 > 3
numpy: 0.03276
numpy strided: 0.03383
numpy unaligned: 0.04211
numexpr: 0.02611 Speed-up of numexpr over numpy: 1.2547
numexpr strided: 0.02713 Speed-up of numexpr strided over numpy: 1.247
numexpr unaligned: 0.03105 Speed-up of numexpr unaligned over numpy: 1.3562
******************* Expression: i2%10 < 4
numpy: 0.03251
numpy strided: 0.03397
numpy unaligned: 0.04227
numexpr: 0.02618 Speed-up of numexpr over numpy: 1.2418
numexpr strided: 0.02714 Speed-up of numexpr strided over numpy: 1.2517
numexpr unaligned: 0.03103 Speed-up of numexpr unaligned over numpy: 1.3622
******************* Expression: i2**2 + (f3+1)**-2.5 < 3
numpy: 0.32202
numpy strided: 0.32757
numpy unaligned: 0.34797
numexpr: 0.04053 Speed-up of numexpr over numpy: 7.9452
numexpr strided: 0.04199 Speed-up of numexpr strided over numpy: 7.8011
numexpr unaligned: 0.04577 Speed-up of numexpr unaligned over numpy: 7.6026
******************* Expression: (f3+1)**50 > i2
numpy: 0.30238
numpy strided: 0.3119
numpy unaligned: 0.31825
numexpr: 0.03251 Speed-up of numexpr over numpy: 9.3011
numexpr strided: 0.02719 Speed-up of numexpr strided over numpy: 11.4711
numexpr unaligned: 0.04017 Speed-up of numexpr unaligned over numpy: 7.9226
******************* Expression: sqrt(i2**2 + f3**2) > 1
numpy: 0.06747
numpy strided: 0.07415
numpy unaligned: 0.09568
numexpr: 0.02703 Speed-up of numexpr over numpy: 2.4961
numexpr strided: 0.02877 Speed-up of numexpr strided over numpy: 2.5773
numexpr unaligned: 0.0321 Speed-up of numexpr unaligned over numpy: 2.9807
******************* Expression: (i2>2) | ((f3**2>3) & ~(i2*f3<2))
numpy: 0.05395
numpy strided: 0.06548
numpy unaligned: 0.08676
numexpr: 0.02541 Speed-up of numexpr over numpy: 2.1232
numexpr strided: 0.02681 Speed-up of numexpr strided over numpy: 2.4424
numexpr unaligned: 0.03101 Speed-up of numexpr unaligned over numpy: 2.7978

*************** TOTALS **************************
numpy total: 1.02574
numpy strided total: 1.11566
numpy unaligned total: 1.27296
numexpr total: 0.32768
Speed-up of numexpr over numpy: 3.13
numexpr strided total: 0.34393
Speed-up of numexpr strided over numpy: 3.244
numexpr unaligned total: 0.39337
Speed-up of numexpr unaligned over numpy: 3.236
-------------- next part --------------
A non-text attachment was scrubbed...
Name: boolean_timing.py
Type: application/x-python
Size: 4606 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20071031/ebcdc934/attachment.bin>

From schut at sarvision.nl  Wed Oct 31 12:20:33 2007
From: schut at sarvision.nl (Vincent Schut)
Date: Wed, 31 Oct 2007 17:20:33 +0100
Subject: [Numpy-discussion] strange problem compiling against ndarrayobject.h
Message-ID: <4728AB51.6050601@sarvision.nl>

Btw forgot some info:

gentoo linux, amd64
numpy 1.0.3.1
gdal from svn, updated today
python 2.5.1

Cheers,
Vincent.



From tim.hochberg at ieee.org  Wed Oct 31 12:55:26 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 31 Oct 2007 09:55:26 -0700
Subject: [Numpy-discussion] matrix problem: float to matrix power
In-Reply-To: <Mahogany-0.67.0-1280-20071031-024027.00@american.edu>
References: <Mahogany-0.67.0-1280-20071031-024027.00@american.edu>
Message-ID: <e4412d6b0710310955y6d9127d7x627984897ac83cf@mail.gmail.com>

On Oct 30, 2007 11:40 PM, Alan G Isaac <aisaac at american.edu> wrote:
> >>> 1.0**numpy.array([1,2,3])
> array([ 1.,  1.,  1.])
> >>> 1.0**numpy.mat([1,2,3])
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: unsupported operand type(s) for ** or pow(): 'float' and 'matrix'
>
> Why the restriction for matrices?
>
> Same question for matrices conformable for broadcasting:
> why not produce the equivalent of the array result?

With the caveat that I don't use the matrix class: I believe it's
because M**n results in the matrix power of n. It would be confusing
if n**M did a broadcast element wise power.


-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org


From robert.kern at gmail.com  Wed Oct 31 13:01:15 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 31 Oct 2007 12:01:15 -0500
Subject: [Numpy-discussion] matrix problem: float to matrix power
In-Reply-To: <e06186140710310049w7b9d1fa7x854b246efc1ea621@mail.gmail.com>
References: <Mahogany-0.67.0-1280-20071031-024027.00@american.edu>
	<e06186140710310049w7b9d1fa7x854b246efc1ea621@mail.gmail.com>
Message-ID: <4728B4DB.6080206@gmail.com>

Charles R Harris wrote:
> On 10/31/07, Alan G Isaac <aisaac at american.edu> wrote:
>>>>> 1.0**numpy.array([1,2,3])
>> array([ 1.,  1.,  1.])
>>>>> 1.0**numpy.mat([1,2,3])
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> TypeError: unsupported operand type(s) for ** or pow(): 'float' and 'matrix'
>>
>> Why the restriction for matrices?
> 
> Looks like it just got overlooked. It works fine for 2D arrays.
> 
> In [11]: 1.0**eye(3)
> Out[11]:
> array([[ 1.,  1.,  1.],
>        [ 1.,  1.,  1.],
>        [ 1.,  1.,  1.]])

For ndarrays, it does elementwise exponentiation, so it is neither here nor
there with respect to matrices. What is the definition of a scalar raised to a
matrix power? I don't ever recall seeing one.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From charlesr.harris at gmail.com  Wed Oct 31 13:26:46 2007
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Wed, 31 Oct 2007 11:26:46 -0600
Subject: [Numpy-discussion] matrix problem: float to matrix power
In-Reply-To: <4728B4DB.6080206@gmail.com>
References: <Mahogany-0.67.0-1280-20071031-024027.00@american.edu>
	<e06186140710310049w7b9d1fa7x854b246efc1ea621@mail.gmail.com>
	<4728B4DB.6080206@gmail.com>
Message-ID: <e06186140710311026s46b34bedq7194f0410aa18a1d@mail.gmail.com>

On 10/31/07, Robert Kern <robert.kern at gmail.com> wrote:
> Charles R Harris wrote:
> > On 10/31/07, Alan G Isaac <aisaac at american.edu> wrote:
> >>>>> 1.0**numpy.array([1,2,3])
> >> array([ 1.,  1.,  1.])
> >>>>> 1.0**numpy.mat([1,2,3])
> >> Traceback (most recent call last):
> >>   File "<stdin>", line 1, in <module>
> >> TypeError: unsupported operand type(s) for ** or pow(): 'float' and 'matrix'
> >>
> >> Why the restriction for matrices?
> >
> > Looks like it just got overlooked. It works fine for 2D arrays.
> >
> > In [11]: 1.0**eye(3)
> > Out[11]:
> > array([[ 1.,  1.,  1.],
> >        [ 1.,  1.,  1.],
> >        [ 1.,  1.,  1.]])
>
> For ndarrays, it does elementwise exponentiation, so it is neither here nor
> there with respect to matrices. What is the definition of a scalar raised to a
> matrix power? I don't ever recall seeing one.
>

I suppose the usual:

a**m = exp(ln(a)*m)

but that isn't the broadcast version.

Chuck


From robert.kern at gmail.com  Wed Oct 31 13:34:43 2007
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 31 Oct 2007 12:34:43 -0500
Subject: [Numpy-discussion] matrix problem: float to matrix power
In-Reply-To: <e06186140710311026s46b34bedq7194f0410aa18a1d@mail.gmail.com>
References: <Mahogany-0.67.0-1280-20071031-024027.00@american.edu>	<e06186140710310049w7b9d1fa7x854b246efc1ea621@mail.gmail.com>	<4728B4DB.6080206@gmail.com>
	<e06186140710311026s46b34bedq7194f0410aa18a1d@mail.gmail.com>
Message-ID: <4728BCB3.5060308@gmail.com>

Charles R Harris wrote:
> On 10/31/07, Robert Kern <robert.kern at gmail.com> wrote:
>> Charles R Harris wrote:
>>> On 10/31/07, Alan G Isaac <aisaac at american.edu> wrote:
>>>>>>> 1.0**numpy.array([1,2,3])
>>>> array([ 1.,  1.,  1.])
>>>>>>> 1.0**numpy.mat([1,2,3])
>>>> Traceback (most recent call last):
>>>>   File "<stdin>", line 1, in <module>
>>>> TypeError: unsupported operand type(s) for ** or pow(): 'float' and 'matrix'
>>>>
>>>> Why the restriction for matrices?
>>> Looks like it just got overlooked. It works fine for 2D arrays.
>>>
>>> In [11]: 1.0**eye(3)
>>> Out[11]:
>>> array([[ 1.,  1.,  1.],
>>>        [ 1.,  1.,  1.],
>>>        [ 1.,  1.,  1.]])
>> For ndarrays, it does elementwise exponentiation, so it is neither here nor
>> there with respect to matrices. What is the definition of a scalar raised to a
>> matrix power? I don't ever recall seeing one.
> 
> I suppose the usual:
> 
> a**m = exp(ln(a)*m)
> 
> but that isn't the broadcast version.

Ah, yes. Of course. How silly of me.

That raises the question of how to calculate the bloody thing, though. We have
three implementations of expm() in scipy.linalg.matfuncs.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco


From tim.hochberg at ieee.org  Wed Oct 31 14:51:37 2007
From: tim.hochberg at ieee.org (Timothy Hochberg)
Date: Wed, 31 Oct 2007 11:51:37 -0700
Subject: [Numpy-discussion] vectorizing loops
In-Reply-To: <200710311118.15239.faltet@carabos.com>
References: <472123B6.1050609@jpl.nasa.gov>
	<bc657ead0710260506s4213b0b5wc811e756becfcaee@mail.gmail.com>
	<4727C701.2030104@jpl.nasa.gov>
	<200710311118.15239.faltet@carabos.com>
Message-ID: <e4412d6b0710311151o2e5486e4kfa72ca077a0949da@mail.gmail.com>

On Oct 31, 2007 3:18 AM, Francesc Altet <faltet at carabos.com> wrote:

[SNIP]

> Incidentally, all the improvements of the PyTables flavor of numexpr
> have been reported to the original authors, but, for the sake of
> keeping numexpr simple, they decided to implement only some of them.
> However, people is encouraged to try out the Pytables flavor from:

My original concern was that we'd start overflowing the code cache and
slow things down. Some experiments seemed to confirm this on some
processors, but it then turned out those were in error. At least
that's my recollection. Because of that, and because PyTables is, as
far as I know, the major user of numexpr, I suspect I'd be fine
putting those changes in now. I say "suspect" since it's been a long
time since I looked at the relevant patches, and I should probably
look at those again before I commit myself. I just haven't had the
free time and motivation to go back and look at the patches.  I can't
speak for David though.




-- 
.  __
.   |-\
.
.  tim.hochberg at ieee.org


From subscriber100 at rjs.org  Wed Oct 31 18:58:32 2007
From: subscriber100 at rjs.org (Ray S)
Date: Wed, 31 Oct 2007 14:58:32 -0800
Subject: [Numpy-discussion] numpy FFT memory accumulation
Message-ID: <6.2.3.4.2.20071031133112.058c23d8@blue-cove.com>

I am using
fftRes = abs(fft.rfft(data_array[end-2**15:end]))
to do running analysis on streaming data. The N never changes.
It sucks memory up at ~1MB/sec with 70kHz data rate and 290 ffts/sec.
(Interestingly, Numeric FFT accumulates much slower..)
(Commenting out that one line stops memory growth.)

What can one do to alleviate this?
Can I del() some numpy object or such?
It's a bit of an issue for a program that needs to run for weeks.

It's purpose is simply to argmax() the largest bin, which always 
falls within a small range - do I have another, better option than 
fft?

Cheers,
Ray Schumacher
Blue Cove Interactive



From oliphant at enthought.com  Wed Oct 31 18:26:42 2007
From: oliphant at enthought.com (Travis E. Oliphant)
Date: Wed, 31 Oct 2007 17:26:42 -0500
Subject: [Numpy-discussion] numpy FFT memory accumulation
In-Reply-To: <6.2.3.4.2.20071031133112.058c23d8@blue-cove.com>
References: <6.2.3.4.2.20071031133112.058c23d8@blue-cove.com>
Message-ID: <47290122.6080407@enthought.com>

Ray S wrote:
> I am using
> fftRes = abs(fft.rfft(data_array[end-2**15:end]))
> to do running analysis on streaming data. The N never changes.
> It sucks memory up at ~1MB/sec with 70kHz data rate and 290 ffts/sec.
> (Interestingly, Numeric FFT accumulates much slower..)
> (Commenting out that one line stops memory growth.)
>   
At first glance, I would say that I don't expect memory to be growing 
here, so it looks like a problem with rfft that deserves looking into.

Does data_array keep growing?


Travis O.




From peridot.faceted at gmail.com  Wed Oct 31 18:50:38 2007
From: peridot.faceted at gmail.com (Anne Archibald)
Date: Wed, 31 Oct 2007 18:50:38 -0400
Subject: [Numpy-discussion] numpy FFT memory accumulation
In-Reply-To: <6.2.3.4.2.20071031133112.058c23d8@blue-cove.com>
References: <6.2.3.4.2.20071031133112.058c23d8@blue-cove.com>
Message-ID: <ce557a360710311550m64867440p6b81144bdb2708d@mail.gmail.com>

On 31/10/2007, Ray S <subscriber100 at rjs.org> wrote:
> I am using
> fftRes = abs(fft.rfft(data_array[end-2**15:end]))
> to do running analysis on streaming data. The N never changes.
> It sucks memory up at ~1MB/sec with 70kHz data rate and 290 ffts/sec.
> (Interestingly, Numeric FFT accumulates much slower..)
> (Commenting out that one line stops memory growth.)
>
> What can one do to alleviate this?
> Can I del() some numpy object or such?
> It's a bit of an issue for a program that needs to run for weeks.
>
> It's purpose is simply to argmax() the largest bin, which always
> falls within a small range - do I have another, better option than
> fft?

If the range is *really* small, you can try using a DFT - sometimes
that is fast enough, and gives you just the bins you're curious about.

If the range is bigger than that, but still a small fraction of the
FFT size, you can do some tricks where you band-pass filter the data
(with a FIR filter, say), downsample (which aliases frequencies
downward), and then FFT. Concretely, if your data is sampled at
80ksamp/s and you're taking 160ksamp FFTs, you're getting the spectrum
up to 35 kHz with a resolution of 0.5 Hz. If all you care about is the
spectral region from 5 kHz to 6 kHz but you still need the 0.5 Hz
spectral resolution, you can run a band-pass filter to throw out
everything outside (say) 4 to 7 kHz, then downsample to 8 ksamp/s; the
4 to 7kHz region will appear mirrored in the 1 to 4 kHz region of the
new FFTs, which will each be a tenth the size.

There are also "zoom fft" and "chirp-z" techniques which are supposed
to give you only part of the FFT, but the wisdom is that unless you
want less than a few percent of the data points you're better just
FFTing and throwing lots of data away.

Anne